2024-05-22T23:34:54.666 INFO:root:teuthology version: 0.0.1.dev287+gb9e3da8 2024-05-22T23:34:54.666 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} --archive /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265 --verbose --name teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/orig.config.yaml 2024-05-22T23:34:54.726 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-22T23:34:54.824 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7721265' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule name: teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 2e1f76222e88afe0f1cda37775b2d45d88289794 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 2e1f76222e88afe0f1cda37775b2d45d88289794 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 935217f1cb4dc722d5303f5ad364c883e9133376 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4697 sha1: 2e1f76222e88afe0f1cda37775b2d45d88289794 sleep_before_teardown: 0 subset: 12/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 935217f1cb4dc722d5303f5ad364c883e9133376 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCb2V8a0o1Mb9y5Nl6ne/jtEYcERaJqNsjv1PZ1ZrK7QYCpuXKTUcgkMpA/PERiXBzxUGXsX8DialyOwhadStpE= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZ+0LAOniidG58fCZigbIyyEyf2D5/BvYlzmSgGMQ4ZqqaRC1QTv+p3i+2zRIlf5OCseSOe66rj/nBUof4p070= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuO27vrnpS4mbnqYUMO7xG0+eTBeQ95bJ6sNoNNA65HzoSP+0wdo7CchLKM/GNm9DtCszfXwFaI8wRc1SHBUoc= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph config set mgr mgr/snap_schedule/log_level debug - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a timestamp: 2024-05-22_21:24:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-22T23:34:54.824 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa; will attempt to use it 2024-05-22T23:34:54.825 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks 2024-05-22T23:34:54.825 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-22T23:34:54.827 INFO:teuthology.task.internal:Checking packages... 2024-05-22T23:34:54.851 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2e1f76222e88afe0f1cda37775b2d45d88289794' 2024-05-22T23:34:54.851 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-22T23:34:54.851 INFO:teuthology.packaging:ref: None 2024-05-22T23:34:54.851 INFO:teuthology.packaging:tag: None 2024-05-22T23:34:54.852 INFO:teuthology.packaging:branch: squid 2024-05-22T23:34:54.852 INFO:teuthology.packaging:sha1: 2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:34:54.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-22T23:34:55.156 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2804.g935217f1 2024-05-22T23:34:55.157 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-22T23:34:55.164 INFO:teuthology.task.internal:no buildpackages task found 2024-05-22T23:34:55.164 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-22T23:34:55.170 INFO:teuthology.task.internal:Saving configuration 2024-05-22T23:34:55.195 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-22T23:34:55.202 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-22T23:34:55.227 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-22 23:29:23.691369', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCb2V8a0o1Mb9y5Nl6ne/jtEYcERaJqNsjv1PZ1ZrK7QYCpuXKTUcgkMpA/PERiXBzxUGXsX8DialyOwhadStpE='} 2024-05-22T23:34:55.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-22 23:29:23.692810', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZ+0LAOniidG58fCZigbIyyEyf2D5/BvYlzmSgGMQ4ZqqaRC1QTv+p3i+2zRIlf5OCseSOe66rj/nBUof4p070='} 2024-05-22T23:34:55.270 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-22 23:29:23.689893', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuO27vrnpS4mbnqYUMO7xG0+eTBeQ95bJ6sNoNNA65HzoSP+0wdo7CchLKM/GNm9DtCszfXwFaI8wRc1SHBUoc='} 2024-05-22T23:34:55.270 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-22T23:34:55.276 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-22T23:34:55.276 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-22T23:34:55.276 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-22T23:34:55.276 INFO:teuthology.run_tasks:Running task console_log... 2024-05-22T23:34:55.362 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fad28af3be0>, signals=[15]) 2024-05-22T23:34:55.363 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-22T23:34:55.368 INFO:teuthology.task.internal:Opening connections... 2024-05-22T23:34:55.368 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-05-22T23:34:55.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:34:55.449 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-05-22T23:34:55.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:34:55.525 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-05-22T23:34:55.526 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:34:55.599 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-22T23:34:55.605 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-05-22T23:34:55.623 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-05-22T23:34:55.623 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-05-22T23:34:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-05-22T23:34:55.680 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-22T23:34:55.681 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-22T23:34:55.682 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-22T23:34:55.682 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-05-22T23:34:55.707 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-05-22T23:34:55.725 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-05-22T23:34:55.725 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-05-22T23:34:55.782 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-05-22T23:34:55.782 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-22T23:34:55.783 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-22T23:34:55.784 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-05-22T23:34:55.807 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-05-22T23:34:55.825 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-05-22T23:34:55.825 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-05-22T23:34:55.881 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-05-22T23:34:55.881 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-22T23:34:55.882 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-22T23:34:55.883 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-05-22T23:34:55.907 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-22T23:34:55.918 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-22T23:34:55.924 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-22T23:34:55.924 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-05-22T23:34:55.926 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-05-22T23:34:55.929 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-05-22T23:34:55.945 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-22T23:34:55.951 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-22T23:34:55.952 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-05-22T23:34:55.984 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-05-22T23:34:55.986 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-05-22T23:34:56.082 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-22T23:34:56.160 INFO:teuthology.run_tasks:Running task kernel... 2024-05-22T23:34:56.174 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-05-22T23:34:56.175 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-05-22T23:34:56.175 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-05-22T23:34:56.175 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-22T23:34:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-22T23:34:56.192 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-05-22T23:34:56.381 DEBUG:teuthology.task.kernel:sha1 for client.0 is 4fed890e8ce3966dbe380e9eb4b0a796eddb841c 2024-05-22T23:34:56.381 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4fed890e8ce3966dbe380e9eb4b0a796eddb841c"... 2024-05-22T23:34:56.382 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-05-22T23:34:56.399 INFO:teuthology.orchestra.run.smithi103.stdout:5.14.0-437.el9.x86_64 2024-05-22T23:34:56.399 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 4fed890e8ce3966dbe380e9eb4b0a796eddb841c 2024-05-22T23:34:56.400 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-05-22T23:34:56.400 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum upgrade -y linux-firmware 2024-05-22T23:34:58.539 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.1 MB 00:00 2024-05-22T23:35:03.307 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 5.2 MB/s | 19 MB 00:03 2024-05-22T23:35:09.549 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.2 MB 00:02 2024-05-22T23:35:11.049 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 112 kB/s | 16 kB 00:00 2024-05-22T23:35:12.134 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-05-22T23:35:17.218 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-05-22T23:35:19.163 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-05-22T23:35:19.164 INFO:teuthology.orchestra.run.smithi103.stdout:Nothing to do. 2024-05-22T23:35:19.164 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-05-22T23:35:19.280 INFO:teuthology.task.kernel:Downloading kernel 4fed890e8ce3966dbe380e9eb4b0a796eddb841c on client.0... 2024-05-22T23:35:19.280 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=4fed890e8ce3966dbe380e9eb4b0a796eddb841c 2024-05-22T23:35:19.500 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/4fed890e8ce3966dbe380e9eb4b0a796eddb841c/centos/9/flavors/default/x86_64/ 2024-05-22T23:35:19.500 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g4fed890e8ce3-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/4fed890e8ce3966dbe380e9eb4b0a796eddb841c/centos/9/flavors/default/x86_64/ --input-file=- 2024-05-22T23:35:19.503 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi103.front.sepia.ceph.com to complete... 2024-05-22T23:35:31.614 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22 23:35:31 URL:https://2.chacra.ceph.com/r/kernel/testing/4fed890e8ce3966dbe380e9eb4b0a796eddb841c/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g4fed890e8ce3-1.x86_64.rpm [357350900/357350900] -> "/tmp/kernel.x86_64.rpm" [1] 2024-05-22T23:35:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:FINISHED --2024-05-22 23:35:31-- 2024-05-22T23:35:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:Total wall clock time: 12s 2024-05-22T23:35:31.615 INFO:teuthology.orchestra.run.smithi103.stderr:Downloaded: 1 files, 341M in 12s (28.9 MB/s) 2024-05-22T23:35:31.618 INFO:teuthology.task.kernel:Installing kernel 4fed890e8ce3966dbe380e9eb4b0a796eddb841c on client.0... 2024-05-22T23:35:31.618 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-05-22T23:35:33.773 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying... ######################################## 2024-05-22T23:35:33.987 INFO:teuthology.orchestra.run.smithi103.stdout:Preparing... ######################################## 2024-05-22T23:35:34.099 INFO:teuthology.orchestra.run.smithi103.stdout:Updating / installing... 2024-05-22T23:35:42.038 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-6.9.0_rc3_g4fed890e8ce3-1 ######################################## 2024-05-22T23:36:21.990 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi103.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-05-22T23:36:21.991 DEBUG:teuthology.orchestra.run.smithi103:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-05-22T23:36:22.078 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/System.map-6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:36:22.078 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/config-6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/boot/vmlinuz-6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/System.map 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/config 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-05-22T23:36:22.079 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/crc32c-intel.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/serpent-avx2.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-05-22T23:36:22.080 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/kernel 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/kernel/cpu 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/kernel/cpu/mce 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/kvm 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/kvm/kvm-amd.ko 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/kvm/kvm-intel.ko 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/arch/x86/kvm/kvm.ko 2024-05-22T23:36:22.081 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/ansi_cprng.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/anubis.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/arc4.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/async_tx 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/async_tx/async_memcpy.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/async_tx/async_pq.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/async_tx/async_raid6_recov.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/async_tx/async_tx.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/async_tx/async_xor.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/async_tx/raid6test.ko 2024-05-22T23:36:22.082 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/authenc.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/authencesn.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/blake2b_generic.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/blowfish_common.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/blowfish_generic.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/camellia_generic.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/cast5_generic.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/cast6_generic.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/cast_common.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/ccm.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/cmac.ko 2024-05-22T23:36:22.083 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/crc32_generic.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/crypto_engine.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/crypto_user.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/cts.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/des_generic.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/dh_generic.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/echainiv.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/essiv.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/fcrypt.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/khazad.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/lz4.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/lz4hc.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/md4.ko 2024-05-22T23:36:22.084 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/michael_mic.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/pcbc.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/pcrypt.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/rmd160.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/seed.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/serpent_generic.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/tcrypt.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/tea.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/twofish_common.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/twofish_generic.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/vmac.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/wp512.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/xcbc.ko 2024-05-22T23:36:22.085 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/xor.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/crypto/xxhash_generic.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/acpi 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/acpi/acpi_ipmi.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/acpi/acpi_pad.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/acpi/ec_sys.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/acpi/platform_profile.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/acpi/sbs.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/acpi/sbshc.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/acpi/video.ko 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata 2024-05-22T23:36:22.086 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/acard-ahci.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/ahci_platform.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/ata_generic.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/libahci_platform.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_acpi.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_ali.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_amd.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_artop.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_atiixp.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_atp867x.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_cmd640.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_cmd64x.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_cypress.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_efar.ko 2024-05-22T23:36:22.087 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_hpt366.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_hpt37x.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_hpt3x2n.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_hpt3x3.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_it8213.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_it821x.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_jmicron.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_marvell.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_mpiix.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_netcell.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_ninja32.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_ns87410.ko 2024-05-22T23:36:22.088 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_ns87415.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_oldpiix.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_opti.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_optidma.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_pcmcia.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_pdc2027x.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_piccolo.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_rdc.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_sch.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_serverworks.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_sil680.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_sis.ko 2024-05-22T23:36:22.089 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_sl82c105.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_triflex.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pata_via.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/pdc_adma.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_inic162x.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_mv.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_nv.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_promise.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_qstor.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_sil.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_sil24.ko 2024-05-22T23:36:22.090 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_sis.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_svw.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_sx4.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_uli.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_via.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ata/sata_vsc.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/auxdisplay 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/auxdisplay/ks0108.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/base 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/base/regmap 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/base/regmap/regmap-i2c.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/base/regmap/regmap-mmio.ko 2024-05-22T23:36:22.091 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/bcma 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/bcma/bcma.ko 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/aoe 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/aoe/aoe.ko 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/brd.ko 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/drbd 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/drbd/drbd.ko 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/floppy.ko 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/loop.ko 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/mtip32xx 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/nbd.ko 2024-05-22T23:36:22.092 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/null_blk 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/null_blk/null_blk.ko 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/pktcdvd.ko 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/rbd.ko 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/virtio_blk.ko 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/xen-blkback 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/xen-blkfront.ko 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/zram 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/block/zram/zram.ko 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/hangcheck-timer.ko 2024-05-22T23:36:22.093 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/hw_random 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/hw_random/amd-rng.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/hw_random/intel-rng.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/hw_random/via-rng.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/hw_random/virtio-rng.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/ipmi 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/ipmi/ipmi_si.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-05-22T23:36:22.094 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/lp.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/mwave 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/mwave/mwave.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/ppdev.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/tlclk.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/tpm 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/tpm/tpm.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/tpm/tpm_atmel.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/tpm/tpm_infineon.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/tpm/tpm_nsc.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/tpm/tpm_tis.ko 2024-05-22T23:36:22.095 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/uv_mmtimer.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/char/virtio_console.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/cpufreq 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/cpufreq/p4-clockmod.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/cpufreq/powernow-k8.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/cpufreq/speedstep-lib.ko 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/ccp 2024-05-22T23:36:22.096 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/ccp/ccp.ko 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/intel 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/intel/qat 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/intel/qat/qat_common 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/padlock-aes.ko 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/padlock-sha.ko 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/virtio 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/dca 2024-05-22T23:36:22.097 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/dca/dca.ko 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/devfreq 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/dma 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/dma/dw 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/dma/dw/dw_dmac.ko 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/dma/ioat 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/dma/ioat/ioatdma.ko 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/amd64_edac.ko 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/e752x_edac.ko 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/edac_mce_amd.ko 2024-05-22T23:36:22.098 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/i3000_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/i3200_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/i5100_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/i5400_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/i7300_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/i7core_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/i82975x_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/ie31200_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/sb_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/edac/x38_edac.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/firmware 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/firmware/edd.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/firmware/iscsi_ibft.ko 2024-05-22T23:36:22.099 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/gpio 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/gpio/gpio-ich.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/gpio/gpio-viperboard.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-appleir.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-aureal.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-axff.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-dr.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-elecom.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-elo.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-emsff.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-gaff.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-gt683r.ko 2024-05-22T23:36:22.100 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-gyration.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-holtek-kbd.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-holtek-mouse.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-holtekff.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-hyperv.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-icade.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-keytouch.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-kye.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-lcpower.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-led.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-lenovo.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-logitech-dj.ko 2024-05-22T23:36:22.101 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-multitouch.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-ortek.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-penmount.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-petalynx.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-picolcd.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-pl.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-plantronics.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-primax.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-rmi.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-arvo.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-common.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-isku.ko 2024-05-22T23:36:22.102 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-kone.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-konepure.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-lua.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-pyra.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-ryos.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat-savu.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-roccat.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-saitek.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-samsung.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-sensor-hub.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-sjoy.ko 2024-05-22T23:36:22.103 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-sony.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-speedlink.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-steelseries.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-sunplus.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-thrustmaster.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-tivo.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-tmff.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-topseed.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-twinhan.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-uclogic.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-waltop.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-wiimote.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-xinmo.ko 2024-05-22T23:36:22.104 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-zpff.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/hid-zydacron.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/uhid.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hid/wacom.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hv 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hv/hv_balloon.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hv/hv_utils.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hv/hv_vmbus.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/abituguru.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/abituguru3.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/acpi_power_meter.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ad7414.ko 2024-05-22T23:36:22.105 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ad7418.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adc128d818.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adm1025.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adm1026.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adm1029.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adm1031.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adm9240.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ads7828.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adt7410.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adt7411.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adt7462.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adt7470.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adt7475.ko 2024-05-22T23:36:22.106 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/adt7x10.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/amc6821.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/applesmc.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/asb100.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/asc7621.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/asus_atk0110.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/atxp1.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/coretemp.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/dme1737.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ds1621.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ds620.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/emc1403.ko 2024-05-22T23:36:22.107 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/emc6w201.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/f71805f.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/f71882fg.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/f75375s.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/fam15h_power.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/fschmd.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/g760a.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/g762.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/gl518sm.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/gl520sm.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/hwmon-vid.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/i5500_temp.ko 2024-05-22T23:36:22.108 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/i5k_amb.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ibmaem.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ibmpex.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ina209.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ina2xx.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/it87.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/k10temp.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/k8temp.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lineage-pem.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm63.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm73.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm75.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm77.ko 2024-05-22T23:36:22.109 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm78.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm80.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm83.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm85.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm87.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm90.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm92.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm93.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm95234.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm95241.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/lm95245.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ltc2945.ko 2024-05-22T23:36:22.110 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ltc4151.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ltc4215.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ltc4222.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ltc4245.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ltc4260.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/ltc4261.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/max16065.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/max1619.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/max1668.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/max197.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/max6639.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/max6650.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/max6697.ko 2024-05-22T23:36:22.111 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/mcp3021.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/nct6683.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/nct6775-core.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/nct6775.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/nct7802.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pc87360.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pc87427.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pcf8591.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/max16064.ko 2024-05-22T23:36:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/max34440.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/max8688.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/powr1220.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/sch5627.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/sch5636.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/sch56xx-common.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/sht15.ko 2024-05-22T23:36:22.113 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/sht21.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/shtc1.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/sis5595.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/smsc47b397.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/smsc47m1.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/smsc47m192.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/thmc50.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/tmp102.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/tmp103.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/tmp401.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/tmp421.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/via-cputemp.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/via686a.ko 2024-05-22T23:36:22.114 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/vt1211.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/vt8231.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83627ehf.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83627hf.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83781d.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83791d.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83792d.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83793.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83795.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83l785ts.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/hwmon/w83l786ng.ko 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/algos 2024-05-22T23:36:22.115 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-i801.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-isch.ko 2024-05-22T23:36:22.116 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-parport.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-via.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-05-22T23:36:22.117 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/i2c-dev.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/i2c-mux.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/i2c-smbus.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/i2c/i2c-stub.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/core 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/core/ib_cm.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/core/ib_core.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/core/ib_umad.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/core/iw_cm.ko 2024-05-22T23:36:22.118 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/core/rdma_cm.ko 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw/cxgb4 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw/mlx4 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw/mlx5 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw/mthca 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/ipoib 2024-05-22T23:36:22.119 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/iser 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/isert 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/srp 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/srpt 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/gameport 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/gameport/fm801-gp.ko 2024-05-22T23:36:22.120 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/gameport/gameport.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/joydev.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/keyboard 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/keyboard/gpio_keys.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse/appletouch.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse/bcm5974.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse/cyapatp.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse/elan_i2c.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse/sermouse.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse/synaptics_usb.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/mouse/vsxxxaa.ko 2024-05-22T23:36:22.121 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/rmi4 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/rmi4/rmi_core.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/serio 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/serio/altera_ps2.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/serio/arc_ps2.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/serio/serio_raw.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/input/sparse-keymap.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/leds-blinkm.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/leds-lm3530.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/leds-lp3944.ko 2024-05-22T23:36:22.122 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/leds-ss4200.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/bcache 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/bcache/bcache.ko 2024-05-22T23:36:22.123 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-bio-prison.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-cache-smq.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-cache.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-crypt.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-delay.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-flakey.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-log-userspace.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-multipath.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-queue-length.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-raid.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-round-robin.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-service-time.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-switch.ko 2024-05-22T23:36:22.124 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-thin-pool.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/dm-verity.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/persistent-data 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/raid0.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/raid1.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/raid10.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/md/raid456.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message/fusion 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message/fusion/mptbase.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message/fusion/mptctl.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message/fusion/mptfc.ko 2024-05-22T23:36:22.125 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message/fusion/mptlan.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message/fusion/mptsas.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message/fusion/mptscsih.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/message/fusion/mptspi.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/mfd 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/mfd/lpc_ich.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/mfd/lpc_sch.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/mfd/mfd-core.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/mfd/sm501.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/mfd/viperboard.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/mfd/vx855.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/mfd/wl1273-core.ko 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc 2024-05-22T23:36:22.126 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/altera-stapl 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/apds9802als.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/apds990x.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/bh1770glc.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/cb710 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/cb710/cb710.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/eeprom 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/eeprom/at24.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/eeprom/max6875.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/enclosure.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/hpilo.ko 2024-05-22T23:36:22.127 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/isl29003.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/isl29020.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/lis3lv02d 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/mei 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/mei/mei-me.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/mei/mei-txe.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/mei/mei.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/sgi-gru 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/sgi-gru/gru.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/sgi-xp 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/sgi-xp/xp.ko 2024-05-22T23:36:22.128 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/sgi-xp/xpc.ko 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/tifm_7xx1.ko 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/tifm_core.ko 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/tsl2550.ko 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/vmw_balloon.ko 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/vmw_vmci 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/bonding 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/bonding/bonding.ko 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/dsa 2024-05-22T23:36:22.129 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/dsa/b53 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/dsa/b53/b53_common.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/dsa/bcm-sf2.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/dsa/mv88e6060.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/dsa/mv88e6xxx 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/dummy.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/eql.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/3com 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-05-22T23:36:22.130 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/8390 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/8390/8390.ko 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/adaptec 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/agere 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/agere/et131x.ko 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/alteon 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/altera 2024-05-22T23:36:22.131 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/amd 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/alx 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/atl1c 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/atl1e 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/atlx 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-05-22T23:36:22.132 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom/genet 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/brocade 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/brocade/bna 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-05-22T23:36:22.133 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/cxgb 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/cisco 2024-05-22T23:36:22.134 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/cisco/enic 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dec 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dec/tulip 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dlink 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-05-22T23:36:22.135 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/dnet.ko 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/emulex 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/emulex/benet 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/ethoc.ko 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/fealnx.ko 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/e100.ko 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/e1000 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/e1000e 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/fm10k 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-05-22T23:36:22.136 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/i40e 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/iavf 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/igb 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/igbvf 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/ixgbe 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/ixgbevf 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/jme.ko 2024-05-22T23:36:22.137 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/marvell 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/marvell/skge.ko 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/mellanox 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/mellanox/mlx4 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/mellanox/mlx5 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/micrel 2024-05-22T23:36:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/myricom 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/myricom/myri10ge 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/natsemi 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/neterion 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/nvidia 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/packetengines 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-05-22T23:36:22.139 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/qlogic 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/qlogic/netxen 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/rdc 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/realtek 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/realtek/atp.ko 2024-05-22T23:36:22.140 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/rocker 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sfc 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/silan 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sis 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sis/sis190.ko 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sis/sis900.ko 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/smsc 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-05-22T23:36:22.141 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/stmicro 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/stmicro/stmmac 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sun 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sun/cassini.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sun/niu.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sun/sungem.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/tehuti 2024-05-22T23:36:22.142 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/ti 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/ti/tlan.ko 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/via 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/wiznet 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/xircom 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/geneve.ko 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/hyperv 2024-05-22T23:36:22.143 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ieee802154 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ieee802154/fakelb.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ifb.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ipvlan 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ipvlan/ipvlan.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/macvlan.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/macvtap.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/mdio 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/mdio.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/mdio/mdio-regmap.ko 2024-05-22T23:36:22.144 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/mii.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/net_failover.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/netconsole.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/nlmon.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/pcs 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/pcs/pcs-lynx.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/amd.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/bcm7xxx.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/bcm87xx.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/broadcom.ko 2024-05-22T23:36:22.145 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/cicada.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/davicom.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/dp83640.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/et1011c.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/icplus.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/lxt.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/marvell.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/micrel.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/national.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/phylink.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/qsemi.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/realtek.ko 2024-05-22T23:36:22.146 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/smsc.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/ste10Xp.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/phy/vitesse.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/bsd_comp.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/ppp_async.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/ppp_deflate.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/ppp_generic.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/ppp_mppe.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/ppp_synctty.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/pppoe.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/pppox.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/ppp/pptp.ko 2024-05-22T23:36:22.147 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/slip 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/slip/slhc.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/slip/slip.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/sungem_phy.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/tap.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/team 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/team/team.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/team/team_mode_activebackup.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/team/team_mode_broadcast.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/team/team_mode_random.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/tun.ko 2024-05-22T23:36:22.148 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/veth.ko 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/virtio_net.ko 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/vmxnet3 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/vxlan 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/vxlan/vxlan.ko 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/xen-netback 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/xen-netback/xen-netback.ko 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/net/xen-netfront.ko 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ntb 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ntb/ntb.ko 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/nvme 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/nvme/host 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/nvme/host/nvme-core.ko 2024-05-22T23:36:22.149 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/nvme/host/nvme.ko 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/parport 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/parport/parport.ko 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/parport/parport_cs.ko 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/parport/parport_pc.ko 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/parport/parport_serial.ko 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pci 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pci/hotplug 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pci/pcie 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pci/pcie/aer_inject.ko 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pci/xen-pcifront.ko 2024-05-22T23:36:22.150 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pcmcia 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pcmcia/i82092.ko 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pcmcia/pd6729.ko 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pcmcia/yenta_socket.ko 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pinctrl 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pinctrl/intel 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/acer-wmi.ko 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/acerhdf.ko 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/apple-gmux.ko 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/asus-laptop.ko 2024-05-22T23:36:22.151 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/asus-wmi.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/classmate-laptop.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/hdaps.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/intel 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/intel_ips.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/msi-wmi.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/mxm-wmi.ko 2024-05-22T23:36:22.152 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/samsung-laptop.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/samsung-q10.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/topstar-laptop.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/toshiba_haps.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/wmi-bmof.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/platform/x86/wmi.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/powercap 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/powercap/intel_rapl_common.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/powercap/intel_rapl_msr.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pps 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pps/clients 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pps/clients/pps-gpio.ko 2024-05-22T23:36:22.153 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pps/clients/pps-ldisc.ko 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pps/clients/pps_parport.ko 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/pps/pps_core.ko 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ptp 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ptp/ptp.ko 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ptp/ptp_kvm.ko 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ras 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ras/amd 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ras/amd/atl 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ras/amd/fmpm.ko 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-bq32k.ko 2024-05-22T23:36:22.154 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds1286.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds1307.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds1374.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds1511.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds1553.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds1672.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds1742.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds2404.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-ds3232.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-em3027.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-fm3130.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-isl12022.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-isl1208.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-m41t80.ko 2024-05-22T23:36:22.155 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-m48t35.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-m48t59.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-max6900.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-msm6242.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-pcf2127.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-pcf85063.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-pcf8523.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-pcf8563.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-pcf8583.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-rp5c01.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-rs5c372.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-rx8025.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-rx8581.ko 2024-05-22T23:36:22.156 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/rtc/rtc-x1205.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/3w-9xxx.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/3w-sas.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/3w-xxxx.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/BusLogic.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/a100u2w.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/aacraid 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/aacraid/aacraid.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/advansys.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/aic7xxx 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-05-22T23:36:22.157 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/am53c974.ko 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/arcmsr 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/atp870u.ko 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/be2iscsi 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/bfa 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/bfa/bfa.ko 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/bnx2fc 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/bnx2i 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-05-22T23:36:22.158 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/ch.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/csiostor 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/csiostor/csiostor.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/cxgbi 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/cxgbi/cxgb3i 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/cxgbi/cxgb4i 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/dc395x.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/device_handler 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-05-22T23:36:22.159 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/dmx3191d.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/esas2r 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/esas2r/esas2r.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/esp_scsi.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/fcoe 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/fcoe/fcoe.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/fnic 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/fnic/fnic.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/hpsa.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/hptiop.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/hv_storvsc.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/initio.ko 2024-05-22T23:36:22.160 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/ipr.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/ips.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/isci 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/isci/isci.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/iscsi_tcp.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/libfc 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/libfc/libfc.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/libiscsi.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/libiscsi_tcp.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/libsas 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/libsas/libsas.ko 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/lpfc 2024-05-22T23:36:22.161 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/lpfc/lpfc.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/megaraid 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/megaraid.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/mpt3sas 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/mvsas 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/mvsas/mvsas.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/mvumi.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/pm8001 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-05-22T23:36:22.162 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/pmcraid.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/qla1280.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/qla2xxx 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/qla4xxx 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/raid_class.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/scsi_debug.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/scsi_transport_fc.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/scsi_transport_sas.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/scsi_transport_spi.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/scsi_transport_srp.ko 2024-05-22T23:36:22.163 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/ses.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/st.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/stex.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/sym53c8xx_2 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/virtio_scsi.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/vmw_pvscsi.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/scsi/wd719x.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ssb 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ssb/ssb.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/iscsi 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-05-22T23:36:22.164 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/loopback 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/loopback/tcm_loop.ko 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/target_core_file.ko 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/target_core_iblock.ko 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/target_core_mod.ko 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/target_core_pscsi.ko 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/target_core_user.ko 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/tcm_fc 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-05-22T23:36:22.165 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-05-22T23:36:22.166 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/ipwireless 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/n_gsm.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/n_hdlc.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/nozomi.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/serial 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/serial/8250 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-05-22T23:36:22.167 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/serial/arc_uart.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/serial/jsm 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/tty/serial/jsm/jsm.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ufs 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ufs/core 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ufs/core/ufshcd-core.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ufs/host 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/uio 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/uio/uio.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/uio/uio_aec.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/uio/uio_cif.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/uio/uio_pci_generic.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/uio/uio_sercos3.ko 2024-05-22T23:36:22.168 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/class 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/class/cdc-acm.ko 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/class/cdc-wdm.ko 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/class/usblp.ko 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/class/usbtmc.ko 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/host 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/host/sl811-hcd.ko 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/image 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/image/mdc800.ko 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/image/microtek.ko 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/adutux.ko 2024-05-22T23:36:22.169 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/appledisplay.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/emi26.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/emi62.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/ezusb.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/idmouse.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/iowarrior.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/isight_firmware.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/ldusb.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/legousbtower.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/sisusbvga 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/trancevibrator.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/usb3503.ko 2024-05-22T23:36:22.170 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/usblcd.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/usbsevseg.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/uss720.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/misc/yurex.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/phy 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/phy/phy-generic.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/aircable.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/ark3116.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/belkin_sa.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/ch341.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/cp210x.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/cyberjack.ko 2024-05-22T23:36:22.171 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/cypress_m8.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/digi_acceleport.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/empeg.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/ftdi_sio.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/garmin_gps.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/io_edgeport.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/io_ti.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/ipaq.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/ipw.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/ir-usb.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/keyspan.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/keyspan_pda.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/kl5kusb105.ko 2024-05-22T23:36:22.172 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/kobil_sct.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/mct_u232.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/mos7720.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/mos7840.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/navman.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/omninet.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/opticon.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/option.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/oti6858.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/pl2303.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/qcaux.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/qcserial.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/quatech2.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/safe_serial.ko 2024-05-22T23:36:22.173 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/sierra.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/spcp8x5.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/ssu100.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/symbolserial.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/usb_debug.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/usb_wwan.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/visor.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/whiteheat.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/serial/xsens_mt.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/uas.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-alauda.ko 2024-05-22T23:36:22.174 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-cypress.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-datafab.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-freecom.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-isd200.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-karma.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-onetouch.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-realtek.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-sddr09.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-sddr55.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/ums-usbat.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/storage/usb-storage.ko 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/usbip 2024-05-22T23:36:22.175 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/usbip/usbip-core.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/usbip/usbip-host.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vfio 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vfio/pci 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vfio/pci/vfio-pci.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vfio/vfio.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vhost 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vhost/vhost.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vhost/vhost_iotlb.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vhost/vhost_net.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/vhost/vhost_scsi.ko 2024-05-22T23:36:22.176 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/video 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/video/backlight 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/video/backlight/apple_bl.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/video/backlight/lcd.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/video/backlight/platform_lcd.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/virtio 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/virtio/virtio.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/virtio/virtio_balloon.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/virtio/virtio_mmio.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/virtio/virtio_pci.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/virtio/virtio_ring.ko 2024-05-22T23:36:22.177 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/masters 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/masters/ds2482.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/masters/ds2490.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-05-22T23:36:22.178 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_smem.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/slaves/w1_therm.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/w1/wire.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/alim1535_wdt.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/alim7101_wdt.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/f71808e_wdt.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/hpwdt.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/i6300esb.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/iTCO_wdt.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/ib700wdt.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/ibmasr.ko 2024-05-22T23:36:22.179 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/it8712f_wdt.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/it87_wdt.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/machzwd.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/nv_tco.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/pcwd_pci.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/pcwd_usb.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/sch311x_wdt.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/softdog.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/sp5100_tco.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/via_wdt.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/w83877f_wdt.ko 2024-05-22T23:36:22.180 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/w83977f_wdt.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/wdt_pci.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/watchdog/xen_wdt.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xen-acpi-processor.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xen-evtchn.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xen-gntalloc.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xen-gntdev.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xen-pciback 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xen-privcmd.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xenfs 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/drivers/xen/xenfs/xenfs.ko 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs 2024-05-22T23:36:22.181 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/9p 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/9p/9p.ko 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/affs 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/affs/affs.ko 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/befs 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/befs/befs.ko 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/binfmt_misc.ko 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/btrfs 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/btrfs/btrfs.ko 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/cachefiles 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/cachefiles/cachefiles.ko 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ceph 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ceph/ceph.ko 2024-05-22T23:36:22.182 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/coda 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/coda/coda.ko 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/cramfs 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/cramfs/cramfs.ko 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/dlm 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/dlm/dlm.ko 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ecryptfs 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ecryptfs/ecryptfs.ko 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/f2fs 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/f2fs/f2fs.ko 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/fat 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/fat/fat.ko 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/fat/msdos.ko 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/fat/vfat.ko 2024-05-22T23:36:22.183 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/fuse 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/fuse/cuse.ko 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/fuse/fuse.ko 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/gfs2 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/gfs2/gfs2.ko 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/hfs 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/hfs/hfs.ko 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/hfsplus 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/hfsplus/hfsplus.ko 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/isofs 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/isofs/isofs.ko 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/jfs 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/jfs/jfs.ko 2024-05-22T23:36:22.184 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/lockd 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/lockd/lockd.ko 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/minix 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/minix/minix.ko 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/netfs 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/netfs/netfs.ko 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/blocklayout 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/filelayout 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/flexfilelayout 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/nfs.ko 2024-05-22T23:36:22.185 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/nfsv3.ko 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs/nfsv4.ko 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs_common 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs_common/grace.ko 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfs_common/nfs_acl.ko 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfsd 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nfsd/nfsd.ko 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nilfs2 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nilfs2/nilfs2.ko 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-celtic.ko 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-centeuro.ko 2024-05-22T23:36:22.186 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-croatian.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-cyrillic.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-gaelic.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-greek.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-iceland.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-inuit.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-roman.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-romanian.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/mac-turkish.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp1250.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp1251.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp1255.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp737.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp775.ko 2024-05-22T23:36:22.187 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp850.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp852.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp855.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp857.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp860.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp861.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp862.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp863.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp864.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp865.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp866.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp869.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp874.ko 2024-05-22T23:36:22.188 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp932.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp936.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp949.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_cp950.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_euc-jp.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-1.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-13.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-14.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-15.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-2.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-3.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-4.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-5.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-6.ko 2024-05-22T23:36:22.189 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-7.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_iso8859-9.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_koi8-r.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_koi8-ru.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_koi8-u.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_ucs2_utils.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/nls/nls_utf8.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/cluster 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/dlm 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/dlmfs 2024-05-22T23:36:22.190 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/ocfs2.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/overlayfs 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/overlayfs/overlay.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/pstore 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/pstore/ramoops.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/reiserfs 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/reiserfs/reiserfs.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/romfs 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/romfs/romfs.ko 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/smb 2024-05-22T23:36:22.191 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/smb/client 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/smb/client/cifs.ko 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/smb/common 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/smb/common/cifs_arc4.ko 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/smb/common/cifs_md4.ko 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/squashfs 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/squashfs/squashfs.ko 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/sysv 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/sysv/sysv.ko 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/udf 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/udf/udf.ko 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ufs 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/ufs/ufs.ko 2024-05-22T23:36:22.192 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/xfs 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/fs/xfs/xfs.ko 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/kernel 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/kernel/rcu 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/kernel/rcu/rcutorture.ko 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/kernel/torture.ko 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/kernel/trace 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/asn1_encoder.ko 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/crc-itu-t.ko 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/crc8.ko 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/crypto 2024-05-22T23:36:22.193 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/crypto/libarc4.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/crypto/libdes.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/libcrc32c.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/lru_cache.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/lz4 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/lz4/lz4_compress.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/lz4/lz4hc_compress.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/math 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/math/cordic.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/raid6 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/raid6/raid6_pq.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/reed_solomon 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/reed_solomon/reed_solomon.ko 2024-05-22T23:36:22.194 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/ts_bm.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/ts_fsm.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/ts_kmp.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/zstd 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/lib/zstd/zstd_compress.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/mm 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/mm/hwpoison-inject.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan/6lowpan.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan/nhc_dest.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan/nhc_fragment.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan/nhc_hop.ko 2024-05-22T23:36:22.195 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan/nhc_ipv6.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan/nhc_mobility.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan/nhc_routing.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/6lowpan/nhc_udp.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/802 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/802/garp.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/802/mrp.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/802/p8022.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/802/psnap.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/802/stp.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/8021q 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/8021q/8021q.ko 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/9p 2024-05-22T23:36:22.196 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/9p/9pnet.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/9p/9pnet_fd.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/9p/9pnet_rdma.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/9p/9pnet_virtio.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/batman-adv 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/batman-adv/batman-adv.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/br_netfilter.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/bridge.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_among.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_arp.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-05-22T23:36:22.197 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_ip.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_limit.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_log.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_mark.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_snat.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_stp.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-05-22T23:36:22.198 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/ebtables.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ceph 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ceph/libceph.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/core 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/core/failover.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/core/pktgen.ko 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dccp 2024-05-22T23:36:22.199 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dccp/dccp.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dccp/dccp_diag.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dccp/dccp_ipv4.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dccp/dccp_ipv6.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dns_resolver 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dns_resolver/dns_resolver.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dsa 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dsa/dsa_core.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dsa/tag_brcm.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dsa/tag_dsa.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dsa/tag_none.ko 2024-05-22T23:36:22.202 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/dsa/tag_trailer.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ieee802154 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ieee802154/6lowpan 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ieee802154/ieee802154.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ieee802154/ieee802154_socket.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/ah4.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/esp4.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/fou.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/gre.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/inet_diag.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/ip_gre.ko 2024-05-22T23:36:22.203 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/ip_tunnel.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/ip_vti.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/ipcomp.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/ipip.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/arp_tables.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-05-22T23:36:22.204 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/iptable_security.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_bic.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_dctcp.ko 2024-05-22T23:36:22.205 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_diag.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_highspeed.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_htcp.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_hybla.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_illinois.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_lp.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_scalable.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_vegas.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_veno.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_westwood.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tcp_yeah.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/tunnel4.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/udp_diag.ko 2024-05-22T23:36:22.206 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/udp_tunnel.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv4/xfrm4_tunnel.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/ah6.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/esp6.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/fou6.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/ip6_tunnel.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/ip6_vti.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/ipcomp6.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-05-22T23:36:22.207 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-05-22T23:36:22.208 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/sit.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/tunnel6.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/ipv6/xfrm6_tunnel.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/key 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/key/af_key.ko 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/l2tp 2024-05-22T23:36:22.209 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/l2tp/l2tp_core.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/l2tp/l2tp_debugfs.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/l2tp/l2tp_eth.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/l2tp/l2tp_ip.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/l2tp/l2tp_ip6.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/l2tp/l2tp_netlink.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/l2tp/l2tp_ppp.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/llc 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/llc/llc.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/mac802154 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/mac802154/mac802154.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/mpls 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/mpls/mpls_gso.ko 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter 2024-05-22T23:36:22.210 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-05-22T23:36:22.211 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-05-22T23:36:22.212 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conncount.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_h323.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_irc.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-05-22T23:36:22.213 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_sane.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_sip.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_dup_netdev.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_flow_table.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_flow_table_inet.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_log_syslog.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_nat.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_nat_amanda.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_nat_ftp.ko 2024-05-22T23:36:22.214 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_nat_irc.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_nat_sip.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_nat_tftp.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_synproxy_core.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nf_tables.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nfnetlink.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nfnetlink_acct.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nfnetlink_log.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nfnetlink_osf.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nfnetlink_queue.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_chain_nat.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_compat.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_ct.ko 2024-05-22T23:36:22.215 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_dup_netdev.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_fib.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_fwd_netdev.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_hash.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_limit.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_log.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_masq.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_nat.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_numgen.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_queue.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_quota.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_redir.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_reject.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_reject_inet.ko 2024-05-22T23:36:22.216 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_socket.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/nft_tunnel.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_AUDIT.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_CHECKSUM.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_CLASSIFY.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_CT.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_DSCP.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_HL.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_HMARK.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_IDLETIMER.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_LED.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_LOG.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_MASQUERADE.ko 2024-05-22T23:36:22.217 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_NETMAP.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_NFLOG.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_NFQUEUE.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_RATEEST.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_REDIRECT.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_SECMARK.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_TCPMSS.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_TEE.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_TPROXY.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_TRACE.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_addrtype.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_bpf.ko 2024-05-22T23:36:22.218 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_cgroup.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_cluster.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_comment.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_connbytes.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_connlabel.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_connlimit.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_connmark.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_conntrack.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_cpu.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_dccp.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_devgroup.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_dscp.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_ecn.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_esp.ko 2024-05-22T23:36:22.219 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_hashlimit.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_helper.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_hl.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_ipcomp.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_iprange.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_ipvs.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_l2tp.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_length.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_limit.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_mac.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_mark.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_multiport.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_nat.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_nfacct.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_osf.ko 2024-05-22T23:36:22.220 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_owner.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_physdev.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_pkttype.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_policy.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_quota.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_rateest.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_realm.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_recent.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_sctp.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_set.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_socket.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_state.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_statistic.ko 2024-05-22T23:36:22.221 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_string.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_tcpmss.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_time.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netfilter/xt_u32.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netlink 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/netlink/netlink_diag.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/nsh 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/nsh/nsh.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/openvswitch 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/openvswitch/openvswitch.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/openvswitch/vport-geneve.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/openvswitch/vport-gre.ko 2024-05-22T23:36:22.222 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/openvswitch/vport-vxlan.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/packet 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/packet/af_packet_diag.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/rds 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/rds/rds.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/rds/rds_rdma.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/rds/rds_tcp.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_csum.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_gact.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_mirred.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_nat.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_pedit.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_police.ko 2024-05-22T23:36:22.223 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_simple.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_skbedit.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/act_vlan.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/cls_basic.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/cls_bpf.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/cls_flow.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/cls_fw.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/cls_route.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/cls_u32.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/em_cmp.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/em_ipset.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/em_meta.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/em_nbyte.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/em_text.ko 2024-05-22T23:36:22.224 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/em_u32.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_choke.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_codel.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_drr.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_fq.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_gred.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_hfsc.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_hhf.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_htb.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_ingress.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_mqprio.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_mqprio_lib.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_multiq.ko 2024-05-22T23:36:22.225 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_netem.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_pie.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_plug.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_prio.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_qfq.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_red.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_sfb.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_sfq.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_tbf.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sched/sch_teql.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sctp 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sctp/sctp.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sctp/sctp_diag.ko 2024-05-22T23:36:22.226 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sunrpc 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sunrpc/auth_gss 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sunrpc/sunrpc.ko 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sunrpc/xprtrdma 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/tipc 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/tipc/tipc.ko 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/tipc/tipc_diag.ko 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/unix 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/unix/unix_diag.ko 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/vmw_vsock 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-05-22T23:36:22.227 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/vmw_vsock/vsock.ko 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/vmw_vsock/vsock_diag.ko 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/vmw_vsock/vsock_loopback.ko 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/xfrm 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/net/xfrm/xfrm_ipcomp.ko 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/security 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/security/keys 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/security/keys/encrypted-keys 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/security/keys/trusted-keys 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/kernel/security/keys/trusted-keys/trusted.ko 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.alias 2024-05-22T23:36:22.228 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.alias.bin 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.builtin 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.builtin.alias.bin 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.builtin.bin 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.builtin.modinfo 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.dep 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.dep.bin 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.devname 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.order 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.softdep 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.symbols 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.symbols.bin 2024-05-22T23:36:22.229 INFO:teuthology.orchestra.run.smithi103.stdout:/lib/modules/6.9.0-rc3-g4fed890e8ce3/vmlinuz 2024-05-22T23:36:22.233 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:36:22.233 DEBUG:teuthology.orchestra.run.smithi103:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-05-22T23:36:22.257 INFO:teuthology.orchestra.run.smithi103.stdout:postinstall scriptlet (using /bin/sh): 2024-05-22T23:36:22.257 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-22T23:36:22.257 INFO:teuthology.orchestra.run.smithi103.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g4fed890e8ce3 /lib/modules/6.9.0-rc3-g4fed890e8ce3/vmlinuz 2024-05-22T23:36:22.257 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-05-22T23:36:22.257 INFO:teuthology.orchestra.run.smithi103.stdout:for file in vmlinuz System.map config; do 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g4fed890e8ce3/${file}" "/boot/${file}-6.9.0-rc3-g4fed890e8ce3"; then 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout: cp "/lib/modules/6.9.0-rc3-g4fed890e8ce3/${file}" "/boot/${file}-6.9.0-rc3-g4fed890e8ce3" 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout: fi 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout:done 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -d "/lib/modules/6.9.0-rc3-g4fed890e8ce3/dtb" ] && \ 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g4fed890e8ce3/dtb" "/boot/dtb-6.9.0-rc3-g4fed890e8ce3" >/dev/null 2>&1; then 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g4fed890e8ce3" 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout: cp -r "/lib/modules/6.9.0-rc3-g4fed890e8ce3/dtb" "/boot/dtb-6.9.0-rc3-g4fed890e8ce3" 2024-05-22T23:36:22.258 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-05-22T23:36:22.259 INFO:teuthology.orchestra.run.smithi103.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g4fed890e8ce3/modules.dep" ]; then 2024-05-22T23:36:22.259 INFO:teuthology.orchestra.run.smithi103.stdout: /usr/sbin/depmod 6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:36:22.259 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-05-22T23:36:22.259 INFO:teuthology.orchestra.run.smithi103.stdout:preuninstall scriptlet (using /bin/sh): 2024-05-22T23:36:22.259 INFO:teuthology.orchestra.run.smithi103.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-05-22T23:36:22.259 INFO:teuthology.orchestra.run.smithi103.stdout:kernel-install remove 6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:36:22.259 INFO:teuthology.orchestra.run.smithi103.stdout:fi 2024-05-22T23:36:22.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo rpm -qi grub2-tools 2024-05-22T23:36:22.334 INFO:teuthology.orchestra.run.smithi103.stdout:Name : grub2-tools 2024-05-22T23:36:22.334 INFO:teuthology.orchestra.run.smithi103.stdout:Epoch : 1 2024-05-22T23:36:22.334 INFO:teuthology.orchestra.run.smithi103.stdout:Version : 2.06 2024-05-22T23:36:22.334 INFO:teuthology.orchestra.run.smithi103.stdout:Release : 68.el9 2024-05-22T23:36:22.334 INFO:teuthology.orchestra.run.smithi103.stdout:Architecture: x86_64 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Group : Unspecified 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Size : 8264254 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:License : GPLv3+ 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Packager : builder@centos.org 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Vendor : CentOS 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Summary : Support tools for GRUB. 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout:Description : 2024-05-22T23:36:22.335 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:36:22.336 INFO:teuthology.orchestra.run.smithi103.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-22T23:36:22.336 INFO:teuthology.orchestra.run.smithi103.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-22T23:36:22.336 INFO:teuthology.orchestra.run.smithi103.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-22T23:36:22.336 INFO:teuthology.orchestra.run.smithi103.stdout:hardware devices. 2024-05-22T23:36:22.336 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:36:22.336 INFO:teuthology.orchestra.run.smithi103.stdout:This subpackage provides tools for support of all platforms. 2024-05-22T23:36:22.337 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-22T23:36:22.337 INFO:teuthology.task.kernel:Updating grub on smithi103 to boot 6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:36:22.337 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-22T23:36:22.947 INFO:teuthology.orchestra.run.smithi103.stderr:Generating grub configuration file ... 2024-05-22T23:36:24.344 INFO:teuthology.orchestra.run.smithi103.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-22T23:36:24.373 INFO:teuthology.orchestra.run.smithi103.stderr:done 2024-05-22T23:36:24.376 DEBUG:teuthology.orchestra.run.smithi103:> mktemp 2024-05-22T23:36:24.392 INFO:teuthology.orchestra.run.smithi103.stdout:/tmp/tmp.AJ80xwSdPp 2024-05-22T23:36:24.392 DEBUG:teuthology.orchestra.run.smithi103:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.AJ80xwSdPp 2024-05-22T23:36:24.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0666 /tmp/tmp.AJ80xwSdPp 2024-05-22T23:36:24.609 DEBUG:teuthology.orchestra.remote:smithi103:/tmp/tmp.AJ80xwSdPp is 6KB 2024-05-22T23:36:24.657 DEBUG:teuthology.orchestra.run.smithi103:> rm -fr /tmp/tmp.AJ80xwSdPp 2024-05-22T23:36:24.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo /bin/ls /boot/loader/entries || true 2024-05-22T23:36:24.733 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-22T23:36:24.733 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-22T23:36:24.733 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-22T23:36:24.733 INFO:teuthology.orchestra.run.smithi103.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g4fed890e8ce3.conf 2024-05-22T23:36:24.735 DEBUG:teuthology.orchestra.run.smithi103:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:36:24.882 DEBUG:teuthology.orchestra.run.smithi103:> sudo shutdown -r now 2024-05-22T23:36:54.885 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-22T23:36:54.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-22T23:36:54.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:37:13.251 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-05-22T23:37:22.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-22T23:37:22.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:37:25.310 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-05-22T23:37:37.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-22T23:37:37.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:37:40.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-05-22T23:37:55.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-22T23:37:55.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:37:58.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.103 2024-05-22T23:38:16.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-22T23:38:16.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:38:48.390 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-05-22T23:38:48.912 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-22T23:38:48.912 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g4fed890e8ce3"... 2024-05-22T23:38:48.913 DEBUG:teuthology.orchestra.run.smithi103:> uname -r 2024-05-22T23:38:48.954 INFO:teuthology.orchestra.run.smithi103.stdout:6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:38:48.954 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g4fed890e8ce3 vs 6.9.0-rc3-g4fed890e8ce3 2024-05-22T23:38:48.954 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-22T23:38:48.955 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-22T23:38:49.956 DEBUG:teuthology.parallel:result is None 2024-05-22T23:38:49.956 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-22T23:38:49.967 INFO:teuthology.task.internal:Creating test directory... 2024-05-22T23:38:49.968 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-22T23:38:49.975 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-22T23:38:49.977 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-22T23:38:50.018 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-22T23:38:50.026 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-22T23:38:50.039 INFO:teuthology.task.internal:Creating archive directory... 2024-05-22T23:38:50.039 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-22T23:38:50.078 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-22T23:38:50.086 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-22T23:38:50.141 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-22T23:38:50.167 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-22T23:38:50.168 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-22T23:38:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-22T23:38:50.198 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-22T23:38:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-22T23:38:50.220 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-22T23:38:50.225 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-22T23:38:50.255 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-22T23:38:50.297 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-22T23:38:50.305 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-22T23:38:50.314 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-22T23:38:50.326 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-22T23:38:50.347 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-22T23:38:50.399 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-22T23:38:50.403 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-22T23:38:50.424 INFO:teuthology.task.internal:Configuring sudo... 2024-05-22T23:38:50.424 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-22T23:38:50.433 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-22T23:38:50.438 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-22T23:38:50.499 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-22T23:38:50.577 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-22T23:38:50.577 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-22T23:38:50.582 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-22T23:38:50.585 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-22T23:38:50.622 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-22T23:38:50.766 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-22T23:38:50.860 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-22T23:38:50.947 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-22T23:38:51.039 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:38:51.039 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-22T23:38:51.189 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-22T23:38:51.237 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-22T23:38:51.297 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-22T23:38:51.352 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-22T23:38:51.389 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:38:51.389 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-22T23:38:51.461 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-22T23:38:51.514 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-22T23:38:51.570 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-22T23:38:51.661 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-22T23:38:51.703 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:38:51.703 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-22T23:38:51.770 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-05-22T23:38:51.776 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-05-22T23:38:51.779 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-05-22T23:38:51.813 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-22T23:38:51.844 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-22T23:38:51.874 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-22T23:38:52.370 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-22T23:38:52.381 INFO:teuthology.task.internal:Starting timer... 2024-05-22T23:38:52.381 INFO:teuthology.run_tasks:Running task pcp... 2024-05-22T23:38:52.394 INFO:teuthology.run_tasks:Running task selinux... 2024-05-22T23:38:52.409 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-22T23:38:52.410 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-05-22T23:38:52.544 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-05-22T23:38:52.547 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-05-22T23:38:52.617 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-05-22T23:38:52.619 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-05-22T23:38:52.677 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-05-22T23:38:52.679 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-22T23:38:52.679 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-05-22T23:38:52.726 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-05-22T23:38:52.727 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-05-22T23:38:52.777 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-05-22T23:38:52.777 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-05-22T23:38:52.807 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-05-22T23:38:52.807 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi174.front.sepia.ceph.com': 'permissive'} 2024-05-22T23:38:52.808 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-22T23:38:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-22T23:38:52.870 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-22T23:38:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-22T23:38:52.895 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-22T23:38:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-22T23:38:52.921 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-22T23:38:52.921 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-05-22T23:38:52.982 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-05-22T23:38:53.006 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-05-22T23:38:53.031 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-22T23:38:53.042 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-22T23:38:53.054 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-22T23:38:53.055 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi103.front.sepia.ceph.com,smithi105.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-22T23:45:08.228 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-05-22T23:45:08.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-22T23:45:08.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:45:08.332 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-05-22T23:45:08.479 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-22T23:45:08.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-05-22T23:45:08.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:45:08.558 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-05-22T23:45:08.598 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-05-22T23:45:08.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-05-22T23:45:08.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-22T23:45:08.679 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-05-22T23:45:08.763 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-05-22T23:45:08.764 INFO:teuthology.run_tasks:Running task clock... 2024-05-22T23:45:08.773 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-22T23:45:08.773 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-22T23:45:08.774 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-22T23:45:08.781 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-22T23:45:08.781 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-22T23:45:08.783 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-22T23:45:08.783 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-22T23:45:08.810 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-22T23:45:08.824 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-22T23:45:08.836 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-22T23:45:08.847 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-05-22T23:45:08.850 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-22T23:45:08.857 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-05-22T23:45:08.868 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-22T23:45:08.870 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-22T23:45:08.875 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-05-22T23:45:08.883 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-22T23:45:08.885 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-05-22T23:45:08.898 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-22T23:45:08.911 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-22T23:45:08.912 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-22T23:45:08.942 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-05-22T23:45:08.946 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-22T23:45:08.946 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-05-22T23:45:08.946 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:08.946 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:08.946 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:08.946 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:08.969 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-05-22T23:45:08.973 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-22T23:45:08.973 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-05-22T23:45:08.973 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:08.973 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:08.973 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:08.974 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:08.984 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-05-22T23:45:09.016 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-05-22T23:45:09.055 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-22T23:45:09.091 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-22T23:45:09.236 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-05-22T23:45:09.245 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-22T23:45:09.245 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-05-22T23:45:09.245 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:09.245 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:09.245 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:09.245 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-22T23:45:09.246 INFO:teuthology.run_tasks:Running task install... 2024-05-22T23:45:09.255 DEBUG:teuthology.task.install:project ceph 2024-05-22T23:45:09.255 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2e1f76222e88afe0f1cda37775b2d45d88289794'}} 2024-05-22T23:45:09.255 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '2e1f76222e88afe0f1cda37775b2d45d88289794'} 2024-05-22T23:45:09.255 INFO:teuthology.task.install:Using flavor: default 2024-05-22T23:45:09.261 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-22T23:45:09.261 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-22T23:45:09.261 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2e1f76222e88afe0f1cda37775b2d45d88289794', 'tag': None, 'wait_for_package': False} 2024-05-22T23:45:09.262 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:45:09.264 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2e1f76222e88afe0f1cda37775b2d45d88289794', 'tag': None, 'wait_for_package': False} 2024-05-22T23:45:09.264 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:45:09.265 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2e1f76222e88afe0f1cda37775b2d45d88289794', 'tag': None, 'wait_for_package': False} 2024-05-22T23:45:09.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:45:09.410 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/ 2024-05-22T23:45:09.410 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2765.g2e1f7622 2024-05-22T23:45:09.491 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/ 2024-05-22T23:45:09.491 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2765.g2e1f7622 2024-05-22T23:45:09.494 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/ 2024-05-22T23:45:09.494 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2765.g2e1f7622 2024-05-22T23:45:09.580 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-22T23:45:09.580 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:45:09.580 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-22T23:45:09.607 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-22T23:45:09.607 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2e1f76222e88afe0f1cda37775b2d45d88289794/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-22T23:45:09.672 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-22T23:45:09.682 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-22T23:45:09.682 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:45:09.683 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-22T23:45:09.703 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-22T23:45:09.703 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:45:09.703 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-22T23:45:09.712 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-22T23:45:09.712 DEBUG:teuthology.orchestra.run.smithi174:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2e1f76222e88afe0f1cda37775b2d45d88289794/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-22T23:45:09.745 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-22T23:45:09.780 DEBUG:teuthology.orchestra.run.smithi174:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-22T23:45:09.792 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-22T23:45:09.793 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2e1f76222e88afe0f1cda37775b2d45d88289794/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-22T23:45:09.812 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-05-22T23:45:09.812 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-05-22T23:45:09.856 DEBUG:teuthology.orchestra.run.smithi174:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-22T23:45:09.920 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-22T23:45:09.920 INFO:teuthology.orchestra.run.smithi174.stdout:check_obsoletes = 1 2024-05-22T23:45:09.921 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum clean all 2024-05-22T23:45:10.035 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-22T23:45:10.035 INFO:teuthology.orchestra.run.smithi105.stdout:45 files removed 2024-05-22T23:45:10.062 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-22T23:45:10.115 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-05-22T23:45:10.119 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-05-22T23:45:10.148 INFO:teuthology.orchestra.run.smithi174.stdout:45 files removed 2024-05-22T23:45:10.175 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-22T23:45:10.449 INFO:teuthology.orchestra.run.smithi103.stdout:45 files removed 2024-05-22T23:45:10.488 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-22T23:45:10.878 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 150 kB/s | 83 kB 00:00 2024-05-22T23:45:10.959 INFO:teuthology.orchestra.run.smithi174.stdout:ceph packages for x86_64 159 kB/s | 83 kB 00:00 2024-05-22T23:45:11.253 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-05-22T23:45:11.331 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 206 kB/s | 83 kB 00:00 2024-05-22T23:45:11.426 INFO:teuthology.orchestra.run.smithi174.stdout:ceph noarch packages 27 kB/s | 11 kB 00:00 2024-05-22T23:45:11.929 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 3.0 kB/s | 1.9 kB 00:00 2024-05-22T23:45:12.160 INFO:teuthology.orchestra.run.smithi174.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-05-22T23:45:12.182 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 15 kB/s | 11 kB 00:00 2024-05-22T23:45:12.924 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-05-22T23:45:13.819 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.1 MB 00:01 2024-05-22T23:45:13.863 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.1 MB 00:01 2024-05-22T23:45:15.720 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-05-22T23:45:17.939 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.1 MB 00:04 2024-05-22T23:45:18.105 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-05-22T23:45:20.128 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.2 MB 00:00 2024-05-22T23:45:21.474 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 19 MB 00:02 2024-05-22T23:45:21.871 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 16 kB 00:00 2024-05-22T23:45:22.190 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-05-22T23:45:23.524 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 81 kB/s | 16 kB 00:00 2024-05-22T23:45:24.335 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 21 MB 00:02 2024-05-22T23:45:25.278 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-22T23:45:26.058 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.2 MB 00:00 2024-05-22T23:45:27.642 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-05-22T23:45:29.531 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-05-22T23:45:29.602 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-22T23:45:30.408 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-05-22T23:45:31.664 INFO:teuthology.orchestra.run.smithi174.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-22T23:45:31.665 INFO:teuthology.orchestra.run.smithi174.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-22T23:45:31.675 INFO:teuthology.orchestra.run.smithi174.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-22T23:45:31.676 INFO:teuthology.orchestra.run.smithi174.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-22T23:45:31.677 INFO:teuthology.orchestra.run.smithi174.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-22T23:45:31.678 INFO:teuthology.orchestra.run.smithi174.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-22T23:45:31.679 INFO:teuthology.orchestra.run.smithi174.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-22T23:45:31.681 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-22T23:45:31.682 INFO:teuthology.orchestra.run.smithi174.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-22T23:45:31.744 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-05-22T23:45:31.763 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: Package Arch Version Repository Size 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout:Installing: 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: ceph x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 6.6 k 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 5.4 M 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 889 k 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 145 k 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 1.5 M 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 142 k 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 3.5 M 2024-05-22T23:45:31.764 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 7.4 M 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 49 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 10 M 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 48 M 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 264 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 763 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 222 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 27 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 18 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 32 k 2024-05-22T23:45:31.765 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 745 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 127 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 162 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 319 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 300 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 99 k 2024-05-22T23:45:31.766 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 85 k 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.1 M 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 171 k 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrading: 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: librados2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.4 M 2024-05-22T23:45:31.767 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.1 M 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout:Installing dependencies: 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 21 M 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 32 k 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 2.4 M 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 249 k 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 4.7 M 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 17 M 2024-05-22T23:45:31.768 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 16 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 25 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 163 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-22T23:45:31.769 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 501 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 5.3 M 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-22T23:45:31.770 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 45 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 135 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-22T23:45:31.771 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-22T23:45:31.772 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-22T23:45:31.773 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-22T23:45:31.774 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-22T23:45:31.775 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout:Installing weak dependencies: 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout:======================================================================================= 2024-05-22T23:45:31.776 INFO:teuthology.orchestra.run.smithi174.stdout:Install 148 Packages 2024-05-22T23:45:31.777 INFO:teuthology.orchestra.run.smithi174.stdout:Upgrade 11 Packages 2024-05-22T23:45:31.777 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:45:31.777 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 225 M 2024-05-22T23:45:31.777 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-05-22T23:45:32.526 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-22T23:45:32.527 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-22T23:45:32.538 INFO:teuthology.orchestra.run.smithi105.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-22T23:45:32.539 INFO:teuthology.orchestra.run.smithi105.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-22T23:45:32.539 INFO:teuthology.orchestra.run.smithi105.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-22T23:45:32.540 INFO:teuthology.orchestra.run.smithi105.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-22T23:45:32.542 INFO:teuthology.orchestra.run.smithi105.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-22T23:45:32.544 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-22T23:45:32.545 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-22T23:45:32.606 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 6.6 k 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 5.4 M 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 889 k 2024-05-22T23:45:32.627 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 145 k 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 1.5 M 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 142 k 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 3.5 M 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 7.4 M 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 49 k 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 10 M 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 48 M 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 264 k 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 763 k 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 222 k 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 27 k 2024-05-22T23:45:32.628 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 18 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 32 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 745 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 127 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-22T23:45:32.629 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 162 k 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 319 k 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 300 k 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 99 k 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 85 k 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.1 M 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 171 k 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-22T23:45:32.630 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.4 M 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.1 M 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-22T23:45:32.631 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 21 M 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 32 k 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 2.4 M 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 249 k 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 4.7 M 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 17 M 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 16 k 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 25 k 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-22T23:45:32.632 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 163 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 501 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 5.3 M 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-22T23:45:32.633 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-22T23:45:32.634 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 45 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 135 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-22T23:45:32.635 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-22T23:45:32.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-22T23:45:32.637 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-22T23:45:32.638 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-22T23:45:32.639 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout:Install 148 Packages 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 11 Packages 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:45:32.640 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 225 M 2024-05-22T23:45:32.641 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-05-22T23:45:32.665 INFO:teuthology.orchestra.run.smithi174.stdout:(1/159): ceph-19.0.0-2765.g2e1f7622.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-05-22T23:45:32.849 INFO:teuthology.orchestra.run.smithi174.stdout:(2/159): ceph-base-19.0.0-2765.g2e1f7622.el9.x8 9.5 MB/s | 5.4 MB 00:00 2024-05-22T23:45:32.916 INFO:teuthology.orchestra.run.smithi174.stdout:(3/159): ceph-immutable-object-cache-19.0.0-276 2.1 MB/s | 145 kB 00:00 2024-05-22T23:45:33.066 INFO:teuthology.orchestra.run.smithi174.stdout:(4/159): ceph-mds-19.0.0-2765.g2e1f7622.el9.x86 16 MB/s | 2.4 MB 00:00 2024-05-22T23:45:33.175 INFO:teuthology.orchestra.run.smithi174.stdout:(5/159): ceph-mgr-19.0.0-2765.g2e1f7622.el9.x86 14 MB/s | 1.5 MB 00:00 2024-05-22T23:45:33.233 INFO:teuthology.orchestra.run.smithi174.stdout:(6/159): ceph-fuse-19.0.0-2765.g2e1f7622.el9.x8 1.5 MB/s | 889 kB 00:00 2024-05-22T23:45:33.418 INFO:teuthology.orchestra.run.smithi105.stdout:(1/159): ceph-19.0.0-2765.g2e1f7622.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-05-22T23:45:33.433 INFO:teuthology.orchestra.run.smithi174.stdout:(7/159): ceph-mon-19.0.0-2765.g2e1f7622.el9.x86 18 MB/s | 4.7 MB 00:00 2024-05-22T23:45:34.068 INFO:teuthology.orchestra.run.smithi105.stdout:(2/159): ceph-fuse-19.0.0-2765.g2e1f7622.el9.x8 1.3 MB/s | 889 kB 00:00 2024-05-22T23:45:34.108 INFO:teuthology.orchestra.run.smithi174.stdout:(8/159): ceph-radosgw-19.0.0-2765.g2e1f7622.el9 15 MB/s | 10 MB 00:00 2024-05-22T23:45:34.135 INFO:teuthology.orchestra.run.smithi105.stdout:(3/159): ceph-immutable-object-cache-19.0.0-276 2.1 MB/s | 145 kB 00:00 2024-05-22T23:45:34.159 INFO:teuthology.orchestra.run.smithi174.stdout:(9/159): ceph-selinux-19.0.0-2765.g2e1f7622.el9 505 kB/s | 25 kB 00:00 2024-05-22T23:45:34.961 INFO:teuthology.orchestra.run.smithi105.stdout:(4/159): ceph-base-19.0.0-2765.g2e1f7622.el9.x8 3.0 MB/s | 5.4 MB 00:01 2024-05-22T23:45:35.036 INFO:teuthology.orchestra.run.smithi105.stdout:(5/159): ceph-mds-19.0.0-2765.g2e1f7622.el9.x86 2.6 MB/s | 2.4 MB 00:00 2024-05-22T23:45:35.470 INFO:teuthology.orchestra.run.smithi105.stdout:(6/159): ceph-mgr-19.0.0-2765.g2e1f7622.el9.x86 2.9 MB/s | 1.5 MB 00:00 2024-05-22T23:45:35.836 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-05-22T23:45:36.170 INFO:teuthology.orchestra.run.smithi105.stdout:(7/159): ceph-mon-19.0.0-2765.g2e1f7622.el9.x86 4.1 MB/s | 4.7 MB 00:01 2024-05-22T23:45:37.592 INFO:teuthology.orchestra.run.smithi174.stdout:(10/159): ceph-test-19.0.0-2765.g2e1f7622.el9.x 14 MB/s | 48 MB 00:03 2024-05-22T23:45:37.659 INFO:teuthology.orchestra.run.smithi174.stdout:(11/159): cephfs-mirror-19.0.0-2765.g2e1f7622.e 3.3 MB/s | 222 kB 00:00 2024-05-22T23:45:37.709 INFO:teuthology.orchestra.run.smithi174.stdout:(12/159): cephfs-shell-19.0.0-2765.g2e1f7622.el 543 kB/s | 27 kB 00:00 2024-05-22T23:45:37.759 INFO:teuthology.orchestra.run.smithi174.stdout:(13/159): libcephfs-devel-19.0.0-2765.g2e1f7622 636 kB/s | 32 kB 00:00 2024-05-22T23:45:37.835 INFO:teuthology.orchestra.run.smithi174.stdout:(14/159): libcephfs2-19.0.0-2765.g2e1f7622.el9. 9.7 MB/s | 745 kB 00:00 2024-05-22T23:45:37.853 INFO:teuthology.orchestra.run.smithi105.stdout:(8/159): ceph-radosgw-19.0.0-2765.g2e1f7622.el9 6.1 MB/s | 10 MB 00:01 2024-05-22T23:45:37.877 INFO:teuthology.orchestra.run.smithi174.stdout:(15/159): libcephsqlite-19.0.0-2765.g2e1f7622.e 3.8 MB/s | 163 kB 00:00 2024-05-22T23:45:37.912 INFO:teuthology.orchestra.run.smithi105.stdout:(9/159): ceph-selinux-19.0.0-2765.g2e1f7622.el9 434 kB/s | 25 kB 00:00 2024-05-22T23:45:37.944 INFO:teuthology.orchestra.run.smithi174.stdout:(16/159): librados-devel-19.0.0-2765.g2e1f7622. 1.9 MB/s | 127 kB 00:00 2024-05-22T23:45:38.028 INFO:teuthology.orchestra.run.smithi174.stdout:(17/159): libradosstriper1-19.0.0-2765.g2e1f762 5.9 MB/s | 501 kB 00:00 2024-05-22T23:45:38.210 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-22T23:45:38.212 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-22T23:45:38.232 INFO:teuthology.orchestra.run.smithi103.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-22T23:45:38.233 INFO:teuthology.orchestra.run.smithi103.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-22T23:45:38.234 INFO:teuthology.orchestra.run.smithi103.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-22T23:45:38.235 INFO:teuthology.orchestra.run.smithi103.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-22T23:45:38.238 INFO:teuthology.orchestra.run.smithi103.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-22T23:45:38.241 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-22T23:45:38.242 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-22T23:45:38.313 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 6.6 k 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 5.4 M 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 889 k 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 145 k 2024-05-22T23:45:38.333 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 1.5 M 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 142 k 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 3.5 M 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 7.4 M 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 49 k 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 10 M 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 48 M 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 264 k 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 763 k 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 222 k 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 27 k 2024-05-22T23:45:38.334 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 18 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 32 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 745 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 127 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-22T23:45:38.335 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 162 k 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 319 k 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 300 k 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 99 k 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 85 k 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.1 M 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 171 k 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-22T23:45:38.336 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.4 M 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 3.1 M 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-22T23:45:38.337 INFO:teuthology.orchestra.run.smithi103.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 21 M 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 32 k 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 2.4 M 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 249 k 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 4.7 M 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 17 M 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2765.g2e1f7622.el9 ceph-noarch 16 k 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 25 k 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-22T23:45:38.338 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 163 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-22T23:45:38.339 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 501 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 5.3 M 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-22T23:45:38.340 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 45 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-2765.g2e1f7622.el9 ceph 135 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-22T23:45:38.341 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-22T23:45:38.342 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-22T23:45:38.343 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-22T23:45:38.344 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-22T23:45:38.345 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout:Install 148 Packages 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 11 Packages 2024-05-22T23:45:38.346 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:45:38.347 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 225 M 2024-05-22T23:45:38.347 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-05-22T23:45:38.486 INFO:teuthology.orchestra.run.smithi174.stdout:(18/159): librgw2-19.0.0-2765.g2e1f7622.el9.x86 12 MB/s | 5.3 MB 00:00 2024-05-22T23:45:38.545 INFO:teuthology.orchestra.run.smithi174.stdout:(19/159): python3-ceph-argparse-19.0.0-2765.g2e 774 kB/s | 45 kB 00:00 2024-05-22T23:45:38.603 INFO:teuthology.orchestra.run.smithi174.stdout:(20/159): python3-ceph-common-19.0.0-2765.g2e1f 2.3 MB/s | 135 kB 00:00 2024-05-22T23:45:38.670 INFO:teuthology.orchestra.run.smithi174.stdout:(21/159): python3-cephfs-19.0.0-2765.g2e1f7622. 2.4 MB/s | 162 kB 00:00 2024-05-22T23:45:38.737 INFO:teuthology.orchestra.run.smithi174.stdout:(22/159): python3-rados-19.0.0-2765.g2e1f7622.e 4.7 MB/s | 319 kB 00:00 2024-05-22T23:45:38.804 INFO:teuthology.orchestra.run.smithi174.stdout:(23/159): python3-rbd-19.0.0-2765.g2e1f7622.el9 4.4 MB/s | 300 kB 00:00 2024-05-22T23:45:38.863 INFO:teuthology.orchestra.run.smithi174.stdout:(24/159): python3-rgw-19.0.0-2765.g2e1f7622.el9 1.7 MB/s | 99 kB 00:00 2024-05-22T23:45:38.922 INFO:teuthology.orchestra.run.smithi174.stdout:(25/159): rbd-fuse-19.0.0-2765.g2e1f7622.el9.x8 1.4 MB/s | 85 kB 00:00 2024-05-22T23:45:39.180 INFO:teuthology.orchestra.run.smithi174.stdout:(26/159): rbd-mirror-19.0.0-2765.g2e1f7622.el9. 12 MB/s | 3.1 MB 00:00 2024-05-22T23:45:39.192 INFO:teuthology.orchestra.run.smithi103.stdout:(1/159): ceph-19.0.0-2765.g2e1f7622.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-05-22T23:45:39.222 INFO:teuthology.orchestra.run.smithi174.stdout:(27/159): rbd-nbd-19.0.0-2765.g2e1f7622.el9.x86 4.0 MB/s | 171 kB 00:00 2024-05-22T23:45:39.370 INFO:teuthology.orchestra.run.smithi105.stdout:(10/159): ceph-osd-19.0.0-2765.g2e1f7622.el9.x8 4.3 MB/s | 17 MB 00:03 2024-05-22T23:45:39.386 INFO:teuthology.orchestra.run.smithi174.stdout:(28/159): ceph-osd-19.0.0-2765.g2e1f7622.el9.x8 2.7 MB/s | 17 MB 00:06 2024-05-22T23:45:39.406 INFO:teuthology.orchestra.run.smithi174.stdout:(29/159): ceph-grafana-dashboards-19.0.0-2765.g 173 kB/s | 32 kB 00:00 2024-05-22T23:45:39.431 INFO:teuthology.orchestra.run.smithi174.stdout:(30/159): ceph-mgr-cephadm-19.0.0-2765.g2e1f762 2.8 MB/s | 142 kB 00:00 2024-05-22T23:45:39.512 INFO:teuthology.orchestra.run.smithi105.stdout:(11/159): cephfs-mirror-19.0.0-2765.g2e1f7622.e 1.5 MB/s | 222 kB 00:00 2024-05-22T23:45:39.596 INFO:teuthology.orchestra.run.smithi105.stdout:(12/159): cephfs-shell-19.0.0-2765.g2e1f7622.el 325 kB/s | 27 kB 00:00 2024-05-22T23:45:39.688 INFO:teuthology.orchestra.run.smithi105.stdout:(13/159): libcephfs-devel-19.0.0-2765.g2e1f7622 347 kB/s | 32 kB 00:00 2024-05-22T23:45:39.707 INFO:teuthology.orchestra.run.smithi174.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2765.g2e1f7 12 MB/s | 3.5 MB 00:00 2024-05-22T23:45:39.773 INFO:teuthology.orchestra.run.smithi174.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2765.g2e 3.7 MB/s | 249 kB 00:00 2024-05-22T23:45:39.840 INFO:teuthology.orchestra.run.smithi174.stdout:(33/159): ceph-mgr-rook-19.0.0-2765.g2e1f7622.e 740 kB/s | 49 kB 00:00 2024-05-22T23:45:39.896 INFO:teuthology.orchestra.run.smithi105.stdout:(14/159): libcephfs2-19.0.0-2765.g2e1f7622.el9. 3.5 MB/s | 745 kB 00:00 2024-05-22T23:45:39.899 INFO:teuthology.orchestra.run.smithi174.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2765.g2 283 kB/s | 16 kB 00:00 2024-05-22T23:45:39.947 INFO:teuthology.orchestra.run.smithi105.stdout:(15/159): libcephsqlite-19.0.0-2765.g2e1f7622.e 3.2 MB/s | 163 kB 00:00 2024-05-22T23:45:39.966 INFO:teuthology.orchestra.run.smithi174.stdout:(35/159): ceph-volume-19.0.0-2765.g2e1f7622.el9 3.9 MB/s | 264 kB 00:00 2024-05-22T23:45:39.996 INFO:teuthology.orchestra.run.smithi103.stdout:(2/159): ceph-fuse-19.0.0-2765.g2e1f7622.el9.x8 1.1 MB/s | 889 kB 00:00 2024-05-22T23:45:40.005 INFO:teuthology.orchestra.run.smithi105.stdout:(16/159): librados-devel-19.0.0-2765.g2e1f7622. 2.1 MB/s | 127 kB 00:00 2024-05-22T23:45:40.050 INFO:teuthology.orchestra.run.smithi174.stdout:(36/159): cephadm-19.0.0-2765.g2e1f7622.el9.noa 8.9 MB/s | 763 kB 00:00 2024-05-22T23:45:40.116 INFO:teuthology.orchestra.run.smithi174.stdout:(37/159): cephfs-top-19.0.0-2765.g2e1f7622.el9. 274 kB/s | 18 kB 00:00 2024-05-22T23:45:40.138 INFO:teuthology.orchestra.run.smithi103.stdout:(3/159): ceph-immutable-object-cache-19.0.0-276 1.0 MB/s | 145 kB 00:00 2024-05-22T23:45:40.164 INFO:teuthology.orchestra.run.smithi105.stdout:(17/159): libradosstriper1-19.0.0-2765.g2e1f762 3.1 MB/s | 501 kB 00:00 2024-05-22T23:45:40.258 INFO:teuthology.orchestra.run.smithi174.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 567 kB/s | 80 kB 00:00 2024-05-22T23:45:40.459 INFO:teuthology.orchestra.run.smithi174.stdout:(39/159): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-05-22T23:45:40.542 INFO:teuthology.orchestra.run.smithi174.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-05-22T23:45:40.634 INFO:teuthology.orchestra.run.smithi174.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-05-22T23:45:40.926 INFO:teuthology.orchestra.run.smithi174.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 4.2 MB/s | 1.2 MB 00:00 2024-05-22T23:45:40.977 INFO:teuthology.orchestra.run.smithi174.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-22T23:45:41.035 INFO:teuthology.orchestra.run.smithi174.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-22T23:45:41.102 INFO:teuthology.orchestra.run.smithi174.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-22T23:45:41.186 INFO:teuthology.orchestra.run.smithi174.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-05-22T23:45:41.213 INFO:teuthology.orchestra.run.smithi103.stdout:(4/159): ceph-mds-19.0.0-2765.g2e1f7622.el9.x86 2.2 MB/s | 2.4 MB 00:01 2024-05-22T23:45:41.278 INFO:teuthology.orchestra.run.smithi174.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 7.4 MB/s | 689 kB 00:00 2024-05-22T23:45:41.378 INFO:teuthology.orchestra.run.smithi174.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 9.2 MB/s | 943 kB 00:00 2024-05-22T23:45:41.445 INFO:teuthology.orchestra.run.smithi174.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-05-22T23:45:41.495 INFO:teuthology.orchestra.run.smithi174.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-05-22T23:45:41.571 INFO:teuthology.orchestra.run.smithi174.stdout:(51/159): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 313 kB 00:00 2024-05-22T23:45:41.621 INFO:teuthology.orchestra.run.smithi174.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 657 kB/s | 33 kB 00:00 2024-05-22T23:45:41.622 INFO:teuthology.orchestra.run.smithi105.stdout:(18/159): librgw2-19.0.0-2765.g2e1f7622.el9.x86 3.6 MB/s | 5.3 MB 00:01 2024-05-22T23:45:41.630 INFO:teuthology.orchestra.run.smithi103.stdout:(5/159): ceph-base-19.0.0-2765.g2e1f7622.el9.x8 1.9 MB/s | 5.4 MB 00:02 2024-05-22T23:45:41.681 INFO:teuthology.orchestra.run.smithi105.stdout:(19/159): python3-ceph-argparse-19.0.0-2765.g2e 775 kB/s | 45 kB 00:00 2024-05-22T23:45:41.739 INFO:teuthology.orchestra.run.smithi103.stdout:(6/159): ceph-mgr-19.0.0-2765.g2e1f7622.el9.x86 2.9 MB/s | 1.5 MB 00:00 2024-05-22T23:45:41.756 INFO:teuthology.orchestra.run.smithi105.stdout:(20/159): python3-ceph-common-19.0.0-2765.g2e1f 1.8 MB/s | 135 kB 00:00 2024-05-22T23:45:41.780 INFO:teuthology.orchestra.run.smithi174.stdout:(53/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-05-22T23:45:41.840 INFO:teuthology.orchestra.run.smithi105.stdout:(21/159): python3-cephfs-19.0.0-2765.g2e1f7622. 1.9 MB/s | 162 kB 00:00 2024-05-22T23:45:41.965 INFO:teuthology.orchestra.run.smithi105.stdout:(22/159): python3-rados-19.0.0-2765.g2e1f7622.e 2.5 MB/s | 319 kB 00:00 2024-05-22T23:45:41.971 INFO:teuthology.orchestra.run.smithi174.stdout:(54/159): ceph-common-19.0.0-2765.g2e1f7622.el9 2.2 MB/s | 21 MB 00:09 2024-05-22T23:45:42.055 INFO:teuthology.orchestra.run.smithi174.stdout:(55/159): ceph-mgr-diskprediction-local-19.0.0- 2.8 MB/s | 7.4 MB 00:02 2024-05-22T23:45:42.080 INFO:teuthology.orchestra.run.smithi174.stdout:(56/159): flexiblas-openblas-openmp-3.0.4-8.el9 61 kB/s | 18 kB 00:00 2024-05-22T23:45:42.106 INFO:teuthology.orchestra.run.smithi174.stdout:(57/159): libaio-devel-0.3.111-13.el9.x86_64.rp 83 kB/s | 11 kB 00:00 2024-05-22T23:45:42.107 INFO:teuthology.orchestra.run.smithi105.stdout:(23/159): python3-rbd-19.0.0-2765.g2e1f7622.el9 2.1 MB/s | 300 kB 00:00 2024-05-22T23:45:42.148 INFO:teuthology.orchestra.run.smithi174.stdout:(58/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-05-22T23:45:42.173 INFO:teuthology.orchestra.run.smithi174.stdout:(59/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 492 kB/s | 45 kB 00:00 2024-05-22T23:45:42.223 INFO:teuthology.orchestra.run.smithi174.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-05-22T23:45:42.248 INFO:teuthology.orchestra.run.smithi174.stdout:(61/159): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-05-22T23:45:42.299 INFO:teuthology.orchestra.run.smithi174.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-05-22T23:45:42.324 INFO:teuthology.orchestra.run.smithi174.stdout:(63/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-05-22T23:45:42.349 INFO:teuthology.orchestra.run.smithi105.stdout:(24/159): ceph-common-19.0.0-2765.g2e1f7622.el9 2.3 MB/s | 21 MB 00:09 2024-05-22T23:45:42.366 INFO:teuthology.orchestra.run.smithi105.stdout:(25/159): python3-rgw-19.0.0-2765.g2e1f7622.el9 385 kB/s | 99 kB 00:00 2024-05-22T23:45:42.374 INFO:teuthology.orchestra.run.smithi174.stdout:(64/159): openblas-0.3.21-2.el9.x86_64.rpm 708 kB/s | 35 kB 00:00 2024-05-22T23:45:42.399 INFO:teuthology.orchestra.run.smithi174.stdout:(65/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-05-22T23:45:42.425 INFO:teuthology.orchestra.run.smithi105.stdout:(26/159): rbd-fuse-19.0.0-2765.g2e1f7622.el9.x8 1.1 MB/s | 85 kB 00:00 2024-05-22T23:45:42.425 INFO:teuthology.orchestra.run.smithi174.stdout:(66/159): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-05-22T23:45:42.558 INFO:teuthology.orchestra.run.smithi174.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-05-22T23:45:42.559 INFO:teuthology.orchestra.run.smithi105.stdout:(27/159): rbd-nbd-19.0.0-2765.g2e1f7622.el9.x86 1.3 MB/s | 171 kB 00:00 2024-05-22T23:45:42.564 INFO:teuthology.orchestra.run.smithi103.stdout:(7/159): ceph-mon-19.0.0-2765.g2e1f7622.el9.x86 5.0 MB/s | 4.7 MB 00:00 2024-05-22T23:45:42.634 INFO:teuthology.orchestra.run.smithi105.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2765.g 422 kB/s | 32 kB 00:00 2024-05-22T23:45:42.650 INFO:teuthology.orchestra.run.smithi174.stdout:(68/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-22T23:45:42.717 INFO:teuthology.orchestra.run.smithi174.stdout:(69/159): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-05-22T23:45:42.726 INFO:teuthology.orchestra.run.smithi105.stdout:(29/159): ceph-mgr-cephadm-19.0.0-2765.g2e1f762 1.5 MB/s | 142 kB 00:00 2024-05-22T23:45:42.760 INFO:teuthology.orchestra.run.smithi174.stdout:(70/159): postgresql-contrib-13.14-1.el9.x86_64 4.4 MB/s | 911 kB 00:00 2024-05-22T23:45:42.801 INFO:teuthology.orchestra.run.smithi174.stdout:(71/159): postgresql-private-libs-13.14-1.el9.x 886 kB/s | 133 kB 00:00 2024-05-22T23:45:42.860 INFO:teuthology.orchestra.run.smithi174.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-05-22T23:45:42.893 INFO:teuthology.orchestra.run.smithi174.stdout:(73/159): protobuf-3.14.0-13.el9.x86_64.rpm 7.5 MB/s | 1.0 MB 00:00 2024-05-22T23:45:42.952 INFO:teuthology.orchestra.run.smithi174.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.0 MB/s | 61 kB 00:00 2024-05-22T23:45:43.077 INFO:teuthology.orchestra.run.smithi174.stdout:(75/159): postgresql-server-13.14-1.el9.x86_64. 16 MB/s | 5.8 MB 00:00 2024-05-22T23:45:43.128 INFO:teuthology.orchestra.run.smithi174.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-05-22T23:45:43.178 INFO:teuthology.orchestra.run.smithi174.stdout:(77/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-05-22T23:45:43.195 INFO:teuthology.orchestra.run.smithi174.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 384 kB/s | 26 kB 00:00 2024-05-22T23:45:43.262 INFO:teuthology.orchestra.run.smithi174.stdout:(79/159): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-05-22T23:45:43.287 INFO:teuthology.orchestra.run.smithi174.stdout:(80/159): python3-jsonpointer-2.0-4.el9.noarch. 176 kB/s | 19 kB 00:00 2024-05-22T23:45:43.329 INFO:teuthology.orchestra.run.smithi174.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 519 kB/s | 35 kB 00:00 2024-05-22T23:45:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:(82/159): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-05-22T23:45:43.367 INFO:teuthology.orchestra.run.smithi105.stdout:(30/159): rbd-mirror-19.0.0-2765.g2e1f7622.el9. 3.1 MB/s | 3.1 MB 00:01 2024-05-22T23:45:43.429 INFO:teuthology.orchestra.run.smithi174.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-05-22T23:45:43.463 INFO:teuthology.orchestra.run.smithi174.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-05-22T23:45:43.530 INFO:teuthology.orchestra.run.smithi174.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 769 kB/s | 77 kB 00:00 2024-05-22T23:45:43.555 INFO:teuthology.orchestra.run.smithi174.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 460 kB/s | 42 kB 00:00 2024-05-22T23:45:43.656 INFO:teuthology.orchestra.run.smithi174.stdout:(87/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-05-22T23:45:43.681 INFO:teuthology.orchestra.run.smithi174.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 1.7 MB/s | 269 kB 00:00 2024-05-22T23:45:43.706 INFO:teuthology.orchestra.run.smithi174.stdout:(89/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-05-22T23:45:43.756 INFO:teuthology.orchestra.run.smithi174.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-05-22T23:45:43.782 INFO:teuthology.orchestra.run.smithi174.stdout:(91/159): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 279 kB 00:00 2024-05-22T23:45:43.807 INFO:teuthology.orchestra.run.smithi174.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 534 kB/s | 54 kB 00:00 2024-05-22T23:45:43.814 INFO:teuthology.orchestra.run.smithi103.stdout:(8/159): ceph-common-19.0.0-2765.g2e1f7622.el9. 4.2 MB/s | 21 MB 00:05 2024-05-22T23:45:43.865 INFO:teuthology.orchestra.run.smithi174.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 762 kB/s | 44 kB 00:00 2024-05-22T23:45:43.891 INFO:teuthology.orchestra.run.smithi174.stdout:(94/159): python3-toml-0.10.2-6.el9.noarch.rpm 500 kB/s | 42 kB 00:00 2024-05-22T23:45:43.893 INFO:teuthology.orchestra.run.smithi105.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2765.g2e1f7 3.0 MB/s | 3.5 MB 00:01 2024-05-22T23:45:43.914 INFO:teuthology.orchestra.run.smithi103.stdout:(9/159): ceph-selinux-19.0.0-2765.g2e1f7622.el9 255 kB/s | 25 kB 00:00 2024-05-22T23:45:43.941 INFO:teuthology.orchestra.run.smithi174.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-05-22T23:45:43.966 INFO:teuthology.orchestra.run.smithi174.stdout:(96/159): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-05-22T23:45:44.018 INFO:teuthology.orchestra.run.smithi105.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2765.g2e 1.9 MB/s | 249 kB 00:00 2024-05-22T23:45:44.033 INFO:teuthology.orchestra.run.smithi174.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 514 kB/s | 47 kB 00:00 2024-05-22T23:45:44.083 INFO:teuthology.orchestra.run.smithi174.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 2.5 MB/s | 305 kB 00:00 2024-05-22T23:45:44.102 INFO:teuthology.orchestra.run.smithi105.stdout:(33/159): ceph-mgr-rook-19.0.0-2765.g2e1f7622.e 592 kB/s | 49 kB 00:00 2024-05-22T23:45:44.109 INFO:teuthology.orchestra.run.smithi174.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 769 kB/s | 58 kB 00:00 2024-05-22T23:45:44.167 INFO:teuthology.orchestra.run.smithi174.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-05-22T23:45:44.185 INFO:teuthology.orchestra.run.smithi105.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2765.g2 198 kB/s | 16 kB 00:00 2024-05-22T23:45:44.193 INFO:teuthology.orchestra.run.smithi174.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 763 kB/s | 64 kB 00:00 2024-05-22T23:45:44.251 INFO:teuthology.orchestra.run.smithi174.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 535 kB/s | 45 kB 00:00 2024-05-22T23:45:44.352 INFO:teuthology.orchestra.run.smithi105.stdout:(35/159): ceph-volume-19.0.0-2765.g2e1f7622.el9 1.5 MB/s | 264 kB 00:00 2024-05-22T23:45:44.402 INFO:teuthology.orchestra.run.smithi174.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-05-22T23:45:44.490 INFO:teuthology.orchestra.run.smithi103.stdout:(10/159): ceph-radosgw-19.0.0-2765.g2e1f7622.el 5.3 MB/s | 10 MB 00:01 2024-05-22T23:45:44.578 INFO:teuthology.orchestra.run.smithi105.stdout:(36/159): cephadm-19.0.0-2765.g2e1f7622.el9.noa 3.3 MB/s | 763 kB 00:00 2024-05-22T23:45:44.619 INFO:teuthology.orchestra.run.smithi105.stdout:(37/159): cephfs-top-19.0.0-2765.g2e1f7622.el9. 440 kB/s | 18 kB 00:00 2024-05-22T23:45:44.728 INFO:teuthology.orchestra.run.smithi105.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 742 kB/s | 80 kB 00:00 2024-05-22T23:45:44.748 INFO:teuthology.orchestra.run.smithi103.stdout:(11/159): ceph-osd-19.0.0-2765.g2e1f7622.el9.x8 5.6 MB/s | 17 MB 00:03 2024-05-22T23:45:44.782 INFO:teuthology.orchestra.run.smithi103.stdout:(12/159): cephfs-mirror-19.0.0-2765.g2e1f7622.e 763 kB/s | 222 kB 00:00 2024-05-22T23:45:44.837 INFO:teuthology.orchestra.run.smithi105.stdout:(39/159): libgfortran-11.4.1-3.el9.x86_64.rpm 7.2 MB/s | 803 kB 00:00 2024-05-22T23:45:44.865 INFO:teuthology.orchestra.run.smithi103.stdout:(13/159): cephfs-shell-19.0.0-2765.g2e1f7622.el 234 kB/s | 27 kB 00:00 2024-05-22T23:45:44.887 INFO:teuthology.orchestra.run.smithi105.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-05-22T23:45:44.916 INFO:teuthology.orchestra.run.smithi103.stdout:(14/159): libcephfs-devel-19.0.0-2765.g2e1f7622 239 kB/s | 32 kB 00:00 2024-05-22T23:45:44.918 INFO:teuthology.orchestra.run.smithi174.stdout:(104/159): lua-devel-5.4.4-4.el9.x86_64.rpm 31 kB/s | 22 kB 00:00 2024-05-22T23:45:44.938 INFO:teuthology.orchestra.run.smithi105.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-05-22T23:45:45.000 INFO:teuthology.orchestra.run.smithi103.stdout:(15/159): libcephsqlite-19.0.0-2765.g2e1f7622.e 1.9 MB/s | 163 kB 00:00 2024-05-22T23:45:45.021 INFO:teuthology.orchestra.run.smithi105.stdout:(42/159): ceph-mgr-diskprediction-local-19.0.0- 4.5 MB/s | 7.4 MB 00:01 2024-05-22T23:45:45.063 INFO:teuthology.orchestra.run.smithi105.stdout:(43/159): python3-cryptography-36.0.1-4.el9.x86 9.8 MB/s | 1.2 MB 00:00 2024-05-22T23:45:45.074 INFO:teuthology.orchestra.run.smithi103.stdout:(16/159): libcephfs2-19.0.0-2765.g2e1f7622.el9. 3.5 MB/s | 745 kB 00:00 2024-05-22T23:45:45.105 INFO:teuthology.orchestra.run.smithi105.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-05-22T23:45:45.110 INFO:teuthology.orchestra.run.smithi174.stdout:(105/159): abseil-cpp-20211102.0-3.el9.x86_64.r 773 kB/s | 548 kB 00:00 2024-05-22T23:45:45.117 INFO:teuthology.orchestra.run.smithi103.stdout:(17/159): librados-devel-19.0.0-2765.g2e1f7622. 1.1 MB/s | 127 kB 00:00 2024-05-22T23:45:45.155 INFO:teuthology.orchestra.run.smithi105.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-22T23:45:45.175 INFO:teuthology.orchestra.run.smithi103.stdout:(18/159): libradosstriper1-19.0.0-2765.g2e1f762 4.9 MB/s | 501 kB 00:00 2024-05-22T23:45:45.181 INFO:teuthology.orchestra.run.smithi105.stdout:(46/159): python3-ply-3.11-14.el9.noarch.rpm 670 kB/s | 106 kB 00:00 2024-05-22T23:45:45.202 INFO:teuthology.orchestra.run.smithi174.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-05-22T23:45:45.206 INFO:teuthology.orchestra.run.smithi105.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-22T23:45:45.228 INFO:teuthology.orchestra.run.smithi174.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-05-22T23:45:45.269 INFO:teuthology.orchestra.run.smithi174.stdout:(108/159): protobuf-compiler-3.14.0-13.el9.x86_ 848 kB/s | 863 kB 00:01 2024-05-22T23:45:45.276 INFO:teuthology.orchestra.run.smithi103.stdout:(19/159): python3-ceph-argparse-19.0.0-2765.g2e 456 kB/s | 45 kB 00:00 2024-05-22T23:45:45.345 INFO:teuthology.orchestra.run.smithi174.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-05-22T23:45:45.385 INFO:teuthology.orchestra.run.smithi103.stdout:(20/159): python3-ceph-common-19.0.0-2765.g2e1f 1.2 MB/s | 135 kB 00:00 2024-05-22T23:45:45.456 INFO:teuthology.orchestra.run.smithi105.stdout:(48/159): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 689 kB 00:00 2024-05-22T23:45:45.478 INFO:teuthology.orchestra.run.smithi174.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-05-22T23:45:45.485 INFO:teuthology.orchestra.run.smithi103.stdout:(21/159): python3-cephfs-19.0.0-2765.g2e1f7622. 1.6 MB/s | 162 kB 00:00 2024-05-22T23:45:45.506 INFO:teuthology.orchestra.run.smithi105.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-05-22T23:45:45.532 INFO:teuthology.orchestra.run.smithi105.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 107 kB 00:00 2024-05-22T23:45:45.557 INFO:teuthology.orchestra.run.smithi105.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 432 kB/s | 22 kB 00:00 2024-05-22T23:45:45.585 INFO:teuthology.orchestra.run.smithi103.stdout:(22/159): python3-rados-19.0.0-2765.g2e1f7622.e 3.2 MB/s | 319 kB 00:00 2024-05-22T23:45:45.607 INFO:teuthology.orchestra.run.smithi105.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 654 kB/s | 33 kB 00:00 2024-05-22T23:45:45.610 INFO:teuthology.orchestra.run.smithi174.stdout:(111/159): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-05-22T23:45:45.629 INFO:teuthology.orchestra.run.smithi174.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 237 kB/s | 67 kB 00:00 2024-05-22T23:45:45.654 INFO:teuthology.orchestra.run.smithi174.stdout:(113/159): luarocks-3.9.2-5.el9.noarch.rpm 873 kB/s | 151 kB 00:00 2024-05-22T23:45:45.657 INFO:teuthology.orchestra.run.smithi105.stdout:(53/159): flex-2.6.4-9.el9.x86_64.rpm 2.4 MB/s | 313 kB 00:00 2024-05-22T23:45:45.677 INFO:teuthology.orchestra.run.smithi103.stdout:(23/159): python3-rbd-19.0.0-2765.g2e1f7622.el9 3.2 MB/s | 300 kB 00:00 2024-05-22T23:45:45.708 INFO:teuthology.orchestra.run.smithi105.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 364 kB/s | 18 kB 00:00 2024-05-22T23:45:45.755 INFO:teuthology.orchestra.run.smithi174.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 6.6 MB/s | 839 kB 00:00 2024-05-22T23:45:45.758 INFO:teuthology.orchestra.run.smithi105.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 224 kB/s | 11 kB 00:00 2024-05-22T23:45:45.769 INFO:teuthology.orchestra.run.smithi103.stdout:(24/159): python3-rgw-19.0.0-2765.g2e1f7622.el9 1.1 MB/s | 99 kB 00:00 2024-05-22T23:45:45.780 INFO:teuthology.orchestra.run.smithi174.stdout:(115/159): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-05-22T23:45:45.805 INFO:teuthology.orchestra.run.smithi174.stdout:(116/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 288 kB/s | 43 kB 00:00 2024-05-22T23:45:45.850 INFO:teuthology.orchestra.run.smithi105.stdout:(56/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-05-22T23:45:45.855 INFO:teuthology.orchestra.run.smithi174.stdout:(117/159): python3-cachetools-4.2.4-1.el9.noarc 324 kB/s | 32 kB 00:00 2024-05-22T23:45:45.861 INFO:teuthology.orchestra.run.smithi103.stdout:(25/159): rbd-fuse-19.0.0-2765.g2e1f7622.el9.x8 936 kB/s | 85 kB 00:00 2024-05-22T23:45:45.905 INFO:teuthology.orchestra.run.smithi174.stdout:(118/159): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-05-22T23:45:45.909 INFO:teuthology.orchestra.run.smithi105.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 779 kB/s | 45 kB 00:00 2024-05-22T23:45:45.931 INFO:teuthology.orchestra.run.smithi174.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-05-22T23:45:45.970 INFO:teuthology.orchestra.run.smithi103.stdout:(26/159): librgw2-19.0.0-2765.g2e1f7622.el9.x86 6.2 MB/s | 5.3 MB 00:00 2024-05-22T23:45:45.973 INFO:teuthology.orchestra.run.smithi174.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-05-22T23:45:46.015 INFO:teuthology.orchestra.run.smithi174.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-05-22T23:45:46.040 INFO:teuthology.orchestra.run.smithi174.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 1.8 MB/s | 206 kB 00:00 2024-05-22T23:45:46.054 INFO:teuthology.orchestra.run.smithi103.stdout:(27/159): rbd-nbd-19.0.0-2765.g2e1f7622.el9.x86 2.0 MB/s | 171 kB 00:00 2024-05-22T23:45:46.149 INFO:teuthology.orchestra.run.smithi174.stdout:(123/159): python3-grpcio-1.46.7-10.el9.x86_64. 12 MB/s | 2.0 MB 00:00 2024-05-22T23:45:46.162 INFO:teuthology.orchestra.run.smithi103.stdout:(28/159): ceph-grafana-dashboards-19.0.0-2765.g 295 kB/s | 32 kB 00:00 2024-05-22T23:45:46.174 INFO:teuthology.orchestra.run.smithi174.stdout:(124/159): python3-grpcio-tools-1.46.7-10.el9.x 912 kB/s | 144 kB 00:00 2024-05-22T23:45:46.199 INFO:teuthology.orchestra.run.smithi174.stdout:(125/159): python3-jaraco-8.2.1-3.el9.noarch.rp 67 kB/s | 11 kB 00:00 2024-05-22T23:45:46.258 INFO:teuthology.orchestra.run.smithi174.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-05-22T23:45:46.296 INFO:teuthology.orchestra.run.smithi103.stdout:(29/159): rbd-mirror-19.0.0-2765.g2e1f7622.el9. 7.1 MB/s | 3.1 MB 00:00 2024-05-22T23:45:46.316 INFO:teuthology.orchestra.run.smithi174.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 199 kB/s | 23 kB 00:00 2024-05-22T23:45:46.321 INFO:teuthology.orchestra.run.smithi103.stdout:(30/159): ceph-mgr-cephadm-19.0.0-2765.g2e1f762 901 kB/s | 142 kB 00:00 2024-05-22T23:45:46.341 INFO:teuthology.orchestra.run.smithi174.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 137 kB/s | 19 kB 00:00 2024-05-22T23:45:46.367 INFO:teuthology.orchestra.run.smithi174.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-05-22T23:45:46.376 INFO:teuthology.orchestra.run.smithi105.stdout:(58/159): ceph-test-19.0.0-2765.g2e1f7622.el9.x 5.7 MB/s | 48 MB 00:08 2024-05-22T23:45:46.410 INFO:teuthology.orchestra.run.smithi105.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 1.3 MB/s | 662 kB 00:00 2024-05-22T23:45:46.450 INFO:teuthology.orchestra.run.smithi174.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-05-22T23:45:46.460 INFO:teuthology.orchestra.run.smithi105.stdout:(60/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.5 MB/s | 3.0 MB 00:00 2024-05-22T23:45:46.475 INFO:teuthology.orchestra.run.smithi174.stdout:(131/159): python3-logutils-0.3.5-21.el9.noarch 346 kB/s | 46 kB 00:00 2024-05-22T23:45:46.485 INFO:teuthology.orchestra.run.smithi105.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 339 kB/s | 25 kB 00:00 2024-05-22T23:45:46.501 INFO:teuthology.orchestra.run.smithi174.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 590 kB/s | 79 kB 00:00 2024-05-22T23:45:46.510 INFO:teuthology.orchestra.run.smithi105.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-05-22T23:45:46.559 INFO:teuthology.orchestra.run.smithi174.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 533 kB/s | 58 kB 00:00 2024-05-22T23:45:46.569 INFO:teuthology.orchestra.run.smithi105.stdout:(63/159): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-05-22T23:45:46.594 INFO:teuthology.orchestra.run.smithi105.stdout:(64/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-05-22T23:45:46.609 INFO:teuthology.orchestra.run.smithi174.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-05-22T23:45:46.620 INFO:teuthology.orchestra.run.smithi105.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 699 kB/s | 35 kB 00:00 2024-05-22T23:45:46.635 INFO:teuthology.orchestra.run.smithi174.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 123 kB/s | 16 kB 00:00 2024-05-22T23:45:46.668 INFO:teuthology.orchestra.run.smithi174.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 825 kB/s | 90 kB 00:00 2024-05-22T23:45:46.670 INFO:teuthology.orchestra.run.smithi105.stdout:(66/159): libtool-2.4.6-45.el9.x86_64.rpm 1.9 MB/s | 582 kB 00:00 2024-05-22T23:45:46.719 INFO:teuthology.orchestra.run.smithi174.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-05-22T23:45:46.744 INFO:teuthology.orchestra.run.smithi174.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-05-22T23:45:46.778 INFO:teuthology.orchestra.run.smithi174.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-05-22T23:45:46.797 INFO:teuthology.orchestra.run.smithi103.stdout:(31/159): ceph-mgr-dashboard-19.0.0-2765.g2e1f7 7.0 MB/s | 3.5 MB 00:00 2024-05-22T23:45:46.828 INFO:teuthology.orchestra.run.smithi174.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-05-22T23:45:46.854 INFO:teuthology.orchestra.run.smithi174.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-05-22T23:45:46.872 INFO:teuthology.orchestra.run.smithi103.stdout:(32/159): ceph-mgr-modules-core-19.0.0-2765.g2e 3.3 MB/s | 249 kB 00:00 2024-05-22T23:45:46.887 INFO:teuthology.orchestra.run.smithi174.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-05-22T23:45:46.945 INFO:teuthology.orchestra.run.smithi174.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-05-22T23:45:46.971 INFO:teuthology.orchestra.run.smithi174.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 765 kB/s | 90 kB 00:00 2024-05-22T23:45:46.989 INFO:teuthology.orchestra.run.smithi103.stdout:(33/159): ceph-mgr-rook-19.0.0-2765.g2e1f7622.e 426 kB/s | 49 kB 00:00 2024-05-22T23:45:47.003 INFO:teuthology.orchestra.run.smithi105.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 4.8 MB/s | 1.6 MB 00:00 2024-05-22T23:45:47.004 INFO:teuthology.orchestra.run.smithi174.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-05-22T23:45:47.055 INFO:teuthology.orchestra.run.smithi174.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-05-22T23:45:47.096 INFO:teuthology.orchestra.run.smithi174.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-05-22T23:45:47.106 INFO:teuthology.orchestra.run.smithi103.stdout:(34/159): ceph-prometheus-alerts-19.0.0-2765.g2 142 kB/s | 16 kB 00:00 2024-05-22T23:45:47.180 INFO:teuthology.orchestra.run.smithi174.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-05-22T23:45:47.320 INFO:teuthology.orchestra.run.smithi105.stdout:(68/159): postgresql-contrib-13.14-1.el9.x86_64 2.8 MB/s | 911 kB 00:00 2024-05-22T23:45:47.348 INFO:teuthology.orchestra.run.smithi103.stdout:(35/159): ceph-mgr-diskprediction-local-19.0.0- 7.2 MB/s | 7.4 MB 00:01 2024-05-22T23:45:47.373 INFO:teuthology.orchestra.run.smithi103.stdout:(36/159): ceph-volume-19.0.0-2765.g2e1f7622.el9 992 kB/s | 264 kB 00:00 2024-05-22T23:45:47.387 INFO:teuthology.orchestra.run.smithi105.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-05-22T23:45:47.454 INFO:teuthology.orchestra.run.smithi105.stdout:(70/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 5.3 MB/s | 4.4 MB 00:00 2024-05-22T23:45:47.464 INFO:teuthology.orchestra.run.smithi174.stdout:(149/159): librbd1-19.0.0-2765.g2e1f7622.el9.x8 8.5 MB/s | 3.1 MB 00:00 2024-05-22T23:45:47.490 INFO:teuthology.orchestra.run.smithi103.stdout:(37/159): cephfs-top-19.0.0-2765.g2e1f7622.el9. 157 kB/s | 18 kB 00:00 2024-05-22T23:45:47.540 INFO:teuthology.orchestra.run.smithi103.stdout:(38/159): cephadm-19.0.0-2765.g2e1f7622.el9.noa 3.9 MB/s | 763 kB 00:00 2024-05-22T23:45:47.591 INFO:teuthology.orchestra.run.smithi103.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 810 kB/s | 80 kB 00:00 2024-05-22T23:45:47.663 INFO:teuthology.orchestra.run.smithi105.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-05-22T23:45:47.666 INFO:teuthology.orchestra.run.smithi103.stdout:(40/159): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-05-22T23:45:47.700 INFO:teuthology.orchestra.run.smithi103.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-05-22T23:45:47.731 INFO:teuthology.orchestra.run.smithi174.stdout:(150/159): NetworkManager-1.47.90-1.el9.x86_64. 4.2 MB/s | 2.3 MB 00:00 2024-05-22T23:45:47.738 INFO:teuthology.orchestra.run.smithi105.stdout:(72/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.1 MB/s | 87 kB 00:00 2024-05-22T23:45:47.767 INFO:teuthology.orchestra.run.smithi103.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-05-22T23:45:47.798 INFO:teuthology.orchestra.run.smithi174.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 624 kB/s | 41 kB 00:00 2024-05-22T23:45:47.830 INFO:teuthology.orchestra.run.smithi105.stdout:(73/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.8 MB/s | 4.7 MB 00:01 2024-05-22T23:45:47.842 INFO:teuthology.orchestra.run.smithi103.stdout:(43/159): python3-cryptography-36.0.1-4.el9.x86 8.7 MB/s | 1.2 MB 00:00 2024-05-22T23:45:47.864 INFO:teuthology.orchestra.run.smithi174.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 3.6 MB/s | 248 kB 00:00 2024-05-22T23:45:47.868 INFO:teuthology.orchestra.run.smithi103.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-05-22T23:45:47.872 INFO:teuthology.orchestra.run.smithi105.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.4 MB/s | 61 kB 00:00 2024-05-22T23:45:47.915 INFO:teuthology.orchestra.run.smithi174.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 735 kB/s | 37 kB 00:00 2024-05-22T23:45:47.926 INFO:teuthology.orchestra.run.smithi103.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 1.6 MB/s | 135 kB 00:00 2024-05-22T23:45:47.956 INFO:teuthology.orchestra.run.smithi105.stdout:(75/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-05-22T23:45:47.960 INFO:teuthology.orchestra.run.smithi103.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-05-22T23:45:47.973 INFO:teuthology.orchestra.run.smithi174.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 3.5 MB/s | 207 kB 00:00 2024-05-22T23:45:48.039 INFO:teuthology.orchestra.run.smithi105.stdout:(76/159): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-05-22T23:45:48.057 INFO:teuthology.orchestra.run.smithi174.stdout:(155/159): NetworkManager-libnm-1.47.90-1.el9.x 3.0 MB/s | 1.8 MB 00:00 2024-05-22T23:45:48.082 INFO:teuthology.orchestra.run.smithi174.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.3 MB/s | 260 kB 00:00 2024-05-22T23:45:48.089 INFO:teuthology.orchestra.run.smithi105.stdout:(77/159): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-05-22T23:45:48.148 INFO:teuthology.orchestra.run.smithi105.stdout:(78/159): python3-jsonpointer-2.0-4.el9.noarch. 328 kB/s | 19 kB 00:00 2024-05-22T23:45:48.182 INFO:teuthology.orchestra.run.smithi174.stdout:(157/159): librados2-19.0.0-2765.g2e1f7622.el9. 3.0 MB/s | 3.4 MB 00:01 2024-05-22T23:45:48.215 INFO:teuthology.orchestra.run.smithi105.stdout:(79/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-05-22T23:45:48.274 INFO:teuthology.orchestra.run.smithi105.stdout:(80/159): python3-markupsafe-1.1.1-12.el9.x86_6 595 kB/s | 35 kB 00:00 2024-05-22T23:45:48.308 INFO:teuthology.orchestra.run.smithi174.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 5.4 MB/s | 1.3 MB 00:00 2024-05-22T23:45:48.458 INFO:teuthology.orchestra.run.smithi105.stdout:(81/159): python3-babel-2.9.1-2.el9.noarch.rpm 8.3 MB/s | 6.0 MB 00:00 2024-05-22T23:45:48.510 INFO:teuthology.orchestra.run.smithi103.stdout:(47/159): ceph-test-19.0.0-2765.g2e1f7622.el9.x 11 MB/s | 48 MB 00:04 2024-05-22T23:45:48.516 INFO:teuthology.orchestra.run.smithi105.stdout:(82/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-05-22T23:45:48.561 INFO:teuthology.orchestra.run.smithi103.stdout:(48/159): python3-urllib3-1.26.5-5.el9.noarch.r 339 kB/s | 215 kB 00:00 2024-05-22T23:45:48.575 INFO:teuthology.orchestra.run.smithi105.stdout:(83/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-05-22T23:45:48.583 INFO:teuthology.orchestra.run.smithi174.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.8 MB/s | 2.4 MB 00:00 2024-05-22T23:45:48.587 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-05-22T23:45:48.587 INFO:teuthology.orchestra.run.smithi174.stdout:Total 13 MB/s | 225 MB 00:16 2024-05-22T23:45:48.667 INFO:teuthology.orchestra.run.smithi105.stdout:(84/159): python3-packaging-20.9-5.el9.noarch.r 839 kB/s | 77 kB 00:00 2024-05-22T23:45:48.717 INFO:teuthology.orchestra.run.smithi105.stdout:(85/159): python3-prettytable-0.7.2-27.el9.noar 849 kB/s | 42 kB 00:00 2024-05-22T23:45:48.778 INFO:teuthology.orchestra.run.smithi103.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 495 kB/s | 107 kB 00:00 2024-05-22T23:45:48.811 INFO:teuthology.orchestra.run.smithi103.stdout:(50/159): automake-1.16.2-8.el9.noarch.rpm 810 kB/s | 689 kB 00:00 2024-05-22T23:45:48.826 INFO:teuthology.orchestra.run.smithi105.stdout:(86/159): postgresql-server-13.14-1.el9.x86_64. 4.0 MB/s | 5.8 MB 00:01 2024-05-22T23:45:48.860 INFO:teuthology.orchestra.run.smithi105.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 1.9 MB/s | 269 kB 00:00 2024-05-22T23:45:48.861 INFO:teuthology.orchestra.run.smithi103.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 262 kB/s | 22 kB 00:00 2024-05-22T23:45:48.885 INFO:teuthology.orchestra.run.smithi105.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-22T23:45:48.911 INFO:teuthology.orchestra.run.smithi103.stdout:(52/159): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 943 kB 00:00 2024-05-22T23:45:48.935 INFO:teuthology.orchestra.run.smithi105.stdout:(89/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-22T23:45:48.937 INFO:teuthology.orchestra.run.smithi103.stdout:(53/159): flex-2.6.4-9.el9.x86_64.rpm 2.5 MB/s | 313 kB 00:00 2024-05-22T23:45:48.977 INFO:teuthology.orchestra.run.smithi105.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-05-22T23:45:48.979 INFO:teuthology.orchestra.run.smithi103.stdout:(54/159): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 33 kB 00:00 2024-05-22T23:45:49.002 INFO:teuthology.orchestra.run.smithi105.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 805 kB/s | 54 kB 00:00 2024-05-22T23:45:49.053 INFO:teuthology.orchestra.run.smithi105.stdout:(92/159): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-05-22T23:45:49.063 INFO:teuthology.orchestra.run.smithi103.stdout:(55/159): libaio-devel-0.3.111-13.el9.x86_64.rp 136 kB/s | 11 kB 00:00 2024-05-22T23:45:49.103 INFO:teuthology.orchestra.run.smithi105.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 886 kB/s | 44 kB 00:00 2024-05-22T23:45:49.104 INFO:teuthology.orchestra.run.smithi103.stdout:(56/159): flexiblas-openblas-openmp-3.0.4-8.el9 146 kB/s | 18 kB 00:00 2024-05-22T23:45:49.146 INFO:teuthology.orchestra.run.smithi103.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-05-22T23:45:49.161 INFO:teuthology.orchestra.run.smithi105.stdout:(94/159): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 221 kB 00:00 2024-05-22T23:45:49.212 INFO:teuthology.orchestra.run.smithi105.stdout:(95/159): qatlib-service-24.02.0-1.el9.x86_64.r 719 kB/s | 36 kB 00:00 2024-05-22T23:45:49.213 INFO:teuthology.orchestra.run.smithi103.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 420 kB/s | 45 kB 00:00 2024-05-22T23:45:49.262 INFO:teuthology.orchestra.run.smithi105.stdout:(96/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-05-22T23:45:49.264 INFO:teuthology.orchestra.run.smithi103.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 5.6 MB/s | 662 kB 00:00 2024-05-22T23:45:49.321 INFO:teuthology.orchestra.run.smithi105.stdout:(97/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-05-22T23:45:49.356 INFO:teuthology.orchestra.run.smithi103.stdout:(60/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.7 MB/s | 3.0 MB 00:00 2024-05-22T23:45:49.371 INFO:teuthology.orchestra.run.smithi105.stdout:(98/159): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-22T23:45:49.406 INFO:teuthology.orchestra.run.smithi103.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 180 kB/s | 25 kB 00:00 2024-05-22T23:45:49.422 INFO:teuthology.orchestra.run.smithi105.stdout:(99/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-05-22T23:45:49.457 INFO:teuthology.orchestra.run.smithi103.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 477 kB/s | 47 kB 00:00 2024-05-22T23:45:49.472 INFO:teuthology.orchestra.run.smithi105.stdout:(100/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-22T23:45:49.490 INFO:teuthology.orchestra.run.smithi103.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.5 MB/s | 292 kB 00:00 2024-05-22T23:45:49.532 INFO:teuthology.orchestra.run.smithi103.stdout:(64/159): libtool-2.4.6-45.el9.x86_64.rpm 1.8 MB/s | 582 kB 00:00 2024-05-22T23:45:49.539 INFO:teuthology.orchestra.run.smithi105.stdout:(101/159): python3-numpy-1.20.1-5.el9.x86_64.rp 4.0 MB/s | 5.1 MB 00:01 2024-05-22T23:45:49.564 INFO:teuthology.orchestra.run.smithi105.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 487 kB/s | 45 kB 00:00 2024-05-22T23:45:49.582 INFO:teuthology.orchestra.run.smithi103.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 386 kB/s | 35 kB 00:00 2024-05-22T23:45:49.616 INFO:teuthology.orchestra.run.smithi103.stdout:(66/159): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-05-22T23:45:49.742 INFO:teuthology.orchestra.run.smithi103.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-22T23:45:49.807 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-05-22T23:45:49.839 INFO:teuthology.orchestra.run.smithi105.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 74 kB/s | 22 kB 00:00 2024-05-22T23:45:49.850 INFO:teuthology.orchestra.run.smithi103.stdout:(68/159): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-05-22T23:45:49.951 INFO:teuthology.orchestra.run.smithi103.stdout:(69/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-05-22T23:45:49.966 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-05-22T23:45:49.966 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-05-22T23:45:49.984 INFO:teuthology.orchestra.run.smithi103.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-05-22T23:45:50.043 INFO:teuthology.orchestra.run.smithi103.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 3.0 MB/s | 911 kB 00:00 2024-05-22T23:45:50.110 INFO:teuthology.orchestra.run.smithi103.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 8.1 MB/s | 1.0 MB 00:00 2024-05-22T23:45:50.135 INFO:teuthology.orchestra.run.smithi103.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 960 kB/s | 87 kB 00:00 2024-05-22T23:45:50.219 INFO:teuthology.orchestra.run.smithi103.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 742 kB/s | 61 kB 00:00 2024-05-22T23:45:50.256 INFO:teuthology.orchestra.run.smithi105.stdout:(104/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-05-22T23:45:50.323 INFO:teuthology.orchestra.run.smithi105.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 1.1 MB/s | 863 kB 00:00 2024-05-22T23:45:50.349 INFO:teuthology.orchestra.run.smithi105.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 3.3 MB/s | 309 kB 00:00 2024-05-22T23:45:50.353 INFO:teuthology.orchestra.run.smithi103.stdout:(75/159): postgresql-server-13.14-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-05-22T23:45:50.386 INFO:teuthology.orchestra.run.smithi103.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 1.4 MB/s | 244 kB 00:00 2024-05-22T23:45:50.428 INFO:teuthology.orchestra.run.smithi103.stdout:(77/159): python3-jinja2-2.11.3-6.el9.noarch.rp 3.3 MB/s | 248 kB 00:00 2024-05-22T23:45:50.470 INFO:teuthology.orchestra.run.smithi103.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 309 kB/s | 26 kB 00:00 2024-05-22T23:45:50.512 INFO:teuthology.orchestra.run.smithi103.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 230 kB/s | 19 kB 00:00 2024-05-22T23:45:50.546 INFO:teuthology.orchestra.run.smithi103.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.3 MB/s | 172 kB 00:00 2024-05-22T23:45:50.549 INFO:teuthology.orchestra.run.smithi105.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-05-22T23:45:50.616 INFO:teuthology.orchestra.run.smithi105.stdout:(108/159): libarrow-9.0.0-11.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-05-22T23:45:50.621 INFO:teuthology.orchestra.run.smithi103.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 322 kB/s | 35 kB 00:00 2024-05-22T23:45:50.632 INFO:teuthology.orchestra.run.smithi105.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-05-22T23:45:50.675 INFO:teuthology.orchestra.run.smithi105.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 829 kB/s | 49 kB 00:00 2024-05-22T23:45:50.750 INFO:teuthology.orchestra.run.smithi105.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-05-22T23:45:50.755 INFO:teuthology.orchestra.run.smithi103.stdout:(82/159): python3-babel-2.9.1-2.el9.noarch.rpm 9.3 MB/s | 6.0 MB 00:00 2024-05-22T23:45:50.775 INFO:teuthology.orchestra.run.smithi105.stdout:(112/159): libunwind-1.6.2-1.el9.x86_64.rpm 472 kB/s | 67 kB 00:00 2024-05-22T23:45:50.826 INFO:teuthology.orchestra.run.smithi105.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-05-22T23:45:50.830 INFO:teuthology.orchestra.run.smithi103.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 250 kB 00:00 2024-05-22T23:45:50.864 INFO:teuthology.orchestra.run.smithi103.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 2.0 MB/s | 222 kB 00:00 2024-05-22T23:45:50.893 INFO:teuthology.orchestra.run.smithi105.stdout:(114/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 653 kB/s | 43 kB 00:00 2024-05-22T23:45:50.898 INFO:teuthology.orchestra.run.smithi103.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-05-22T23:45:50.948 INFO:teuthology.orchestra.run.smithi103.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 509 kB/s | 42 kB 00:00 2024-05-22T23:45:50.959 INFO:teuthology.orchestra.run.smithi105.stdout:(115/159): python3-cachetools-4.2.4-1.el9.noarc 484 kB/s | 32 kB 00:00 2024-05-22T23:45:50.990 INFO:teuthology.orchestra.run.smithi103.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-05-22T23:45:50.993 INFO:teuthology.orchestra.run.smithi105.stdout:(116/159): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-05-22T23:45:51.023 INFO:teuthology.orchestra.run.smithi103.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.1 MB/s | 159 kB 00:00 2024-05-22T23:45:51.027 INFO:teuthology.orchestra.run.smithi105.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-05-22T23:45:51.065 INFO:teuthology.orchestra.run.smithi103.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 3.7 MB/s | 279 kB 00:00 2024-05-22T23:45:51.069 INFO:teuthology.orchestra.run.smithi105.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-05-22T23:45:51.103 INFO:teuthology.orchestra.run.smithi105.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-05-22T23:45:51.108 INFO:teuthology.orchestra.run.smithi103.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 612 kB/s | 51 kB 00:00 2024-05-22T23:45:51.133 INFO:teuthology.orchestra.run.smithi103.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 805 kB/s | 54 kB 00:00 2024-05-22T23:45:51.153 INFO:teuthology.orchestra.run.smithi105.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-05-22T23:45:51.178 INFO:teuthology.orchestra.run.smithi105.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-05-22T23:45:51.250 INFO:teuthology.orchestra.run.smithi103.stdout:(92/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.3 MB/s | 5.1 MB 00:00 2024-05-22T23:45:51.261 INFO:teuthology.orchestra.run.smithi105.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-05-22T23:45:51.299 INFO:teuthology.orchestra.run.smithi103.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 252 kB/s | 42 kB 00:00 2024-05-22T23:45:51.329 INFO:teuthology.orchestra.run.smithi105.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-05-22T23:45:51.334 INFO:teuthology.orchestra.run.smithi103.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 531 kB/s | 44 kB 00:00 2024-05-22T23:45:51.368 INFO:teuthology.orchestra.run.smithi103.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-05-22T23:45:51.370 INFO:teuthology.orchestra.run.smithi105.stdout:(124/159): python3-grpcio-1.46.7-10.el9.x86_64. 9.4 MB/s | 2.0 MB 00:00 2024-05-22T23:45:51.396 INFO:teuthology.orchestra.run.smithi105.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 264 kB/s | 18 kB 00:00 2024-05-22T23:45:51.401 INFO:teuthology.orchestra.run.smithi103.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 541 kB/s | 36 kB 00:00 2024-05-22T23:45:51.435 INFO:teuthology.orchestra.run.smithi103.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 716 kB/s | 47 kB 00:00 2024-05-22T23:45:51.438 INFO:teuthology.orchestra.run.smithi105.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 345 kB/s | 23 kB 00:00 2024-05-22T23:45:51.463 INFO:teuthology.orchestra.run.smithi105.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 290 kB/s | 19 kB 00:00 2024-05-22T23:45:51.485 INFO:teuthology.orchestra.run.smithi103.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-05-22T23:45:51.511 INFO:teuthology.orchestra.run.smithi103.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 775 kB/s | 58 kB 00:00 2024-05-22T23:45:51.522 INFO:teuthology.orchestra.run.smithi105.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 235 kB/s | 20 kB 00:00 2024-05-22T23:45:51.553 INFO:teuthology.orchestra.run.smithi103.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-05-22T23:45:51.564 INFO:teuthology.orchestra.run.smithi105.stdout:(129/159): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-05-22T23:45:51.578 INFO:teuthology.orchestra.run.smithi103.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 962 kB/s | 64 kB 00:00 2024-05-22T23:45:51.589 INFO:teuthology.orchestra.run.smithi105.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 693 kB/s | 46 kB 00:00 2024-05-22T23:45:51.620 INFO:teuthology.orchestra.run.smithi103.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 679 kB/s | 45 kB 00:00 2024-05-22T23:45:51.631 INFO:teuthology.orchestra.run.smithi105.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-05-22T23:45:51.656 INFO:teuthology.orchestra.run.smithi105.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 861 kB/s | 58 kB 00:00 2024-05-22T23:45:51.679 INFO:teuthology.orchestra.run.smithi103.stdout:(103/159): lua-devel-5.4.4-4.el9.x86_64.rpm 224 kB/s | 22 kB 00:00 2024-05-22T23:45:51.707 INFO:teuthology.orchestra.run.smithi105.stdout:(133/159): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-05-22T23:45:51.732 INFO:teuthology.orchestra.run.smithi105.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-05-22T23:45:51.762 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-05-22T23:45:51.762 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-05-22T23:45:51.774 INFO:teuthology.orchestra.run.smithi105.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-05-22T23:45:51.799 INFO:teuthology.orchestra.run.smithi105.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-05-22T23:45:51.841 INFO:teuthology.orchestra.run.smithi105.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-05-22T23:45:51.871 INFO:teuthology.orchestra.run.smithi103.stdout:(104/159): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-05-22T23:45:51.875 INFO:teuthology.orchestra.run.smithi105.stdout:(138/159): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-22T23:45:51.925 INFO:teuthology.orchestra.run.smithi105.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2024-05-22T23:45:51.951 INFO:teuthology.orchestra.run.smithi105.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 474 kB/s | 36 kB 00:00 2024-05-22T23:45:51.992 INFO:teuthology.orchestra.run.smithi105.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 759 kB/s | 51 kB 00:00 2024-05-22T23:45:52.026 INFO:teuthology.orchestra.run.smithi105.stdout:(142/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-05-22T23:45:52.051 INFO:teuthology.orchestra.run.smithi105.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-05-22T23:45:52.079 INFO:teuthology.orchestra.run.smithi103.stdout:(105/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-05-22T23:45:52.101 INFO:teuthology.orchestra.run.smithi105.stdout:(144/159): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-05-22T23:45:52.127 INFO:teuthology.orchestra.run.smithi105.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-05-22T23:45:52.177 INFO:teuthology.orchestra.run.smithi105.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-05-22T23:45:52.227 INFO:teuthology.orchestra.run.smithi105.stdout:(147/159): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-05-22T23:45:52.536 INFO:teuthology.orchestra.run.smithi105.stdout:(148/159): librados2-19.0.0-2765.g2e1f7622.el9. 9.4 MB/s | 3.4 MB 00:00 2024-05-22T23:45:52.538 INFO:teuthology.orchestra.run.smithi103.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 50 kB/s | 19 kB 00:00 2024-05-22T23:45:52.622 INFO:teuthology.orchestra.run.smithi103.stdout:(107/159): abseil-cpp-20211102.0-3.el9.x86_64.r 582 kB/s | 548 kB 00:00 2024-05-22T23:45:52.678 INFO:teuthology.orchestra.run.smithi105.stdout:(149/159): librbd1-19.0.0-2765.g2e1f7622.el9.x8 7.0 MB/s | 3.1 MB 00:00 2024-05-22T23:45:52.722 INFO:teuthology.orchestra.run.smithi103.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 262 kB/s | 26 kB 00:00 2024-05-22T23:45:52.782 INFO:teuthology.orchestra.run.smithi103.stdout:(109/159): gperftools-libs-2.9.1-2.el9.x86_64.r 340 kB/s | 309 kB 00:00 2024-05-22T23:45:52.786 INFO:teuthology.orchestra.run.smithi105.stdout:(150/159): NetworkManager-1.47.90-1.el9.x86_64. 9.2 MB/s | 2.3 MB 00:00 2024-05-22T23:45:52.822 INFO:teuthology.orchestra.run.smithi103.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 488 kB/s | 49 kB 00:00 2024-05-22T23:45:52.828 INFO:teuthology.orchestra.run.smithi105.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 1.0 MB/s | 41 kB 00:00 2024-05-22T23:45:52.879 INFO:teuthology.orchestra.run.smithi105.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 4.8 MB/s | 248 kB 00:00 2024-05-22T23:45:52.898 INFO:teuthology.orchestra.run.smithi103.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 579 kB/s | 67 kB 00:00 2024-05-22T23:45:52.921 INFO:teuthology.orchestra.run.smithi105.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 7.4 MB/s | 1.8 MB 00:00 2024-05-22T23:45:52.937 INFO:teuthology.orchestra.run.smithi105.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 628 kB/s | 37 kB 00:00 2024-05-22T23:45:52.948 INFO:teuthology.orchestra.run.smithi103.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-05-22T23:45:52.988 INFO:teuthology.orchestra.run.smithi105.stdout:(155/159): elfutils-libs-0.191-4.el9.x86_64.rpm 5.1 MB/s | 260 kB 00:00 2024-05-22T23:45:53.022 INFO:teuthology.orchestra.run.smithi105.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 2.0 MB/s | 207 kB 00:00 2024-05-22T23:45:53.080 INFO:teuthology.orchestra.run.smithi105.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 15 MB/s | 1.3 MB 00:00 2024-05-22T23:45:53.115 INFO:teuthology.orchestra.run.smithi103.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-05-22T23:45:53.172 INFO:teuthology.orchestra.run.smithi105.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-05-22T23:45:53.173 INFO:teuthology.orchestra.run.smithi103.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.0 MB/s | 839 kB 00:00 2024-05-22T23:45:53.215 INFO:teuthology.orchestra.run.smithi103.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 437 kB/s | 43 kB 00:00 2024-05-22T23:45:53.316 INFO:teuthology.orchestra.run.smithi103.stdout:(116/159): libarrow-9.0.0-11.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2024-05-22T23:45:53.341 INFO:teuthology.orchestra.run.smithi103.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-05-22T23:45:53.366 INFO:teuthology.orchestra.run.smithi103.stdout:(118/159): python3-cachetools-4.2.4-1.el9.noarc 168 kB/s | 32 kB 00:00 2024-05-22T23:45:53.434 INFO:teuthology.orchestra.run.smithi103.stdout:(119/159): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-05-22T23:45:53.501 INFO:teuthology.orchestra.run.smithi103.stdout:(120/159): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-05-22T23:45:53.534 INFO:teuthology.orchestra.run.smithi103.stdout:(121/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2024-05-22T23:45:53.568 INFO:teuthology.orchestra.run.smithi103.stdout:(122/159): python3-google-auth-2.29.0-1.el9.noa 1.5 MB/s | 206 kB 00:00 2024-05-22T23:45:53.652 INFO:teuthology.orchestra.run.smithi103.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-05-22T23:45:53.677 INFO:teuthology.orchestra.run.smithi103.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-05-22T23:45:53.736 INFO:teuthology.orchestra.run.smithi103.stdout:(125/159): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-05-22T23:45:53.752 INFO:teuthology.orchestra.run.smithi103.stdout:(126/159): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-05-22T23:45:53.769 INFO:teuthology.orchestra.run.smithi103.stdout:(127/159): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-05-22T23:45:53.837 INFO:teuthology.orchestra.run.smithi103.stdout:(128/159): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-05-22T23:45:53.853 INFO:teuthology.orchestra.run.smithi103.stdout:(129/159): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-05-22T23:45:53.954 INFO:teuthology.orchestra.run.smithi103.stdout:(130/159): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-05-22T23:45:53.979 INFO:teuthology.orchestra.run.smithi103.stdout:(131/159): python3-logutils-0.3.5-21.el9.noarch 327 kB/s | 46 kB 00:00 2024-05-22T23:45:54.006 INFO:teuthology.orchestra.run.smithi103.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 526 kB/s | 79 kB 00:00 2024-05-22T23:45:54.055 INFO:teuthology.orchestra.run.smithi103.stdout:(133/159): python3-natsort-7.1.1-5.el9.noarch.r 579 kB/s | 58 kB 00:00 2024-05-22T23:45:54.097 INFO:teuthology.orchestra.run.smithi103.stdout:(134/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-05-22T23:45:54.122 INFO:teuthology.orchestra.run.smithi103.stdout:(135/159): python3-portend-3.1.0-2.el9.noarch.r 141 kB/s | 16 kB 00:00 2024-05-22T23:45:54.172 INFO:teuthology.orchestra.run.smithi105.stdout:(159/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.1 MB/s | 16 MB 00:05 2024-05-22T23:45:54.178 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-05-22T23:45:54.178 INFO:teuthology.orchestra.run.smithi105.stdout:Total 10 MB/s | 225 MB 00:21 2024-05-22T23:45:54.179 INFO:teuthology.orchestra.run.smithi103.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 727 kB/s | 90 kB 00:00 2024-05-22T23:45:54.206 INFO:teuthology.orchestra.run.smithi103.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 261 kB/s | 28 kB 00:00 2024-05-22T23:45:54.231 INFO:teuthology.orchestra.run.smithi103.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-05-22T23:45:54.290 INFO:teuthology.orchestra.run.smithi103.stdout:(139/159): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-05-22T23:45:54.315 INFO:teuthology.orchestra.run.smithi103.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 546 kB/s | 59 kB 00:00 2024-05-22T23:45:54.340 INFO:teuthology.orchestra.run.smithi103.stdout:(141/159): python3-tempora-5.0.0-2.el9.noarch.r 330 kB/s | 36 kB 00:00 2024-05-22T23:45:54.399 INFO:teuthology.orchestra.run.smithi103.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 472 kB/s | 51 kB 00:00 2024-05-22T23:45:54.433 INFO:teuthology.orchestra.run.smithi103.stdout:(143/159): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-05-22T23:45:54.458 INFO:teuthology.orchestra.run.smithi103.stdout:(144/159): python3-websocket-client-1.2.3-2.el9 768 kB/s | 90 kB 00:00 2024-05-22T23:45:54.542 INFO:teuthology.orchestra.run.smithi103.stdout:(145/159): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-05-22T23:45:54.567 INFO:teuthology.orchestra.run.smithi103.stdout:(146/159): python3-zc-lockfile-2.0-10.el9.noarc 150 kB/s | 20 kB 00:00 2024-05-22T23:45:54.601 INFO:teuthology.orchestra.run.smithi103.stdout:(147/159): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-05-22T23:45:54.985 INFO:teuthology.orchestra.run.smithi103.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 3.6 MB/s | 1.6 MB 00:00 2024-05-22T23:45:55.093 INFO:teuthology.orchestra.run.smithi103.stdout:(149/159): librados2-19.0.0-2765.g2e1f7622.el9. 6.4 MB/s | 3.4 MB 00:00 2024-05-22T23:45:55.168 INFO:teuthology.orchestra.run.smithi103.stdout:(150/159): librbd1-19.0.0-2765.g2e1f7622.el9.x8 5.5 MB/s | 3.1 MB 00:00 2024-05-22T23:45:55.235 INFO:teuthology.orchestra.run.smithi103.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 638 kB/s | 41 kB 00:00 2024-05-22T23:45:55.311 INFO:teuthology.orchestra.run.smithi103.stdout:(152/159): NetworkManager-1.47.90-1.el9.x86_64. 7.1 MB/s | 2.3 MB 00:00 2024-05-22T23:45:55.378 INFO:teuthology.orchestra.run.smithi103.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 6.4 MB/s | 1.8 MB 00:00 2024-05-22T23:45:55.403 INFO:teuthology.orchestra.run.smithi103.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 404 kB/s | 37 kB 00:00 2024-05-22T23:45:55.414 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-05-22T23:45:55.437 INFO:teuthology.orchestra.run.smithi103.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 1.2 MB/s | 248 kB 00:00 2024-05-22T23:45:55.487 INFO:teuthology.orchestra.run.smithi103.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 1.9 MB/s | 207 kB 00:00 2024-05-22T23:45:55.529 INFO:teuthology.orchestra.run.smithi103.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 2.0 MB/s | 260 kB 00:00 2024-05-22T23:45:55.573 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-05-22T23:45:55.573 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-05-22T23:45:55.604 INFO:teuthology.orchestra.run.smithi103.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 8.1 MB/s | 1.3 MB 00:00 2024-05-22T23:45:55.704 INFO:teuthology.orchestra.run.smithi103.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 11 MB/s | 2.4 MB 00:00 2024-05-22T23:45:55.717 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-05-22T23:45:55.718 INFO:teuthology.orchestra.run.smithi103.stdout:Total 13 MB/s | 225 MB 00:17 2024-05-22T23:45:57.447 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-05-22T23:45:57.447 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-05-22T23:45:58.027 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-05-22T23:45:58.089 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-22T23:45:58.125 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-22T23:45:58.179 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-22T23:45:58.213 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-22T23:45:58.238 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-22T23:45:58.280 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-22T23:45:58.601 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-22T23:45:58.706 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:45:58.827 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:45:58.891 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:45:58.966 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-05-22T23:45:59.122 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:45:59.152 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 10/170 2024-05-22T23:45:59.185 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-22T23:45:59.217 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-05-22T23:45:59.218 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-05-22T23:45:59.376 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-22T23:45:59.434 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-22T23:45:59.454 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:45:59.541 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:45:59.563 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 15/170 2024-05-22T23:45:59.609 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 15/170 2024-05-22T23:45:59.631 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-22T23:45:59.651 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-22T23:45:59.839 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-22T23:45:59.964 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-22T23:46:00.113 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-22T23:46:00.182 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-22T23:46:00.286 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:00.358 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:00.561 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:00.668 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-22T23:46:00.759 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-22T23:46:00.797 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-05-22T23:46:00.832 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-05-22T23:46:00.906 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-05-22T23:46:00.940 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-05-22T23:46:00.984 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-05-22T23:46:01.016 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-05-22T23:46:01.044 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-05-22T23:46:01.113 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-05-22T23:46:01.168 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-05-22T23:46:01.198 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-05-22T23:46:01.233 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-05-22T23:46:01.309 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-05-22T23:46:01.344 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9. 34/170 2024-05-22T23:46:01.379 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el 35/170 2024-05-22T23:46:01.408 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_6 36/170 2024-05-22T23:46:01.477 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-05-22T23:46:01.523 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-05-22T23:46:01.596 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-05-22T23:46:01.729 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-05-22T23:46:01.782 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-05-22T23:46:01.895 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-05-22T23:46:01.895 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-05-22T23:46:02.311 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-05-22T23:46:02.395 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-05-22T23:46:02.424 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-05-22T23:46:02.460 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-05-22T23:46:02.501 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-05-22T23:46:02.531 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_6 47/170 2024-05-22T23:46:02.572 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-05-22T23:46:02.724 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-05-22T23:46:02.783 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-05-22T23:46:02.837 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-05-22T23:46:02.910 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-22T23:46:02.941 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-22T23:46:02.978 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-22T23:46:03.025 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-22T23:46:03.061 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-22T23:46:03.095 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-22T23:46:03.138 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-22T23:46:03.273 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-22T23:46:03.314 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-22T23:46:03.355 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-22T23:46:03.450 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-22T23:46:03.648 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-22T23:46:03.688 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-22T23:46:03.721 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-22T23:46:03.746 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-22T23:46:03.823 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-22T23:46:03.830 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-05-22T23:46:03.893 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-22T23:46:03.932 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-22T23:46:03.955 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-22T23:46:03.985 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-22T23:46:04.003 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-22T23:46:04.019 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-22T23:46:04.030 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-22T23:46:04.045 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-22T23:46:04.065 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-22T23:46:04.087 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-22T23:46:04.104 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-22T23:46:04.136 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-22T23:46:04.165 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-22T23:46:04.191 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-22T23:46:04.216 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-22T23:46:04.250 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-22T23:46:04.284 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-22T23:46:04.370 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-22T23:46:04.408 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-22T23:46:04.415 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-22T23:46:04.446 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-22T23:46:04.493 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-22T23:46:04.591 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-22T23:46:04.612 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:46:04.642 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-22T23:46:04.742 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:46:04.772 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:46:04.846 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:46:04.876 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 10/170 2024-05-22T23:46:04.908 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-22T23:46:05.085 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-22T23:46:05.141 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-22T23:46:05.169 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:46:05.256 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:46:05.278 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 15/170 2024-05-22T23:46:05.333 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 15/170 2024-05-22T23:46:05.353 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-22T23:46:05.354 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-22T23:46:05.374 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-22T23:46:05.395 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-22T23:46:05.428 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-22T23:46:05.526 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-22T23:46:05.563 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-22T23:46:05.574 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-22T23:46:05.619 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-22T23:46:05.694 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-22T23:46:05.706 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-22T23:46:05.738 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-22T23:46:05.843 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-22T23:46:05.904 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-22T23:46:05.914 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-22T23:46:05.945 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-22T23:46:06.001 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:06.022 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-22T23:46:06.080 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:06.280 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:06.390 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-22T23:46:06.503 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-22T23:46:06.546 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-05-22T23:46:06.581 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-05-22T23:46:06.655 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-05-22T23:46:06.669 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-22T23:46:06.689 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-05-22T23:46:06.732 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-05-22T23:46:06.764 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-05-22T23:46:06.793 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-05-22T23:46:06.793 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-22T23:46:06.854 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-05-22T23:46:06.915 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-05-22T23:46:06.945 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-05-22T23:46:06.981 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-05-22T23:46:07.057 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-05-22T23:46:07.091 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9. 34/170 2024-05-22T23:46:07.143 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el 35/170 2024-05-22T23:46:07.172 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_6 36/170 2024-05-22T23:46:07.243 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-05-22T23:46:07.286 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-05-22T23:46:07.362 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-05-22T23:46:07.520 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-05-22T23:46:07.572 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-05-22T23:46:08.018 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-22T23:46:08.106 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-05-22T23:46:08.114 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-22T23:46:08.142 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-22T23:46:08.175 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-22T23:46:08.186 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-05-22T23:46:08.214 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-05-22T23:46:08.250 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-05-22T23:46:08.299 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-05-22T23:46:08.337 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_6 47/170 2024-05-22T23:46:08.378 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-05-22T23:46:08.484 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-22T23:46:08.530 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 103/170 2024-05-22T23:46:08.549 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-05-22T23:46:08.607 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-05-22T23:46:08.629 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 103/170 2024-05-22T23:46:08.660 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 104/170 2024-05-22T23:46:08.662 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-05-22T23:46:08.696 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-22T23:46:08.707 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-05-22T23:46:08.735 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-22T23:46:08.765 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-22T23:46:08.802 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-22T23:46:08.812 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-22T23:46:08.849 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-22T23:46:08.893 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-22T23:46:08.894 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-22T23:46:08.936 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-22T23:46:08.978 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-22T23:46:09.034 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-22T23:46:09.098 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-22T23:46:09.119 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-22T23:46:09.128 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-22T23:46:09.164 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-22T23:46:09.187 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-22T23:46:09.205 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-22T23:46:09.206 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-22T23:46:09.300 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-22T23:46:09.330 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-22T23:46:09.370 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-22T23:46:09.395 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-22T23:46:09.420 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-22T23:46:09.489 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 107/170 2024-05-22T23:46:09.496 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-22T23:46:09.588 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 107/170 2024-05-22T23:46:09.615 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-22T23:46:09.618 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 108/170 2024-05-22T23:46:09.632 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-22T23:46:09.669 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-22T23:46:09.697 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-22T23:46:09.731 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-22T23:46:09.758 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:46:09.770 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-22T23:46:09.802 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-22T23:46:09.832 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-22T23:46:09.867 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-22T23:46:09.908 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-22T23:46:09.929 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:46:09.950 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-22T23:46:09.963 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:46:09.984 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-22T23:46:10.072 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-22T23:46:10.095 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:46:10.127 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-22T23:46:10.128 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 10/170 2024-05-22T23:46:10.154 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-22T23:46:10.201 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-22T23:46:10.204 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-22T23:46:10.301 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-22T23:46:10.356 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-22T23:46:10.532 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-22T23:46:10.603 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-22T23:46:10.628 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:46:10.776 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:46:10.812 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 15/170 2024-05-22T23:46:10.889 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 15/170 2024-05-22T23:46:10.922 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-22T23:46:10.978 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-22T23:46:11.092 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-22T23:46:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-22T23:46:11.186 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-22T23:46:11.289 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-22T23:46:11.293 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-22T23:46:11.358 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-22T23:46:11.406 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-22T23:46:11.492 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-22T23:46:11.539 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-22T23:46:11.581 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-22T23:46:11.716 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-22T23:46:11.745 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-22T23:46:11.778 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-22T23:46:11.822 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-22T23:46:11.883 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-22T23:46:11.927 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:11.956 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:12.061 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:12.165 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:12.481 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-22T23:46:12.498 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:12.500 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-22T23:46:12.542 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 110/170 2024-05-22T23:46:12.585 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-22T23:46:12.635 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 110/170 2024-05-22T23:46:12.636 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-22T23:46:12.636 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-22T23:46:12.636 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:12.660 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 111/170 2024-05-22T23:46:12.714 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-22T23:46:12.835 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-22T23:46:12.882 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 22/170 2024-05-22T23:46:12.931 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 23/170 2024-05-22T23:46:13.034 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/170 2024-05-22T23:46:13.086 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 25/170 2024-05-22T23:46:13.146 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 26/170 2024-05-22T23:46:13.182 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 27/170 2024-05-22T23:46:13.221 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 28/170 2024-05-22T23:46:13.361 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 29/170 2024-05-22T23:46:13.434 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 30/170 2024-05-22T23:46:13.473 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 31/170 2024-05-22T23:46:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 32/170 2024-05-22T23:46:13.638 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 33/170 2024-05-22T23:46:13.677 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9. 34/170 2024-05-22T23:46:13.717 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el 35/170 2024-05-22T23:46:13.750 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_6 36/170 2024-05-22T23:46:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-22T23:46:13.850 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 37/170 2024-05-22T23:46:13.907 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-22T23:46:13.913 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-5.el9.noarch 38/170 2024-05-22T23:46:13.935 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-22T23:46:13.968 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-22T23:46:14.044 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 39/170 2024-05-22T23:46:14.214 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 40/170 2024-05-22T23:46:14.278 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-22T23:46:14.282 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 41/170 2024-05-22T23:46:14.331 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 103/170 2024-05-22T23:46:14.405 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 103/170 2024-05-22T23:46:14.436 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 104/170 2024-05-22T23:46:14.474 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-22T23:46:14.942 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-22T23:46:14.968 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 107/170 2024-05-22T23:46:14.971 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 42/170 2024-05-22T23:46:15.039 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 107/170 2024-05-22T23:46:15.069 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 108/170 2024-05-22T23:46:15.079 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 43/170 2024-05-22T23:46:15.116 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 44/170 2024-05-22T23:46:15.162 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 45/170 2024-05-22T23:46:15.230 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 46/170 2024-05-22T23:46:15.293 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_6 47/170 2024-05-22T23:46:15.356 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 48/170 2024-05-22T23:46:15.555 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/170 2024-05-22T23:46:15.618 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 50/170 2024-05-22T23:46:15.694 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 51/170 2024-05-22T23:46:15.790 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-22T23:46:15.836 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-22T23:46:15.873 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-22T23:46:15.954 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-22T23:46:16.000 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-22T23:46:16.054 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-22T23:46:16.118 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-22T23:46:16.315 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-22T23:46:16.360 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-22T23:46:16.412 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-22T23:46:16.540 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-22T23:46:16.592 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-22T23:46:16.639 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-22T23:46:16.674 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-22T23:46:16.708 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-22T23:46:16.946 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-22T23:46:17.130 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-22T23:46:17.172 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-22T23:46:17.210 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-22T23:46:17.251 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-22T23:46:17.314 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-22T23:46:17.363 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:17.376 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-22T23:46:17.399 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:17.418 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-22T23:46:17.488 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-22T23:46:17.541 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-22T23:46:17.617 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-22T23:46:17.662 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-22T23:46:17.788 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-22T23:46:17.849 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-22T23:46:17.904 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-22T23:46:17.941 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:17.971 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-22T23:46:17.985 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 110/170 2024-05-22T23:46:18.067 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 110/170 2024-05-22T23:46:18.067 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-22T23:46:18.067 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-22T23:46:18.067 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:18.095 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 111/170 2024-05-22T23:46:18.109 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-22T23:46:18.204 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-22T23:46:19.148 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-22T23:46:19.206 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-22T23:46:19.248 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-22T23:46:19.382 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-22T23:46:19.448 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-22T23:46:19.504 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-22T23:46:19.604 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-22T23:46:19.643 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-22T23:46:19.876 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-22T23:46:19.915 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-22T23:46:20.061 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-22T23:46:20.901 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-22T23:46:21.073 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-22T23:46:22.687 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-22T23:46:22.787 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-22T23:46:22.822 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-22T23:46:22.858 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-22T23:46:23.221 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-22T23:46:23.270 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 103/170 2024-05-22T23:46:23.388 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 103/170 2024-05-22T23:46:23.422 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 104/170 2024-05-22T23:46:23.486 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-22T23:46:23.994 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 111/170 2024-05-22T23:46:23.994 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /sys 2024-05-22T23:46:23.994 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /proc 2024-05-22T23:46:23.994 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /mnt 2024-05-22T23:46:23.994 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /var/tmp 2024-05-22T23:46:23.994 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /home 2024-05-22T23:46:23.995 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /root 2024-05-22T23:46:23.995 INFO:teuthology.orchestra.run.smithi174.stdout:skipping the directory /tmp 2024-05-22T23:46:23.995 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:24.026 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-22T23:46:24.061 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 107/170 2024-05-22T23:46:24.161 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 107/170 2024-05-22T23:46:24.169 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 112/170 2024-05-22T23:46:24.185 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 112/170 2024-05-22T23:46:24.185 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:24.185 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-22T23:46:24.185 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:24.188 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 108/170 2024-05-22T23:46:25.689 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 113/170 2024-05-22T23:46:25.714 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 113/170 2024-05-22T23:46:25.715 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:25.715 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-22T23:46:25.715 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-22T23:46:25.715 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-22T23:46:25.715 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:25.939 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 114/170 2024-05-22T23:46:25.962 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 114/170 2024-05-22T23:46:25.962 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:25.962 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-22T23:46:25.962 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-22T23:46:25.962 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-22T23:46:25.962 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:26.407 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 115/170 2024-05-22T23:46:26.429 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 115/170 2024-05-22T23:46:26.430 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:26.430 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-22T23:46:26.430 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-22T23:46:26.430 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-22T23:46:26.430 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:26.528 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-22T23:46:26.585 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-22T23:46:26.640 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-22T23:46:26.827 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-22T23:46:26.852 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:26.882 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-22T23:46:26.893 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:26.959 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-22T23:46:27.052 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-22T23:46:27.188 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-22T23:46:27.232 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-22T23:46:27.264 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-22T23:46:27.291 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-22T23:46:27.527 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 109/170 2024-05-22T23:46:27.572 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 110/170 2024-05-22T23:46:27.702 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 127/170 2024-05-22T23:46:27.731 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 110/170 2024-05-22T23:46:27.731 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-22T23:46:27.732 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-22T23:46:27.732 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:27.732 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 127/170 2024-05-22T23:46:27.763 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.e 128/170 2024-05-22T23:46:27.764 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 111/170 2024-05-22T23:46:27.824 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622. 129/170 2024-05-22T23:46:27.854 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 130/170 2024-05-22T23:46:28.891 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 130/170 2024-05-22T23:46:28.923 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 131/170 2024-05-22T23:46:29.645 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 111/170 2024-05-22T23:46:29.645 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-05-22T23:46:29.645 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-05-22T23:46:29.645 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-05-22T23:46:29.645 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-05-22T23:46:29.646 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-05-22T23:46:29.646 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-05-22T23:46:29.646 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-05-22T23:46:29.646 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:29.819 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 112/170 2024-05-22T23:46:29.834 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 112/170 2024-05-22T23:46:29.834 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:29.834 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-22T23:46:29.834 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:29.894 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 131/170 2024-05-22T23:46:29.915 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 132/170 2024-05-22T23:46:30.018 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 132/170 2024-05-22T23:46:30.178 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el 133/170 2024-05-22T23:46:30.208 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 134/170 2024-05-22T23:46:30.231 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 134/170 2024-05-22T23:46:30.231 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:30.231 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-22T23:46:30.231 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-22T23:46:30.231 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-22T23:46:30.231 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:30.275 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 135/170 2024-05-22T23:46:30.287 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 135/170 2024-05-22T23:46:30.386 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 136/170 2024-05-22T23:46:30.440 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 137/170 2024-05-22T23:46:31.322 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 113/170 2024-05-22T23:46:31.338 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-22T23:46:31.348 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 113/170 2024-05-22T23:46:31.348 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:31.348 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-22T23:46:31.348 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-22T23:46:31.348 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-22T23:46:31.349 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:31.395 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 139/170 2024-05-22T23:46:31.417 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 139/170 2024-05-22T23:46:31.417 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:31.418 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-22T23:46:31.418 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-22T23:46:31.418 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-22T23:46:31.418 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:31.462 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 140/170 2024-05-22T23:46:31.484 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 140/170 2024-05-22T23:46:31.484 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:31.484 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-22T23:46:31.484 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:31.529 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 141/170 2024-05-22T23:46:31.552 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 141/170 2024-05-22T23:46:31.552 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:31.552 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-22T23:46:31.553 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-22T23:46:31.553 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-22T23:46:31.553 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:31.582 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 114/170 2024-05-22T23:46:31.605 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 114/170 2024-05-22T23:46:31.605 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:31.605 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-22T23:46:31.605 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-22T23:46:31.605 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-22T23:46:31.605 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:31.855 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 142/170 2024-05-22T23:46:31.877 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 142/170 2024-05-22T23:46:31.877 INFO:teuthology.orchestra.run.smithi174.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:31.877 INFO:teuthology.orchestra.run.smithi174.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-22T23:46:31.877 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-22T23:46:31.877 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-22T23:46:31.877 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:32.049 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 115/170 2024-05-22T23:46:32.072 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 115/170 2024-05-22T23:46:32.072 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:32.073 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-22T23:46:32.073 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-22T23:46:32.073 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-22T23:46:32.073 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:32.179 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-22T23:46:32.236 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-22T23:46:32.292 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-22T23:46:32.478 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-22T23:46:32.526 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-22T23:46:32.603 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-22T23:46:32.692 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-22T23:46:32.849 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-22T23:46:32.893 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-22T23:46:32.924 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-22T23:46:32.951 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-22T23:46:33.312 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 127/170 2024-05-22T23:46:33.342 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 127/170 2024-05-22T23:46:33.373 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.e 128/170 2024-05-22T23:46:33.436 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622. 129/170 2024-05-22T23:46:33.465 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 130/170 2024-05-22T23:46:34.503 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 130/170 2024-05-22T23:46:34.541 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 131/170 2024-05-22T23:46:35.500 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 131/170 2024-05-22T23:46:35.533 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 132/170 2024-05-22T23:46:35.637 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 132/170 2024-05-22T23:46:35.813 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el 133/170 2024-05-22T23:46:35.853 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 134/170 2024-05-22T23:46:35.875 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 134/170 2024-05-22T23:46:35.876 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:35.876 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-22T23:46:35.876 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-22T23:46:35.876 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-22T23:46:35.876 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:35.928 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 135/170 2024-05-22T23:46:35.939 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 135/170 2024-05-22T23:46:36.039 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 136/170 2024-05-22T23:46:36.085 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 137/170 2024-05-22T23:46:36.416 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 143/170 2024-05-22T23:46:36.457 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 144/170 2024-05-22T23:46:36.498 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 145/170 2024-05-22T23:46:36.530 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-22T23:46:36.557 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-22T23:46:36.742 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 148/170 2024-05-22T23:46:36.786 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-22T23:46:36.811 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_ 150/170 2024-05-22T23:46:36.842 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-22T23:46:36.872 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-22T23:46:36.910 INFO:teuthology.orchestra.run.smithi174.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-22T23:46:36.978 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-22T23:46:37.006 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 139/170 2024-05-22T23:46:37.029 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 139/170 2024-05-22T23:46:37.030 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:37.030 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-22T23:46:37.030 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-22T23:46:37.030 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-22T23:46:37.030 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:37.073 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 140/170 2024-05-22T23:46:37.095 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 140/170 2024-05-22T23:46:37.095 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:37.095 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-22T23:46:37.095 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:37.140 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 141/170 2024-05-22T23:46:37.163 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 141/170 2024-05-22T23:46:37.163 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:37.163 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-22T23:46:37.163 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-22T23:46:37.163 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-22T23:46:37.163 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:37.466 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 142/170 2024-05-22T23:46:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 142/170 2024-05-22T23:46:37.489 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:37.489 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-22T23:46:37.489 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-22T23:46:37.489 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-22T23:46:37.489 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:37.534 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:37.574 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:37.594 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:38.421 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 155/170 2024-05-22T23:46:38.519 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-22T23:46:38.561 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-22T23:46:38.641 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-22T23:46:38.681 INFO:teuthology.orchestra.run.smithi174.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-22T23:46:38.706 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-22T23:46:38.732 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-22T23:46:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-22T23:46:38.758 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-22T23:46:38.850 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-22T23:46:38.875 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-22T23:46:38.875 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-22T23:46:38.976 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-22T23:46:39.001 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-22T23:46:39.007 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:46:39.007 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:46:39.195 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:46:39.195 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-22T23:46:39.279 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-22T23:46:39.304 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-22T23:46:39.366 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-22T23:46:40.773 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 111/170 2024-05-22T23:46:40.773 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-05-22T23:46:40.773 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-05-22T23:46:40.773 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-05-22T23:46:40.773 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-05-22T23:46:40.773 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-05-22T23:46:40.774 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-05-22T23:46:40.774 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-05-22T23:46:40.774 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:40.989 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 112/170 2024-05-22T23:46:41.038 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 112/170 2024-05-22T23:46:41.038 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:41.038 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-22T23:46:41.038 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:41.986 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 143/170 2024-05-22T23:46:42.027 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 144/170 2024-05-22T23:46:42.068 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 145/170 2024-05-22T23:46:42.100 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-22T23:46:42.128 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-22T23:46:42.312 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 148/170 2024-05-22T23:46:42.365 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-22T23:46:42.398 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_ 150/170 2024-05-22T23:46:42.438 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-22T23:46:42.468 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-22T23:46:42.514 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-22T23:46:42.674 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 113/170 2024-05-22T23:46:42.754 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 113/170 2024-05-22T23:46:42.754 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:42.754 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-22T23:46:42.754 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-22T23:46:42.754 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-22T23:46:42.754 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:43.024 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 114/170 2024-05-22T23:46:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 114/170 2024-05-22T23:46:43.095 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:43.095 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-22T23:46:43.095 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-22T23:46:43.095 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-22T23:46:43.095 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:43.098 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:43.137 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:43.157 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:43.616 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 115/170 2024-05-22T23:46:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 115/170 2024-05-22T23:46:43.685 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:43.685 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-22T23:46:43.685 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-22T23:46:43.685 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-22T23:46:43.685 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:43.834 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-22T23:46:43.907 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-22T23:46:43.982 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-22T23:46:43.997 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 155/170 2024-05-22T23:46:44.083 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-22T23:46:44.124 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-22T23:46:44.208 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-22T23:46:44.241 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-22T23:46:44.244 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-22T23:46:44.270 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-22T23:46:44.296 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-22T23:46:44.313 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-22T23:46:44.321 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-22T23:46:44.322 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-22T23:46:44.406 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-22T23:46:44.412 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-22T23:46:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-22T23:46:44.439 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-22T23:46:44.507 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-22T23:46:44.532 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-22T23:46:44.537 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:46:44.537 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:46:44.549 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-22T23:46:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-22T23:46:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 1/170 2024-05-22T23:46:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 2/170 2024-05-22T23:46:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 3/170 2024-05-22T23:46:44.695 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 4/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 5/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 6/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 7/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 10/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 11/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 12/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 13/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_ 15/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 16/170 2024-05-22T23:46:44.696 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 17/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_6 18/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 19/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 20/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el 21/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9. 22/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_6 23/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 24/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 25/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 26/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 27/170 2024-05-22T23:46:44.697 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 28/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 29/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622. 30/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 31/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 32/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 33/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el 34/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 35/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.e 36/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 37/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 38/170 2024-05-22T23:46:44.698 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 39/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-22T23:46:44.699 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-22T23:46:44.700 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-22T23:46:44.701 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-22T23:46:44.702 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-22T23:46:44.703 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-22T23:46:44.704 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-22T23:46:44.705 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-22T23:46:44.706 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-22T23:46:44.707 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 149/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 151/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-22T23:46:44.708 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-22T23:46:44.709 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-22T23:46:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-22T23:46:44.710 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-22T23:46:44.717 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:46:44.717 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-22T23:46:44.801 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-22T23:46:44.826 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-22T23:46:44.891 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-22T23:46:44.920 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-22T23:46:44.981 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-22T23:46:45.025 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-22T23:46:45.071 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-22T23:46:45.713 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 127/170 2024-05-22T23:46:45.750 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 127/170 2024-05-22T23:46:45.805 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.e 128/170 2024-05-22T23:46:45.912 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622. 129/170 2024-05-22T23:46:45.952 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 130/170 2024-05-22T23:46:47.280 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 130/170 2024-05-22T23:46:47.329 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 131/170 2024-05-22T23:46:47.472 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-22T23:46:47.472 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:47.472 INFO:teuthology.orchestra.run.smithi174.stdout:Upgraded: 2024-05-22T23:46:47.472 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout:Installed: 2024-05-22T23:46:47.473 INFO:teuthology.orchestra.run.smithi174.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: automake-1.16.2-8.el9.noarch 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: bison-3.7.4-5.el9.x86_64 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.474 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.475 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: flex-2.6.4-9.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-22T23:46:47.476 INFO:teuthology.orchestra.run.smithi174.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-22T23:46:47.477 INFO:teuthology.orchestra.run.smithi174.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: lua-5.4.4-4.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-22T23:46:47.478 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-13.14-1.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-22T23:46:47.479 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-22T23:46:47.480 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-22T23:46:47.481 INFO:teuthology.orchestra.run.smithi174.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ply-3.11-14.el9.noarch 2024-05-22T23:46:47.482 INFO:teuthology.orchestra.run.smithi174.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-22T23:46:47.483 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-22T23:46:47.484 INFO:teuthology.orchestra.run.smithi174.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: re2-1:20211101-3.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-22T23:46:47.485 INFO:teuthology.orchestra.run.smithi174.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-22T23:46:47.486 INFO:teuthology.orchestra.run.smithi174.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-22T23:46:47.486 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:46:47.486 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-05-22T23:46:47.912 DEBUG:teuthology.parallel:result is None 2024-05-22T23:46:48.428 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 131/170 2024-05-22T23:46:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 132/170 2024-05-22T23:46:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 132/170 2024-05-22T23:46:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el 133/170 2024-05-22T23:46:48.823 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 134/170 2024-05-22T23:46:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 134/170 2024-05-22T23:46:48.916 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:48.916 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-22T23:46:48.916 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-22T23:46:48.916 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-22T23:46:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:48.982 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 135/170 2024-05-22T23:46:49.020 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 135/170 2024-05-22T23:46:49.165 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 136/170 2024-05-22T23:46:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 137/170 2024-05-22T23:46:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-22T23:46:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 139/170 2024-05-22T23:46:50.356 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 139/170 2024-05-22T23:46:50.356 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:50.356 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-22T23:46:50.357 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-22T23:46:50.357 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-22T23:46:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 140/170 2024-05-22T23:46:50.409 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-22T23:46:50.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 1/170 2024-05-22T23:46:50.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 2/170 2024-05-22T23:46:50.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 3/170 2024-05-22T23:46:50.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 4/170 2024-05-22T23:46:50.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 5/170 2024-05-22T23:46:50.409 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 6/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 7/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 10/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 11/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 12/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 13/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_ 15/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 16/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 17/170 2024-05-22T23:46:50.410 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_6 18/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 19/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 20/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el 21/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9. 22/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_6 23/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 24/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 25/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 26/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 27/170 2024-05-22T23:46:50.411 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 28/170 2024-05-22T23:46:50.424 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 29/170 2024-05-22T23:46:50.424 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622. 30/170 2024-05-22T23:46:50.424 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 31/170 2024-05-22T23:46:50.424 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 32/170 2024-05-22T23:46:50.424 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 33/170 2024-05-22T23:46:50.424 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el 34/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 35/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.e 36/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 37/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 38/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 39/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-22T23:46:50.425 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-22T23:46:50.426 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-22T23:46:50.427 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-22T23:46:50.428 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-22T23:46:50.429 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-22T23:46:50.430 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-22T23:46:50.431 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-22T23:46:50.432 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-22T23:46:50.433 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 149/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 151/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-22T23:46:50.434 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-22T23:46:50.435 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-22T23:46:50.436 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-22T23:46:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 140/170 2024-05-22T23:46:50.498 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:50.498 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-22T23:46:50.498 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 141/170 2024-05-22T23:46:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 141/170 2024-05-22T23:46:50.681 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:50.682 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-22T23:46:50.682 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-22T23:46:50.682 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-22T23:46:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:51.088 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 142/170 2024-05-22T23:46:51.164 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 142/170 2024-05-22T23:46:51.165 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-22T23:46:51.165 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-22T23:46:51.165 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-22T23:46:51.165 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-22T23:46:51.165 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-22T23:46:53.213 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: automake-1.16.2-8.el9.noarch 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: bison-3.7.4-5.el9.x86_64 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-22T23:46:53.214 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.215 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.216 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.216 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.216 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.216 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.216 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.216 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.216 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.216 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: flex-2.6.4-9.el9.x86_64 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-22T23:46:53.217 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-22T23:46:53.218 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: lua-5.4.4-4.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-22T23:46:53.219 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-13.14-1.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-22T23:46:53.220 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-22T23:46:53.221 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-22T23:46:53.222 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-05-22T23:46:53.223 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-22T23:46:53.224 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-22T23:46:53.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: re2-1:20211101-3.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-22T23:46:53.226 INFO:teuthology.orchestra.run.smithi105.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-22T23:46:53.227 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-22T23:46:53.227 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-22T23:46:53.227 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-22T23:46:53.227 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-05-22T23:46:53.227 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-05-22T23:46:53.670 DEBUG:teuthology.parallel:result is None 2024-05-22T23:46:56.093 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 143/170 2024-05-22T23:46:56.139 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 144/170 2024-05-22T23:46:56.193 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 145/170 2024-05-22T23:46:56.226 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-22T23:46:56.264 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-22T23:46:56.554 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 148/170 2024-05-22T23:46:56.607 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-22T23:46:56.643 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_ 150/170 2024-05-22T23:46:56.700 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-22T23:46:56.752 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-22T23:46:56.816 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-22T23:46:57.780 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:57.829 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:57.894 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-22T23:46:59.238 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 155/170 2024-05-22T23:46:59.353 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-22T23:46:59.413 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-22T23:46:59.554 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-22T23:46:59.588 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-22T23:46:59.630 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-22T23:46:59.664 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-22T23:46:59.706 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-22T23:46:59.707 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-22T23:46:59.840 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-22T23:46:59.886 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-22T23:46:59.886 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-22T23:47:00.022 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-22T23:47:00.053 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-22T23:47:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:47:00.080 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:47:00.415 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-22T23:47:00.415 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-22T23:47:00.561 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-22T23:47:00.607 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-22T23:47:00.702 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-22T23:47:00.967 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-22T23:47:00.968 INFO:teuthology.orchestra.run.smithi103.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-05-22T23:47:00.968 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:47:06.996 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 1/170 2024-05-22T23:47:06.996 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 2/170 2024-05-22T23:47:06.996 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 3/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 4/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7 5/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 6/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 7/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 8/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 9/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 10/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 11/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 12/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 13/170 2024-05-22T23:47:06.997 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 14/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_ 15/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 16/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 17/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_6 18/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86 19/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 20/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el 21/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9. 22/170 2024-05-22T23:47:06.998 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_6 23/170 2024-05-22T23:47:06.999 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 24/170 2024-05-22T23:47:06.999 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 25/170 2024-05-22T23:47:06.999 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 26/170 2024-05-22T23:47:06.999 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 27/170 2024-05-22T23:47:06.999 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 28/170 2024-05-22T23:47:06.999 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 29/170 2024-05-22T23:47:07.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622. 30/170 2024-05-22T23:47:07.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noa 31/170 2024-05-22T23:47:07.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.n 32/170 2024-05-22T23:47:07.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1 33/170 2024-05-22T23:47:07.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el 34/170 2024-05-22T23:47:07.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 35/170 2024-05-22T23:47:07.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.e 36/170 2024-05-22T23:47:07.012 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 37/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 38/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 39/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-22T23:47:07.013 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-22T23:47:07.014 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-22T23:47:07.015 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-22T23:47:07.016 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-22T23:47:07.017 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-22T23:47:07.018 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-22T23:47:07.019 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-22T23:47:07.020 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 149/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 151/170 2024-05-22T23:47:07.021 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-22T23:47:07.022 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-22T23:47:07.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-22T23:47:07.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-22T23:47:07.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-22T23:47:07.023 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-22T23:47:09.320 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-22T23:47:09.320 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-22T23:47:09.321 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: automake-1.16.2-8.el9.noarch 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: bison-3.7.4-5.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.322 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.323 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top-2:19.0.0-2765.g2e1f7622.el9.noarch 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: flex-2.6.4-9.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-22T23:47:09.324 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.325 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-22T23:47:09.326 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-13.14-1.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-22T23:47:09.327 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-22T23:47:09.328 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-22T23:47:09.329 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-22T23:47:09.330 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-22T23:47:09.331 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-22T23:47:09.332 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-2765.g2e1f7622.el9.x86_64 2024-05-22T23:47:09.333 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-05-22T23:47:09.334 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-22T23:47:09.334 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-22T23:47:09.334 INFO:teuthology.orchestra.run.smithi103.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-22T23:47:09.334 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-22T23:47:09.334 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-22T23:47:09.334 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-22T23:47:09.334 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:47:09.335 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-05-22T23:47:10.016 DEBUG:teuthology.parallel:result is None 2024-05-22T23:47:10.017 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:47:10.160 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-22T23:47:10.227 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-2765.g2e1f7622.el9 2024-05-22T23:47:10.228 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2765.g2e1f7622.el9 2024-05-22T23:47:10.228 INFO:teuthology.task.install:The correct ceph version 19.0.0-2765.g2e1f7622 is installed. 2024-05-22T23:47:10.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:47:10.372 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-22T23:47:10.397 INFO:teuthology.orchestra.run.smithi105.stdout:19.0.0-2765.g2e1f7622.el9 2024-05-22T23:47:10.397 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2765.g2e1f7622.el9 2024-05-22T23:47:10.397 INFO:teuthology.task.install:The correct ceph version 19.0.0-2765.g2e1f7622 is installed. 2024-05-22T23:47:10.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:47:10.544 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-22T23:47:10.571 INFO:teuthology.orchestra.run.smithi174.stdout:19.0.0-2765.g2e1f7622.el9 2024-05-22T23:47:10.571 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2765.g2e1f7622.el9 2024-05-22T23:47:10.571 INFO:teuthology.task.install:The correct ceph version 19.0.0-2765.g2e1f7622 is installed. 2024-05-22T23:47:10.573 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-22T23:47:10.573 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:47:10.573 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-22T23:47:10.640 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:47:10.641 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-22T23:47:10.672 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:47:10.672 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-22T23:47:10.703 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-22T23:47:10.704 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:47:10.704 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-05-22T23:47:10.769 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-22T23:47:10.877 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:47:10.877 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-05-22T23:47:10.905 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-22T23:47:10.972 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:47:10.972 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-05-22T23:47:10.999 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-22T23:47:11.063 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-22T23:47:11.063 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:47:11.063 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-22T23:47:11.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-22T23:47:11.238 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:47:11.239 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-22T23:47:11.263 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-22T23:47:11.327 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:47:11.327 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-22T23:47:11.351 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-22T23:47:11.414 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-22T23:47:11.415 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:47:11.415 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-05-22T23:47:11.476 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-22T23:47:11.578 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:47:11.578 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-05-22T23:47:11.604 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-22T23:47:11.669 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:47:11.732 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-05-22T23:47:11.756 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-22T23:47:11.819 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-22T23:47:11.920 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '2e1f76222e88afe0f1cda37775b2d45d88289794', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-05-22T23:47:11.920 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:47:11.921 INFO:tasks.cephadm:Cluster fsid is 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:47:11.921 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-22T23:47:11.921 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.103', 'mon.b': '172.21.15.105', 'mon.c': '172.21.15.174'} 2024-05-22T23:47:11.921 INFO:tasks.cephadm:First mon is mon.a on smithi103 2024-05-22T23:47:11.922 INFO:tasks.cephadm:First mgr is x 2024-05-22T23:47:11.922 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-22T23:47:11.922 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-05-22T23:47:11.986 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-05-22T23:47:12.010 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-05-22T23:47:12.033 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-22T23:47:12.033 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:47:12.250 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2e1f76222e88afe0f1cda37775b2d45d88289794', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55202/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2765-g2e1f7622', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2765.g2e1f7622'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-22 16:50:39.768604', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-05-22T23:47:12.463 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 2e1f76222e88afe0f1cda37775b2d45d88289794 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:47:12.465 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/x86_64/flavors/default/cephadm 2024-05-22T23:47:12.465 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:13.141 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 781817 May 22 23:47 /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:13.142 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:13.748 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 781817 May 22 23:47 /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:13.748 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/2e1f76222e88afe0f1cda37775b2d45d88289794/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:14.224 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 781817 May 22 23:47 /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:14.225 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:14.282 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:14.301 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-22T23:47:14.334 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 on all hosts... 2024-05-22T23:47:14.334 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 pull 2024-05-22T23:47:14.341 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 pull 2024-05-22T23:47:14.344 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 pull 2024-05-22T23:47:15.198 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794... 2024-05-22T23:47:15.228 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794... 2024-05-22T23:47:15.643 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794... 2024-05-22T23:47:52.747 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-22T23:47:52.748 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 19.0.0-2765-g2e1f7622 (2e1f76222e88afe0f1cda37775b2d45d88289794) squid (rc)", 2024-05-22T23:47:52.748 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41", 2024-05-22T23:47:52.748 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-05-22T23:47:52.748 INFO:teuthology.orchestra.run.smithi103.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036" 2024-05-22T23:47:52.748 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-22T23:47:52.748 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-22T23:48:07.563 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-05-22T23:48:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.0.0-2765-g2e1f7622 (2e1f76222e88afe0f1cda37775b2d45d88289794) squid (rc)", 2024-05-22T23:48:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41", 2024-05-22T23:48:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-05-22T23:48:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036" 2024-05-22T23:48:07.564 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-05-22T23:48:07.564 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-05-22T23:48:08.833 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-22T23:48:08.834 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 19.0.0-2765-g2e1f7622 (2e1f76222e88afe0f1cda37775b2d45d88289794) squid (rc)", 2024-05-22T23:48:08.834 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41", 2024-05-22T23:48:08.834 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-05-22T23:48:08.834 INFO:teuthology.orchestra.run.smithi105.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036" 2024-05-22T23:48:08.834 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-05-22T23:48:08.834 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-22T23:48:08.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-05-22T23:48:08.951 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-05-22T23:48:08.978 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-05-22T23:48:09.009 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-05-22T23:48:09.096 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-05-22T23:48:09.120 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-05-22T23:48:09.149 INFO:tasks.cephadm:Writing seed config... 2024-05-22T23:48:09.150 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-22T23:48:09.151 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-22T23:48:09.152 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-22T23:48:09.153 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-22T23:48:09.154 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-22T23:48:09.155 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-22T23:48:09.155 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-22T23:48:09.155 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-22T23:48:09.155 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-05-22T23:48:09.156 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:48:09.156 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-22T23:48:09.212 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9c0f3dc8-1895-11ef-bc9a-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-22T23:48:09.213 DEBUG:teuthology.orchestra.run.smithi103:mon.a> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a.service 2024-05-22T23:48:09.260 DEBUG:teuthology.orchestra.run.smithi103:mgr.x> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.x.service 2024-05-22T23:48:09.306 INFO:tasks.cephadm:Bootstrapping... 2024-05-22T23:48:09.306 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 -v bootstrap --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.103 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-22T23:48:09.578 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-05-22T23:48:09.579 INFO:teuthology.orchestra.run.smithi103.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794', '-v', 'bootstrap', '--fsid', '9c0f3dc8-1895-11ef-bc9a-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.103', '--skip-admin-label'] 2024-05-22T23:48:09.579 INFO:teuthology.orchestra.run.smithi103.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-22T23:48:09.580 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying podman|docker is present... 2024-05-22T23:48:09.635 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 5.0.2 2024-05-22T23:48:09.635 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying lvm2 is present... 2024-05-22T23:48:09.635 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying time synchronization is in place... 2024-05-22T23:48:09.650 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-22T23:48:09.650 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-22T23:48:09.664 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-22T23:48:09.664 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-05-22T23:48:09.678 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout enabled 2024-05-22T23:48:09.692 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout active 2024-05-22T23:48:09.692 INFO:teuthology.orchestra.run.smithi103.stdout:Unit chronyd.service is enabled and running 2024-05-22T23:48:09.692 INFO:teuthology.orchestra.run.smithi103.stdout:Repeating the final host check... 2024-05-22T23:48:09.741 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout 5.0.2 2024-05-22T23:48:09.741 INFO:teuthology.orchestra.run.smithi103.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-22T23:48:09.741 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl is present 2024-05-22T23:48:09.741 INFO:teuthology.orchestra.run.smithi103.stdout:lvcreate is present 2024-05-22T23:48:09.755 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-22T23:48:09.755 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-22T23:48:09.768 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-22T23:48:09.768 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-05-22T23:48:09.784 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout enabled 2024-05-22T23:48:09.800 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout active 2024-05-22T23:48:09.800 INFO:teuthology.orchestra.run.smithi103.stdout:Unit chronyd.service is enabled and running 2024-05-22T23:48:09.800 INFO:teuthology.orchestra.run.smithi103.stdout:Host looks OK 2024-05-22T23:48:09.800 INFO:teuthology.orchestra.run.smithi103.stdout:Cluster fsid: 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:09.800 INFO:teuthology.orchestra.run.smithi103.stdout:Acquiring lock 140485710062976 on /run/cephadm/9c0f3dc8-1895-11ef-bc9a-c7b262605968.lock 2024-05-22T23:48:09.800 INFO:teuthology.orchestra.run.smithi103.stdout:Lock 140485710062976 acquired on /run/cephadm/9c0f3dc8-1895-11ef-bc9a-c7b262605968.lock 2024-05-22T23:48:09.801 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying IP 172.21.15.103 port 3300 ... 2024-05-22T23:48:09.801 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying IP 172.21.15.103 port 6789 ... 2024-05-22T23:48:09.801 INFO:teuthology.orchestra.run.smithi103.stdout:Base mon IP(s) is [172.21.15.103:3300, 172.21.15.103:6789], mon addrv is [v2:172.21.15.103:3300,v1:172.21.15.103:6789] 2024-05-22T23:48:09.808 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.103 metric 100 2024-05-22T23:48:09.809 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.103 metric 100 2024-05-22T23:48:09.815 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-22T23:48:09.815 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-22T23:48:09.815 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1782sec hoplimit 64 pref medium 2024-05-22T23:48:09.821 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-22T23:48:09.821 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-22T23:48:09.821 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-22T23:48:09.821 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-22T23:48:09.821 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-05-22T23:48:09.821 INFO:teuthology.orchestra.run.smithi103.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-22T23:48:09.822 INFO:teuthology.orchestra.run.smithi103.stdout:Mon IP `172.21.15.103` is in CIDR network `172.21.0.0/20` 2024-05-22T23:48:09.822 INFO:teuthology.orchestra.run.smithi103.stdout:Mon IP `172.21.15.103` is in CIDR network `172.21.0.0/20` 2024-05-22T23:48:09.822 INFO:teuthology.orchestra.run.smithi103.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-22T23:48:09.823 INFO:teuthology.orchestra.run.smithi103.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-22T23:48:09.824 INFO:teuthology.orchestra.run.smithi103.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794... 2024-05-22T23:48:10.442 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stdout f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 2024-05-22T23:48:10.443 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794... 2024-05-22T23:48:10.443 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Getting image source signatures 2024-05-22T23:48:10.443 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying blob sha256:f82b46892f110bee096f0ba81d48593d085ecabe0b8d4a508f5ad49a8eb9726e 2024-05-22T23:48:10.443 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying blob sha256:6ac36e7c57f227110df5cd65cfdca1cc05e0b17df6748637a6aa04c5c80f89c6 2024-05-22T23:48:10.443 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Copying config sha256:f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 2024-05-22T23:48:10.443 INFO:teuthology.orchestra.run.smithi103.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-22T23:48:11.663 INFO:teuthology.orchestra.run.smithi103.stdout:ceph: stdout ceph version 19.0.0-2765-g2e1f7622 (2e1f76222e88afe0f1cda37775b2d45d88289794) squid (rc) 2024-05-22T23:48:11.663 INFO:teuthology.orchestra.run.smithi103.stdout:Ceph version: ceph version 19.0.0-2765-g2e1f7622 (2e1f76222e88afe0f1cda37775b2d45d88289794) squid (rc) 2024-05-22T23:48:11.663 INFO:teuthology.orchestra.run.smithi103.stdout:Extracting ceph user uid/gid from container image... 2024-05-22T23:48:12.886 INFO:teuthology.orchestra.run.smithi103.stdout:stat: stdout 167 167 2024-05-22T23:48:12.886 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial keys... 2024-05-22T23:48:13.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQA9hE5mQCl8FBAAL4G1eA9QORXAIqQE38GW0A== 2024-05-22T23:48:15.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQA+hE5mlApLGRAAMMaum/1Wxl7Mvtu9ncDaXQ== 2024-05-22T23:48:16.378 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph-authtool: stdout AQA/hE5maKXnHxAAtsmU+m+7L0/BinwWlYGvqw== 2024-05-22T23:48:16.380 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial monmap... 2024-05-22T23:48:17.510 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-22T23:48:17.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-22T23:48:17.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:17.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-22T23:48:17.511 INFO:teuthology.orchestra.run.smithi103.stdout:monmaptool for a [v2:172.21.15.103:3300,v1:172.21.15.103:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-22T23:48:17.511 INFO:teuthology.orchestra.run.smithi103.stdout:setting min_mon_release = quincy 2024-05-22T23:48:17.511 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: set fsid to 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:17.512 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-22T23:48:17.512 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:48:17.512 INFO:teuthology.orchestra.run.smithi103.stdout:Creating mon... 2024-05-22T23:48:18.593 INFO:teuthology.orchestra.run.smithi103.stdout:create mon.a on 2024-05-22T23:48:18.951 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-22T23:48:19.191 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-22T23:48:19.475 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968.target → /etc/systemd/system/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968.target. 2024-05-22T23:48:19.476 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968.target → /etc/systemd/system/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968.target. 2024-05-22T23:48:19.867 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a 2024-05-22T23:48:19.867 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a.service: Unit ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a.service not loaded. 2024-05-22T23:48:20.150 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968.target.wants/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a.service → /etc/systemd/system/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@.service. 2024-05-22T23:48:20.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:20 smithi103 podman[30973]: 2024-05-22 23:48:20.571273294 +0000 UTC m=+0.162899690 container create dfa949ac8503eb095026325192f14976a4816cf49867e7444d499e435fe455c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-22T23:48:21.381 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-22T23:48:21.381 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-05-22T23:48:21.395 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-22T23:48:21.395 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-05-22T23:48:21.396 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-05-22T23:48:21.396 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to enable service . firewalld.service is not available 2024-05-22T23:48:21.396 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mon to start... 2024-05-22T23:48:21.396 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mon... 2024-05-22T23:48:21.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:21 smithi103 podman[30973]: 2024-05-22 23:48:21.250729538 +0000 UTC m=+0.842355993 container init dfa949ac8503eb095026325192f14976a4816cf49867e7444d499e435fe455c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-22T23:48:21.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:21 smithi103 podman[30973]: 2024-05-22 23:48:21.263041624 +0000 UTC m=+0.854668056 container start dfa949ac8503eb095026325192f14976a4816cf49867e7444d499e435fe455c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-22T23:48:21.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:21 smithi103 ceph-mon[30992]: mkfs 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:21.544 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:21 smithi103 ceph-mon[30992]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-22T23:48:21.545 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:21 smithi103 bash[30973]: dfa949ac8503eb095026325192f14976a4816cf49867e7444d499e435fe455c9 2024-05-22T23:48:21.545 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:21 smithi103 systemd[1]: Started Ceph mon.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:48:22.735 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-22T23:48:22.735 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: monmap epoch 1 2024-05-22T23:48:22.735 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:22.735 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: last_changed 2024-05-22T23:48:16.842516+0000 2024-05-22T23:48:22.735 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:48:22.735 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: min_mon_release 19 (squid) 2024-05-22T23:48:22.735 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: election_strategy: 1 2024-05-22T23:48:22.735 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:48:22.736 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: fsmap 2024-05-22T23:48:22.736 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: osdmap e1: 0 total, 0 up, 0 in 2024-05-22T23:48:22.736 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: mgrmap e1: no daemons active 2024-05-22T23:48:22.736 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:22 smithi103 ceph-mon[30992]: from='client.? 172.21.15.103:0/2660855465' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-22T23:48:22.756 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout cluster: 2024-05-22T23:48:22.756 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout id: 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:22.756 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-22T23:48:22.756 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:22.756 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout services: 2024-05-22T23:48:22.756 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.745685s) 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout data: 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout pgs: 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.991+0000 7fc2fb000700 1 Processor -- start 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.991+0000 7fc2fb000700 1 -- start start 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.992+0000 7fc2fb000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f410ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:22.757 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.992+0000 7fc2fb000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f407beb0 con 0x7fc2f4108870 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.993+0000 7fc2f3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f410ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.993+0000 7fc2f3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f410ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44292/0 (socket says 172.21.15.103:44292) 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.993+0000 7fc2f3e00700 1 -- 172.21.15.103:0/1974044733 learned_addr learned my addr 172.21.15.103:0/1974044733 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.993+0000 7fc2f3e00700 1 -- 172.21.15.103:0/1974044733 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f410b1a0 con 0x7fc2f4108870 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.994+0000 7fc2f3e00700 1 --2- 172.21.15.103:0/1974044733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f410ac60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40098a0 tx=0x7fc2e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65da0c2c05447e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.994+0000 7fc2f2a00700 1 -- 172.21.15.103:0/1974044733 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2e4004160 con 0x7fc2f4108870 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.994+0000 7fc2f2a00700 1 -- 172.21.15.103:0/1974044733 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e4004300 con 0x7fc2f4108870 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.994+0000 7fc2f2a00700 1 -- 172.21.15.103:0/1974044733 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2e40036a0 con 0x7fc2f4108870 2024-05-22T23:48:22.758 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.995+0000 7fc2fb000700 1 -- 172.21.15.103:0/1974044733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 msgr2=0x7fc2f410ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.995+0000 7fc2fb000700 1 --2- 172.21.15.103:0/1974044733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f410ac60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40098a0 tx=0x7fc2e4009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.995+0000 7fc2fb000700 1 -- 172.21.15.103:0/1974044733 shutdown_connections 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.995+0000 7fc2fb000700 1 --2- 172.21.15.103:0/1974044733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f410ac60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.995+0000 7fc2fb000700 1 -- 172.21.15.103:0/1974044733 >> 172.21.15.103:0/1974044733 conn(0x7fc2f41022a0 msgr2=0x7fc2f41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.995+0000 7fc2fb000700 1 -- 172.21.15.103:0/1974044733 shutdown_connections 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.995+0000 7fc2fb000700 1 -- 172.21.15.103:0/1974044733 wait complete. 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.996+0000 7fc2fb000700 1 Processor -- start 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.996+0000 7fc2fb000700 1 -- start start 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.997+0000 7fc2fb000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f419b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.997+0000 7fc2fb000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f410b670 con 0x7fc2f4108870 2024-05-22T23:48:22.759 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.997+0000 7fc2f3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f419b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.997+0000 7fc2f3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f419b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44304/0 (socket says 172.21.15.103:44304) 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.997+0000 7fc2f3e00700 1 -- 172.21.15.103:0/2660855465 learned_addr learned my addr 172.21.15.103:0/2660855465 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.997+0000 7fc2f3e00700 1 -- 172.21.15.103:0/2660855465 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f419b960 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.998+0000 7fc2f3e00700 1 --2- 172.21.15.103:0/2660855465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f419b420 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4000c00 tx=0x7fc2e4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.998+0000 7fc2f0c00700 1 -- 172.21.15.103:0/2660855465 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2e4003db0 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.998+0000 7fc2f0c00700 1 -- 172.21.15.103:0/2660855465 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2e4011630 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.998+0000 7fc2f0c00700 1 -- 172.21.15.103:0/2660855465 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc2e4011950 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.998+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f419bbf0 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.998+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f419e8e0 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.999+0000 7fc2f0c00700 1 -- 172.21.15.103:0/2660855465 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc2e4011af0 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.999+0000 7fc2f0c00700 1 -- 172.21.15.103:0/2660855465 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc2e401c450 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:21.999+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f40639c0 con 0x7fc2f4108870 2024-05-22T23:48:22.760 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.002+0000 7fc2f0c00700 1 -- 172.21.15.103:0/2660855465 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc2e4032020 con 0x7fc2f4108870 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.080+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fc2f419ee00 con 0x7fc2f4108870 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.081+0000 7fc2f0c00700 1 -- 172.21.15.103:0/2660855465 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fc2e4004260 con 0x7fc2f4108870 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.083+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 msgr2=0x7fc2f419b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.083+0000 7fc2fb000700 1 --2- 172.21.15.103:0/2660855465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f419b420 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4000c00 tx=0x7fc2e4004750 comp rx=0 tx=0).stop 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.083+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 shutdown_connections 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.083+0000 7fc2fb000700 1 --2- 172.21.15.103:0/2660855465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2f4108870 0x7fc2f419b420 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.083+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 >> 172.21.15.103:0/2660855465 conn(0x7fc2f41022a0 msgr2=0x7fc2f4103090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.083+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 shutdown_connections 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:22.083+0000 7fc2fb000700 1 -- 172.21.15.103:0/2660855465 wait complete. 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:mon is available 2024-05-22T23:48:22.761 INFO:teuthology.orchestra.run.smithi103.stdout:Assimilating anything we can from ceph.conf... 2024-05-22T23:48:23.715 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:23 smithi103 ceph-mon[30992]: from='client.? 172.21.15.103:0/4228481825' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-22T23:48:23.715 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:23 smithi103 ceph-mon[30992]: from='client.? 172.21.15.103:0/4228481825' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-22T23:48:24.156 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:24.156 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [global] 2024-05-22T23:48:24.156 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout fsid = 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.103:3300,v1:172.21.15.103:6789] 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [osd] 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-22T23:48:24.157 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.332+0000 7fea4f000700 1 Processor -- start 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.332+0000 7fea4f000700 1 -- start start 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.333+0000 7fea4f000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4807c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.333+0000 7fea4f000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea4807c880 con 0x7fea4807ddd0 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.333+0000 7fea4dc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4807c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.334+0000 7fea4dc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4807c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44306/0 (socket says 172.21.15.103:44306) 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.334+0000 7fea4dc00700 1 -- 172.21.15.103:0/636660147 learned_addr learned my addr 172.21.15.103:0/636660147 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.334+0000 7fea4dc00700 1 -- 172.21.15.103:0/636660147 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4807ca00 con 0x7fea4807ddd0 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.334+0000 7fea4dc00700 1 --2- 172.21.15.103:0/636660147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4807c340 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0098a0 tx=0x7fea3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488d1336a99dc245 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:24.158 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.335+0000 7fea47e00700 1 -- 172.21.15.103:0/636660147 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea3c004160 con 0x7fea4807ddd0 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.335+0000 7fea47e00700 1 -- 172.21.15.103:0/636660147 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea3c004300 con 0x7fea4807ddd0 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.335+0000 7fea47e00700 1 -- 172.21.15.103:0/636660147 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea3c0036a0 con 0x7fea4807ddd0 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.336+0000 7fea4f000700 1 -- 172.21.15.103:0/636660147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 msgr2=0x7fea4807c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.336+0000 7fea4f000700 1 --2- 172.21.15.103:0/636660147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4807c340 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fea3c0098a0 tx=0x7fea3c009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.336+0000 7fea4f000700 1 -- 172.21.15.103:0/636660147 shutdown_connections 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.336+0000 7fea4f000700 1 --2- 172.21.15.103:0/636660147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4807c340 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.336+0000 7fea4f000700 1 -- 172.21.15.103:0/636660147 >> 172.21.15.103:0/636660147 conn(0x7fea48103300 msgr2=0x7fea48105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.336+0000 7fea4f000700 1 -- 172.21.15.103:0/636660147 shutdown_connections 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.336+0000 7fea4f000700 1 -- 172.21.15.103:0/636660147 wait complete. 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.337+0000 7fea4f000700 1 Processor -- start 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.338+0000 7fea4f000700 1 -- start start 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.338+0000 7fea4f000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4819d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:24.159 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.338+0000 7fea4f000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea4807ced0 con 0x7fea4807ddd0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.338+0000 7fea4dc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4819d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.338+0000 7fea4dc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4819d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44314/0 (socket says 172.21.15.103:44314) 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.338+0000 7fea4dc00700 1 -- 172.21.15.103:0/4228481825 learned_addr learned my addr 172.21.15.103:0/4228481825 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.338+0000 7fea4dc00700 1 -- 172.21.15.103:0/4228481825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4819d9e0 con 0x7fea4807ddd0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.339+0000 7fea4dc00700 1 --2- 172.21.15.103:0/4228481825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4819d4a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fea3c000c00 tx=0x7fea3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.339+0000 7fea46000700 1 -- 172.21.15.103:0/4228481825 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea3c003db0 con 0x7fea4807ddd0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.339+0000 7fea46000700 1 -- 172.21.15.103:0/4228481825 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea3c011630 con 0x7fea4807ddd0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.339+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4819dc70 con 0x7fea4807ddd0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.339+0000 7fea46000700 1 -- 172.21.15.103:0/4228481825 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea3c011950 con 0x7fea4807ddd0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.339+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea481a0960 con 0x7fea4807ddd0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.340+0000 7fea46000700 1 -- 172.21.15.103:0/4228481825 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fea3c01a460 con 0x7fea4807ddd0 2024-05-22T23:48:24.160 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.340+0000 7fea46000700 1 -- 172.21.15.103:0/4228481825 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fea3c01b780 con 0x7fea4807ddd0 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.340+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea48069490 con 0x7fea4807ddd0 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.344+0000 7fea46000700 1 -- 172.21.15.103:0/4228481825 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fea3c01ba20 con 0x7fea4807ddd0 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.419+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fea481a0b30 con 0x7fea4807ddd0 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.425+0000 7fea46000700 1 -- 172.21.15.103:0/4228481825 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+431 (secure 0 0 0) 0x7fea3c01bc00 con 0x7fea4807ddd0 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.426+0000 7fea46000700 1 -- 172.21.15.103:0/4228481825 <== mon.0 v2:172.21.15.103:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fea3c01bde0 con 0x7fea4807ddd0 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.426+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 msgr2=0x7fea4819d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.426+0000 7fea4f000700 1 --2- 172.21.15.103:0/4228481825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4819d4a0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fea3c000c00 tx=0x7fea3c004750 comp rx=0 tx=0).stop 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.426+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 shutdown_connections 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.426+0000 7fea4f000700 1 --2- 172.21.15.103:0/4228481825 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea4807ddd0 0x7fea4819d4a0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.426+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 >> 172.21.15.103:0/4228481825 conn(0x7fea48103300 msgr2=0x7fea48104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.427+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 shutdown_connections 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:23.427+0000 7fea4f000700 1 -- 172.21.15.103:0/4228481825 wait complete. 2024-05-22T23:48:24.161 INFO:teuthology.orchestra.run.smithi103.stdout:Generating new minimal ceph.conf... 2024-05-22T23:48:25.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:24 smithi103 ceph-mon[30992]: from='client.? 172.21.15.103:0/1429884317' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:48:25.544 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.771+0000 7f209f600700 1 Processor -- start 2024-05-22T23:48:25.544 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.771+0000 7f209f600700 1 -- start start 2024-05-22T23:48:25.544 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.772+0000 7f209f600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209810a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:25.545 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.772+0000 7f209f600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f209810ab40 con 0x7f209810a150 2024-05-22T23:48:25.545 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.772+0000 7f209cc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209810a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:25.545 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.772+0000 7f209cc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209810a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45028/0 (socket says 172.21.15.103:45028) 2024-05-22T23:48:25.546 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.772+0000 7f209cc00700 1 -- 172.21.15.103:0/1435155054 learned_addr learned my addr 172.21.15.103:0/1435155054 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:25.546 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.773+0000 7f209cc00700 1 -- 172.21.15.103:0/1435155054 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209810b260 con 0x7f209810a150 2024-05-22T23:48:25.546 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.773+0000 7f209cc00700 1 --2- 172.21.15.103:0/1435155054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209810a570 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f208c0098a0 tx=0x7f208c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406fd1775883ba17 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:25.546 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.773+0000 7f2097400700 1 -- 172.21.15.103:0/1435155054 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f208c004160 con 0x7f209810a150 2024-05-22T23:48:25.546 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.773+0000 7f2097400700 1 -- 172.21.15.103:0/1435155054 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f208c004300 con 0x7f209810a150 2024-05-22T23:48:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.773+0000 7f2097400700 1 -- 172.21.15.103:0/1435155054 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f208c011b50 con 0x7f209810a150 2024-05-22T23:48:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.774+0000 7f209f600700 1 -- 172.21.15.103:0/1435155054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 msgr2=0x7f209810a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.774+0000 7f209f600700 1 --2- 172.21.15.103:0/1435155054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209810a570 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f208c0098a0 tx=0x7f208c009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.774+0000 7f209f600700 1 -- 172.21.15.103:0/1435155054 shutdown_connections 2024-05-22T23:48:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.774+0000 7f209f600700 1 --2- 172.21.15.103:0/1435155054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209810a570 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:25.548 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.774+0000 7f209f600700 1 -- 172.21.15.103:0/1435155054 >> 172.21.15.103:0/1435155054 conn(0x7f209807d6c0 msgr2=0x7f209807daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.774+0000 7f209f600700 1 -- 172.21.15.103:0/1435155054 shutdown_connections 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.774+0000 7f209f600700 1 -- 172.21.15.103:0/1435155054 wait complete. 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.775+0000 7f209f600700 1 Processor -- start 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.775+0000 7f209f600700 1 -- start start 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.776+0000 7f209f600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209819ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.776+0000 7f209f600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f209810e610 con 0x7f209810a150 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.776+0000 7f209cc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209819ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.776+0000 7f209cc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209819ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45030/0 (socket says 172.21.15.103:45030) 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.776+0000 7f209cc00700 1 -- 172.21.15.103:0/1429884317 learned_addr learned my addr 172.21.15.103:0/1429884317 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.776+0000 7f209cc00700 1 -- 172.21.15.103:0/1429884317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f209819e410 con 0x7f209810a150 2024-05-22T23:48:25.549 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.777+0000 7f209cc00700 1 --2- 172.21.15.103:0/1429884317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209819ded0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f208c000c00 tx=0x7f208c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:25.551 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.777+0000 7f2095600700 1 -- 172.21.15.103:0/1429884317 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f208c003b20 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.777+0000 7f2095600700 1 -- 172.21.15.103:0/1429884317 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f208c003cc0 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.777+0000 7f2095600700 1 -- 172.21.15.103:0/1429884317 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f208c024910 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.777+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f209819e6a0 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.777+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20981a1390 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.777+0000 7f2095600700 1 -- 172.21.15.103:0/1429884317 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f208c022020 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.777+0000 7f2095600700 1 -- 172.21.15.103:0/1429884317 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f208c011e00 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.778+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20980639c0 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.781+0000 7f2095600700 1 -- 172.21.15.103:0/1429884317 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f208c02ddf0 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.857+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f20981a1560 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.858+0000 7f2095600700 1 -- 172.21.15.103:0/1429884317 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f208c032030 con 0x7f209810a150 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.859+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 msgr2=0x7f209819ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.859+0000 7f209f600700 1 --2- 172.21.15.103:0/1429884317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209819ded0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f208c000c00 tx=0x7f208c011c10 comp rx=0 tx=0).stop 2024-05-22T23:48:25.552 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.859+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 shutdown_connections 2024-05-22T23:48:25.553 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.859+0000 7f209f600700 1 --2- 172.21.15.103:0/1429884317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f209810a150 0x7f209819ded0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:25.553 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.859+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 >> 172.21.15.103:0/1429884317 conn(0x7f209807d6c0 msgr2=0x7f2098107cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:25.553 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.859+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 shutdown_connections 2024-05-22T23:48:25.553 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:24.859+0000 7f209f600700 1 -- 172.21.15.103:0/1429884317 wait complete. 2024-05-22T23:48:25.553 INFO:teuthology.orchestra.run.smithi103.stdout:Restarting the monitor... 2024-05-22T23:48:25.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:25 smithi103 systemd[1]: Stopping Ceph mon.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:48:26.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[30988]: 2024-05-22T23:48:26.079+0000 7f9532000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-22T23:48:26.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[30988]: 2024-05-22T23:48:26.079+0000 7f9532000700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-22T23:48:26.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:26 smithi103 podman[31247]: 2024-05-22 23:48:26.347751523 +0000 UTC m=+0.478082650 container died dfa949ac8503eb095026325192f14976a4816cf49867e7444d499e435fe455c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-22T23:48:26.926 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:26 smithi103 podman[31247]: 2024-05-22 23:48:26.745686672 +0000 UTC m=+0.876017809 container cleanup dfa949ac8503eb095026325192f14976a4816cf49867e7444d499e435fe455c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-22T23:48:26.926 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:26 smithi103 bash[31247]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a 2024-05-22T23:48:27.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:26 smithi103 podman[31261]: 2024-05-22 23:48:26.92161706 +0000 UTC m=+0.555535389 container remove dfa949ac8503eb095026325192f14976a4816cf49867e7444d499e435fe455c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-22T23:48:27.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:27 smithi103 systemd[1]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-22T23:48:27.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:27 smithi103 systemd[1]: Stopped Ceph mon.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:48:27.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:27 smithi103 systemd[1]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a.service: Consumed 1.897s CPU time. 2024-05-22T23:48:27.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:27 smithi103 systemd[1]: Starting Ceph mon.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:48:28.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:27 smithi103 podman[31352]: 2024-05-22 23:48:27.730484124 +0000 UTC m=+0.033122793 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:48:28.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:27 smithi103 podman[31352]: 2024-05-22 23:48:27.838974586 +0000 UTC m=+0.141613294 container create 5991f9961eca9c90b37667cedc4d48ea60dc400738805add2efcf99fc3cc9477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, io.buildah.version=1.33.5) 2024-05-22T23:48:28.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 podman[31352]: 2024-05-22 23:48:28.070170717 +0000 UTC m=+0.372809445 container init 5991f9961eca9c90b37667cedc4d48ea60dc400738805add2efcf99fc3cc9477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-22T23:48:28.179 INFO:teuthology.orchestra.run.smithi103.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-22T23:48:28.338 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 podman[31352]: 2024-05-22 23:48:28.08065855 +0000 UTC m=+0.383297236 container start 5991f9961eca9c90b37667cedc4d48ea60dc400738805add2efcf99fc3cc9477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: set uid:gid to 167:167 (ceph:ceph) 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: ceph version 19.0.0-2765-g2e1f7622 (2e1f76222e88afe0f1cda37775b2d45d88289794) squid (rc), process ceph-mon, pid 2 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: pidfile_write: ignore empty --pid-file 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: load: jerasure load: lrc 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: RocksDB version: 7.9.2 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Git sha 0 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Compile date 2024-05-22 16:19:42 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: DB SUMMARY 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: DB Session ID: KOXLMDUQXCBOYAWC3FVJ 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: CURRENT file: CURRENT 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: IDENTITY file: IDENTITY 2024-05-22T23:48:28.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105340 ; 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.error_if_exists: 0 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.create_if_missing: 0 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.paranoid_checks: 1 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.env: 0x560b5855bc60 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.fs: PosixFileSystem 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.info_log: 0x560b82cd3b20 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_file_opening_threads: 16 2024-05-22T23:48:28.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.statistics: (nil) 2024-05-22T23:48:28.341 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.use_fsync: 0 2024-05-22T23:48:28.341 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_log_file_size: 0 2024-05-22T23:48:28.341 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-22T23:48:28.341 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-22T23:48:28.341 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.keep_log_file_num: 1000 2024-05-22T23:48:28.341 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.recycle_log_file_num: 0 2024-05-22T23:48:28.341 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.allow_fallocate: 1 2024-05-22T23:48:28.342 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.allow_mmap_reads: 0 2024-05-22T23:48:28.342 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.allow_mmap_writes: 0 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.use_direct_reads: 0 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.create_missing_column_families: 0 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.db_log_dir: 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.wal_dir: 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.advise_random_on_open: 1 2024-05-22T23:48:28.343 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.db_write_buffer_size: 0 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.write_buffer_manager: 0x560b82ca41e0 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.rate_limiter: (nil) 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.wal_recovery_mode: 2 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.enable_thread_tracking: 0 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.enable_pipelined_write: 0 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.unordered_write: 0 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-22T23:48:28.344 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.row_cache: None 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.wal_filter: None 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.allow_ingest_behind: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.two_write_queues: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.manual_wal_flush: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.wal_compression: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.atomic_flush: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.log_readahead_size: 0 2024-05-22T23:48:28.345 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.best_efforts_recovery: 0 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.allow_data_in_errors: 0 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.db_host_id: __hostname__ 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_background_jobs: 2 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_background_compactions: -1 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_subcompactions: 1 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-22T23:48:28.346 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_total_wal_size: 0 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_open_files: -1 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bytes_per_sync: 0 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_readahead_size: 0 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_background_flushes: -1 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Compression algorithms supported: 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: kZSTD supported: 0 2024-05-22T23:48:28.347 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: kXpressCompression supported: 0 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: kLZ4HCCompression supported: 1 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: kZlibCompression supported: 1 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: kSnappyCompression supported: 1 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: kLZ4Compression supported: 1 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: kBZip2Compression supported: 0 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.merge_operator: 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_filter: None 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_filter_factory: None 2024-05-22T23:48:28.348 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.sst_partitioner_factory: None 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560b82cd3c40) 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: cache_index_and_filter_blocks: 1 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: pin_top_level_index_and_filter: 1 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_type: 0 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: data_block_index_type: 0 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_shortening: 1 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: checksum: 4 2024-05-22T23:48:28.349 INFO:journalctl@ceph.mon.a.smithi103.stdout: no_block_cache: 0 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache: 0x560b82cd5090 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_name: BinnedLRUCache 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_options: 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: capacity : 536870912 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: num_shard_bits : 4 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: strict_capacity_limit : 0 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: high_pri_pool_ratio: 0.000 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_cache_compressed: (nil) 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: persistent_cache: (nil) 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_size: 4096 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_size_deviation: 10 2024-05-22T23:48:28.350 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_restart_interval: 16 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: index_block_restart_interval: 1 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: metadata_block_size: 4096 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: partition_filters: 0 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: use_delta_encoding: 1 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: filter_policy: bloomfilter 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: whole_key_filtering: 1 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: verify_compression: 0 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: read_amp_bytes_per_bit: 0 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: format_version: 5 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: enable_index_compression: 1 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: block_align: 0 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: max_auto_readahead_size: 262144 2024-05-22T23:48:28.351 INFO:journalctl@ceph.mon.a.smithi103.stdout: prepopulate_block_cache: 0 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout: initial_auto_readahead_size: 8192 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout: num_file_reads_for_auto_readahead: 2 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.write_buffer_size: 33554432 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_write_buffer_number: 2 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression: NoCompression 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression: Disabled 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.prefix_extractor: nullptr 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.num_levels: 7 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-22T23:48:28.352 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.level: 32767 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.strategy: 0 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-22T23:48:28.353 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.enabled: false 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.target_file_size_base: 67108864 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-22T23:48:28.354 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.arena_block_size: 1048576 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-22T23:48:28.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.disable_auto_compactions: 0 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-22T23:48:28.356 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.table_properties_collectors: 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.inplace_update_support: 0 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.bloom_locality: 0 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.max_successive_merges: 0 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.paranoid_file_checks: 0 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.force_consistency_checks: 1 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.report_bg_io_stats: 0 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.ttl: 2592000 2024-05-22T23:48:28.357 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.enable_blob_files: false 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.min_blob_size: 0 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.blob_file_size: 268435456 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.blob_file_starting_level: 0 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-22T23:48:28.358 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5fc11563-17f2-40ff-85ed-5c4a93512a9f 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716421708139713, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716421708142892, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100369, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 328, "table_properties": {"data_size": 98289, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14298, "raw_average_key_size": 49, "raw_value_size": 90557, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 287, "num_filter_entries": 287, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716421708, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5fc11563-17f2-40ff-85ed-5c4a93512a9f", "db_session_id": "KOXLMDUQXCBOYAWC3FVJ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716421708143106, "job": 1, "event": "recovery_finished"} 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560b82dc4000 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: DB pointer 0x560b82dae000 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** DB Stats ** 2024-05-22T23:48:28.359 INFO:journalctl@ceph.mon.a.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** Compaction Stats [default] ** 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: L0 2/0 99.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 47.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Sum 2/0 99.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 47.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-22T23:48:28.360 INFO:journalctl@ceph.mon.a.smithi103.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 47.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** Compaction Stats [default] ** 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 47.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-22T23:48:28.361 INFO:journalctl@ceph.mon.a.smithi103.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout: Cumulative compaction: 0.00 GB write, 9.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout: Interval compaction: 0.00 GB write, 9.77 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout: Block cache BinnedLRUCache@0x560b82cd5090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: starting mon.a rank 0 at public addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] at bind addrs [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a@-1(???) e1 preinit fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a@-1(???).mds e1 new map 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a@-1(???).mds e1 print_map 2024-05-22T23:48:28.362 INFO:journalctl@ceph.mon.a.smithi103.stdout: e1 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout: legacy client fscid: -1 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout: 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout: No filesystems configured 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 bash[31352]: 5991f9961eca9c90b37667cedc4d48ea60dc400738805add2efcf99fc3cc9477 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 systemd[1]: Started Ceph mon.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:48:28.363 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: monmap epoch 1 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: last_changed 2024-05-22T23:48:16.842516+0000 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: min_mon_release 19 (squid) 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: election_strategy: 1 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: fsmap 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: osdmap e1: 0 total, 0 up, 0 in 2024-05-22T23:48:28.364 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:28 smithi103 ceph-mon[31370]: mgrmap e1: no daemons active 2024-05-22T23:48:29.640 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.864+0000 7f3c34a00700 1 Processor -- start 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.865+0000 7f3c34a00700 1 -- start start 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.865+0000 7f3c34a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3010a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.865+0000 7f3c34a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3010aab0 con 0x7f3c3010a150 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.866+0000 7f3c2e800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3010a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.866+0000 7f3c2e800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3010a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45044/0 (socket says 172.21.15.103:45044) 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.866+0000 7f3c2e800700 1 -- 172.21.15.103:0/3385587264 learned_addr learned my addr 172.21.15.103:0/3385587264 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.867+0000 7f3c2e800700 1 -- 172.21.15.103:0/3385587264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3010af70 con 0x7f3c3010a150 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.867+0000 7f3c2e800700 1 --2- 172.21.15.103:0/3385587264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3010a570 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3c200098a0 tx=0x7f3c20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a115866494218a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.868+0000 7f3c2d400700 1 -- 172.21.15.103:0/3385587264 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c20004160 con 0x7f3c3010a150 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.868+0000 7f3c2d400700 1 -- 172.21.15.103:0/3385587264 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3c20004300 con 0x7f3c3010a150 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.868+0000 7f3c2d400700 1 -- 172.21.15.103:0/3385587264 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c20011b50 con 0x7f3c3010a150 2024-05-22T23:48:29.641 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.868+0000 7f3c34a00700 1 -- 172.21.15.103:0/3385587264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 msgr2=0x7f3c3010a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.868+0000 7f3c34a00700 1 --2- 172.21.15.103:0/3385587264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3010a570 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3c200098a0 tx=0x7f3c20009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.869+0000 7f3c34a00700 1 -- 172.21.15.103:0/3385587264 shutdown_connections 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.869+0000 7f3c34a00700 1 --2- 172.21.15.103:0/3385587264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3010a570 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.869+0000 7f3c34a00700 1 -- 172.21.15.103:0/3385587264 >> 172.21.15.103:0/3385587264 conn(0x7f3c3007d230 msgr2=0x7f3c3007d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.869+0000 7f3c34a00700 1 -- 172.21.15.103:0/3385587264 shutdown_connections 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.869+0000 7f3c34a00700 1 -- 172.21.15.103:0/3385587264 wait complete. 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.870+0000 7f3c34a00700 1 Processor -- start 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.870+0000 7f3c34a00700 1 -- start start 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.870+0000 7f3c34a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.870+0000 7f3c34a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c3010e230 con 0x7f3c3010a150 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.871+0000 7f3c2e800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:29.642 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.871+0000 7f3c2e800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3019dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45058/0 (socket says 172.21.15.103:45058) 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.871+0000 7f3c2e800700 1 -- 172.21.15.103:0/2437025934 learned_addr learned my addr 172.21.15.103:0/2437025934 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.871+0000 7f3c2e800700 1 -- 172.21.15.103:0/2437025934 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c3019e2a0 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.871+0000 7f3c2e800700 1 --2- 172.21.15.103:0/2437025934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3019dd60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3c20011b60 tx=0x7f3c20011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.872+0000 7f3c27400700 1 -- 172.21.15.103:0/2437025934 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c20003970 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.872+0000 7f3c27400700 1 -- 172.21.15.103:0/2437025934 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3c20003b10 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.872+0000 7f3c27400700 1 -- 172.21.15.103:0/2437025934 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3c20024610 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.872+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c3019e530 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.872+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c301a1220 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.872+0000 7f3c27400700 1 -- 172.21.15.103:0/2437025934 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3c20024c50 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.872+0000 7f3c27400700 1 -- 172.21.15.103:0/2437025934 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3c20011e00 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.873+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c30069490 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.876+0000 7f3c27400700 1 -- 172.21.15.103:0/2437025934 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3c20043dd0 con 0x7f3c3010a150 2024-05-22T23:48:29.643 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.952+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f3c301a13f0 con 0x7f3c3010a150 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.956+0000 7f3c27400700 1 -- 172.21.15.103:0/2437025934 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f3c20032030 con 0x7f3c3010a150 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.957+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 msgr2=0x7f3c3019dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.957+0000 7f3c34a00700 1 --2- 172.21.15.103:0/2437025934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3019dd60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f3c20011b60 tx=0x7f3c20011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.957+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 shutdown_connections 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.957+0000 7f3c34a00700 1 --2- 172.21.15.103:0/2437025934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c3010a150 0x7f3c3019dd60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.957+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 >> 172.21.15.103:0/2437025934 conn(0x7f3c3007d230 msgr2=0x7f3c30107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.958+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 shutdown_connections 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:28.958+0000 7f3c34a00700 1 -- 172.21.15.103:0/2437025934 wait complete. 2024-05-22T23:48:29.644 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-22T23:48:29.668 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-22T23:48:29.668 INFO:teuthology.orchestra.run.smithi103.stdout:Creating mgr... 2024-05-22T23:48:29.670 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-22T23:48:29.671 INFO:teuthology.orchestra.run.smithi103.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-22T23:48:30.108 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.x 2024-05-22T23:48:30.108 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.x.service: Unit ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.x.service not loaded. 2024-05-22T23:48:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:29 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2437025934' entity='client.admin' 2024-05-22T23:48:30.375 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968.target.wants/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@.service. 2024-05-22T23:48:30.774 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:30 smithi103 systemd[1]: Starting Ceph mgr.x for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:48:31.082 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:30 smithi103 podman[31578]: 2024-05-22 23:48:30.672751508 +0000 UTC m=+0.032070976 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 2024-05-22T23:48:31.082 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:30 smithi103 podman[31578]: 2024-05-22 23:48:30.783696264 +0000 UTC m=+0.143015749 container create d82c41646a04b592d6af0d70616f081d66eecc37e39a94f26853d0ba47446467 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-05-22T23:48:31.391 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-22T23:48:31.391 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-05-22T23:48:31.411 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-22T23:48:31.411 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-05-22T23:48:31.411 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-05-22T23:48:31.411 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to enable service . firewalld.service is not available 2024-05-22T23:48:31.437 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-22T23:48:31.437 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-05-22T23:48:31.458 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-22T23:48:31.458 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-05-22T23:48:31.458 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-05-22T23:48:31.458 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-22T23:48:31.459 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr to start... 2024-05-22T23:48:31.459 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr... 2024-05-22T23:48:31.582 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:31 smithi103 podman[31578]: 2024-05-22 23:48:31.161446011 +0000 UTC m=+0.520765500 container init d82c41646a04b592d6af0d70616f081d66eecc37e39a94f26853d0ba47446467 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-22T23:48:31.582 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:31 smithi103 podman[31578]: 2024-05-22 23:48:31.173335311 +0000 UTC m=+0.532654822 container start d82c41646a04b592d6af0d70616f081d66eecc37e39a94f26853d0ba47446467 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-22T23:48:31.582 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:31 smithi103 bash[31578]: d82c41646a04b592d6af0d70616f081d66eecc37e39a94f26853d0ba47446467 2024-05-22T23:48:31.582 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:31 smithi103 systemd[1]: Started Ceph mgr.x for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:48:31.582 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:31.410+0000 7fdd0ba52200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-22T23:48:31.928 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:31.670+0000 7fdd0ba52200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:48:31.928 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:31.809+0000 7fdd0ba52200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-22T23:48:32.566 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:32 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/218881505' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-22T23:48:32.567 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:32.232+0000 7fdd0ba52200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-22T23:48:32.567 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:32.351+0000 7fdd0ba52200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-22T23:48:32.889 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:32.889 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-05-22T23:48:32.889 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-22T23:48:32.889 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-05-22T23:48:32.889 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-22T23:48:32.889 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:32.890 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-22T23:48:32.891 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-22T23:48:32.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-22T23:48:32.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-22T23:48:32.897 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-22T23:48:32.898 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-22T23:48:32.899 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-05-22T23:48:21.339086+0000", 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-22T23:48:32.900 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-05-22T23:48:32.901 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.077+0000 7f72db200700 1 Processor -- start 2024-05-22T23:48:32.901 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.078+0000 7f72db200700 1 -- start start 2024-05-22T23:48:32.901 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.078+0000 7f72db200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d4108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:32.901 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.078+0000 7f72db200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72d41085e0 con 0x7f72d4107bf0 2024-05-22T23:48:32.901 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.079+0000 7f72d9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d4108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:32.901 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.079+0000 7f72d9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d4108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45098/0 (socket says 172.21.15.103:45098) 2024-05-22T23:48:32.901 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.079+0000 7f72d9e00700 1 -- 172.21.15.103:0/1375480822 learned_addr learned my addr 172.21.15.103:0/1375480822 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:32.901 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.079+0000 7f72d9e00700 1 -- 172.21.15.103:0/1375480822 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d4108df0 con 0x7f72d4107bf0 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.079+0000 7f72d9e00700 1 --2- 172.21.15.103:0/1375480822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d4108010 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f72c80098a0 tx=0x7f72c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81ebdae53f69e87 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.080+0000 7f72d8a00700 1 -- 172.21.15.103:0/1375480822 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72c8004160 con 0x7f72d4107bf0 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.080+0000 7f72d8a00700 1 -- 172.21.15.103:0/1375480822 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f72c8004300 con 0x7f72d4107bf0 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.080+0000 7f72d8a00700 1 -- 172.21.15.103:0/1375480822 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72c8011b50 con 0x7f72d4107bf0 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.080+0000 7f72db200700 1 -- 172.21.15.103:0/1375480822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 msgr2=0x7f72d4108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.080+0000 7f72db200700 1 --2- 172.21.15.103:0/1375480822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d4108010 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f72c80098a0 tx=0x7f72c8009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.080+0000 7f72db200700 1 -- 172.21.15.103:0/1375480822 shutdown_connections 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.080+0000 7f72db200700 1 --2- 172.21.15.103:0/1375480822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d4108010 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.080+0000 7f72db200700 1 -- 172.21.15.103:0/1375480822 >> 172.21.15.103:0/1375480822 conn(0x7f72d4103190 msgr2=0x7f72d41055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.081+0000 7f72db200700 1 -- 172.21.15.103:0/1375480822 shutdown_connections 2024-05-22T23:48:32.902 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.081+0000 7f72db200700 1 -- 172.21.15.103:0/1375480822 wait complete. 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.082+0000 7f72db200700 1 Processor -- start 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.082+0000 7f72db200700 1 -- start start 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.082+0000 7f72db200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d419bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.082+0000 7f72db200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72d410c1a0 con 0x7f72d4107bf0 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.082+0000 7f72d9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d419bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.083+0000 7f72d9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d419bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45110/0 (socket says 172.21.15.103:45110) 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.083+0000 7f72d9e00700 1 -- 172.21.15.103:0/218881505 learned_addr learned my addr 172.21.15.103:0/218881505 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.083+0000 7f72d9e00700 1 -- 172.21.15.103:0/218881505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72d419c0e0 con 0x7f72d4107bf0 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.083+0000 7f72d9e00700 1 --2- 172.21.15.103:0/218881505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d419bba0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f72c8011850 tx=0x7f72c8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.084+0000 7f72cea00700 1 -- 172.21.15.103:0/218881505 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72c8003af0 con 0x7f72d4107bf0 2024-05-22T23:48:32.903 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.084+0000 7f72cea00700 1 -- 172.21.15.103:0/218881505 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f72c8020070 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.084+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72d419c370 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.084+0000 7f72cea00700 1 -- 172.21.15.103:0/218881505 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f72c80195a0 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.084+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72d419f060 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.084+0000 7f72cea00700 1 -- 172.21.15.103:0/218881505 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f72c8019be0 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.084+0000 7f72cea00700 1 -- 172.21.15.103:0/218881505 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f72c8011e00 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.085+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72d4069490 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.088+0000 7f72cea00700 1 -- 172.21.15.103:0/218881505 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f72c8003c90 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.167+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f72d419f230 con 0x7f72d4107bf0 2024-05-22T23:48:32.904 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.167+0000 7f72cea00700 1 -- 172.21.15.103:0/218881505 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f72c8025460 con 0x7f72d4107bf0 2024-05-22T23:48:32.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.168+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 msgr2=0x7f72d419bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:32.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.168+0000 7f72db200700 1 --2- 172.21.15.103:0/218881505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d419bba0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f72c8011850 tx=0x7f72c8011c10 comp rx=0 tx=0).stop 2024-05-22T23:48:32.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.169+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 shutdown_connections 2024-05-22T23:48:32.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.169+0000 7f72db200700 1 --2- 172.21.15.103:0/218881505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72d4107bf0 0x7f72d419bba0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:32.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.169+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 >> 172.21.15.103:0/218881505 conn(0x7f72d4103190 msgr2=0x7f72d4103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:32.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.169+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 shutdown_connections 2024-05-22T23:48:32.905 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:32.169+0000 7f72db200700 1 -- 172.21.15.103:0/218881505 wait complete. 2024-05-22T23:48:32.905 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (1/15)... 2024-05-22T23:48:34.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:33.914+0000 7fdd0ba52200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-22T23:48:34.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:34.131+0000 7fdd0ba52200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-22T23:48:34.605 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:34.340+0000 7fdd0ba52200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-22T23:48:35.015 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:34.722+0000 7fdd0ba52200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-22T23:48:35.015 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:34.841+0000 7fdd0ba52200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-22T23:48:35.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:35.097+0000 7fdd0ba52200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-22T23:48:35.617 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:35.353+0000 7fdd0ba52200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:48:36.003 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:35 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/594536792' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-22T23:48:36.328 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:36 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:36.162+0000 7fdd0ba52200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-22T23:48:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-05-22T23:48:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-22T23:48:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-05-22T23:48:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-22T23:48:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-22T23:48:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-22T23:48:36.345 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-22T23:48:36.346 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:36.347 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-22T23:48:36.347 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:36.347 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-22T23:48:36.347 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-22T23:48:36.355 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-22T23:48:36.356 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-05-22T23:48:21.339086+0000", 2024-05-22T23:48:36.357 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.524+0000 7fe3b2200700 1 Processor -- start 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.524+0000 7fe3b2200700 1 -- start start 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.524+0000 7fe3b2200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac1080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.525+0000 7fe3b2200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac1085e0 con 0x7fe3ac107c80 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.525+0000 7fe3b0e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac1080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.525+0000 7fe3b0e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac1080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60638/0 (socket says 172.21.15.103:60638) 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.525+0000 7fe3b0e00700 1 -- 172.21.15.103:0/2534180662 learned_addr learned my addr 172.21.15.103:0/2534180662 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:36.358 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.525+0000 7fe3b0e00700 1 -- 172.21.15.103:0/2534180662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ac108760 con 0x7fe3ac107c80 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.526+0000 7fe3b0e00700 1 --2- 172.21.15.103:0/2534180662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac1080a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00098a0 tx=0x7fe3a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518940c14dd116ab server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.526+0000 7fe3ab400700 1 -- 172.21.15.103:0/2534180662 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a0004160 con 0x7fe3ac107c80 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.526+0000 7fe3ab400700 1 -- 172.21.15.103:0/2534180662 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe3a0004300 con 0x7fe3ac107c80 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.526+0000 7fe3ab400700 1 -- 172.21.15.103:0/2534180662 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a0011b50 con 0x7fe3ac107c80 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.527+0000 7fe3b2200700 1 -- 172.21.15.103:0/2534180662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 msgr2=0x7fe3ac1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.527+0000 7fe3b2200700 1 --2- 172.21.15.103:0/2534180662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac1080a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00098a0 tx=0x7fe3a0009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.527+0000 7fe3b2200700 1 -- 172.21.15.103:0/2534180662 shutdown_connections 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.527+0000 7fe3b2200700 1 --2- 172.21.15.103:0/2534180662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac1080a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.527+0000 7fe3b2200700 1 -- 172.21.15.103:0/2534180662 >> 172.21.15.103:0/2534180662 conn(0x7fe3ac103300 msgr2=0x7fe3ac105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:36.359 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.527+0000 7fe3b2200700 1 -- 172.21.15.103:0/2534180662 shutdown_connections 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.527+0000 7fe3b2200700 1 -- 172.21.15.103:0/2534180662 wait complete. 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.528+0000 7fe3b2200700 1 Processor -- start 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.528+0000 7fe3b2200700 1 -- start start 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.528+0000 7fe3b2200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac19bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.528+0000 7fe3b2200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe3ac10c0f0 con 0x7fe3ac107c80 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.529+0000 7fe3b0e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac19bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.529+0000 7fe3b0e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac19bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60654/0 (socket says 172.21.15.103:60654) 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.529+0000 7fe3b0e00700 1 -- 172.21.15.103:0/594536792 learned_addr learned my addr 172.21.15.103:0/594536792 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.529+0000 7fe3b0e00700 1 -- 172.21.15.103:0/594536792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3ac19c090 con 0x7fe3ac107c80 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.529+0000 7fe3b0e00700 1 --2- 172.21.15.103:0/594536792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac19bb50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0011b60 tx=0x7fe3a0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.530+0000 7fe3a9600700 1 -- 172.21.15.103:0/594536792 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a0003970 con 0x7fe3ac107c80 2024-05-22T23:48:36.360 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.530+0000 7fe3a9600700 1 -- 172.21.15.103:0/594536792 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe3a0003b10 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.530+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3ac19c320 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.530+0000 7fe3a9600700 1 -- 172.21.15.103:0/594536792 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe3a0024610 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.530+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3ac19f010 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.530+0000 7fe3a9600700 1 -- 172.21.15.103:0/594536792 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe3a0024c50 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.530+0000 7fe3a9600700 1 -- 172.21.15.103:0/594536792 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe3a0011e00 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.531+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3ac069490 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.533+0000 7fe3a9600700 1 -- 172.21.15.103:0/594536792 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe3a001a9f0 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.612+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe3ac19f1e0 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.613+0000 7fe3a9600700 1 -- 172.21.15.103:0/594536792 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe3a001a430 con 0x7fe3ac107c80 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.614+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 msgr2=0x7fe3ac19bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.614+0000 7fe3b2200700 1 --2- 172.21.15.103:0/594536792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac19bb50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0011b60 tx=0x7fe3a0011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:36.361 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.615+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 shutdown_connections 2024-05-22T23:48:36.362 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.615+0000 7fe3b2200700 1 --2- 172.21.15.103:0/594536792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe3ac107c80 0x7fe3ac19bb50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:36.362 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.615+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 >> 172.21.15.103:0/594536792 conn(0x7fe3ac103300 msgr2=0x7fe3ac103d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:36.362 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.616+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 shutdown_connections 2024-05-22T23:48:36.362 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:35.616+0000 7fe3b2200700 1 -- 172.21.15.103:0/594536792 wait complete. 2024-05-22T23:48:36.362 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (2/15)... 2024-05-22T23:48:37.081 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:36 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:36.731+0000 7fdd0ba52200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-22T23:48:37.081 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:36 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:36.864+0000 7fdd0ba52200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-22T23:48:37.581 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:37 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:37.129+0000 7fdd0ba52200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-22T23:48:38.081 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:37 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:37.770+0000 7fdd0ba52200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-22T23:48:38.081 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:37 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:37.893+0000 7fdd0ba52200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-22T23:48:38.510 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:38.214+0000 7fdd0ba52200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-22T23:48:38.510 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:38.408+0000 7fdd0ba52200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-22T23:48:38.831 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:38.534+0000 7fdd0ba52200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-22T23:48:38.831 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:38.691+0000 7fdd0ba52200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-22T23:48:39.105 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:39.066+0000 7fdd0ba52200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-22T23:48:39.449 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:39 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1359730684' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-22T23:48:39.768 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:39.768 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-05-22T23:48:39.768 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-22T23:48:39.768 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-05-22T23:48:39.768 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-22T23:48:39.770 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-22T23:48:39.770 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-22T23:48:39.770 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:39.770 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-22T23:48:39.771 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-22T23:48:39.771 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-05-22T23:48:39.771 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:39.771 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-22T23:48:39.771 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-05-22T23:48:39.771 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:39.771 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-05-22T23:48:39.771 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-22T23:48:39.772 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:39.772 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-22T23:48:39.772 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-22T23:48:39.772 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:39.772 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-22T23:48:39.772 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-22T23:48:39.773 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-22T23:48:39.774 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-22T23:48:39.775 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-05-22T23:48:21.339086+0000", 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.008+0000 7ff94f200700 1 Processor -- start 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.008+0000 7ff94f200700 1 -- start start 2024-05-22T23:48:39.776 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.008+0000 7ff94f200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94807c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.008+0000 7ff94f200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff94807c8b0 con 0x7ff94807de80 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.009+0000 7ff94de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94807c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.009+0000 7ff94de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94807c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60660/0 (socket says 172.21.15.103:60660) 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.009+0000 7ff94de00700 1 -- 172.21.15.103:0/1174730503 learned_addr learned my addr 172.21.15.103:0/1174730503 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.009+0000 7ff94de00700 1 -- 172.21.15.103:0/1174730503 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94807ca30 con 0x7ff94807de80 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.010+0000 7ff94de00700 1 --2- 172.21.15.103:0/1174730503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94807c2e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff93c0098a0 tx=0x7ff93c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc3bc6b24008e4b server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.010+0000 7ff94ca00700 1 -- 172.21.15.103:0/1174730503 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff93c004160 con 0x7ff94807de80 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.010+0000 7ff94ca00700 1 -- 172.21.15.103:0/1174730503 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff93c004300 con 0x7ff94807de80 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.010+0000 7ff94ca00700 1 -- 172.21.15.103:0/1174730503 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff93c011b50 con 0x7ff94807de80 2024-05-22T23:48:39.777 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.011+0000 7ff94f200700 1 -- 172.21.15.103:0/1174730503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 msgr2=0x7ff94807c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:39.778 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.011+0000 7ff94f200700 1 --2- 172.21.15.103:0/1174730503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94807c2e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7ff93c0098a0 tx=0x7ff93c009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:39.778 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.011+0000 7ff94f200700 1 -- 172.21.15.103:0/1174730503 shutdown_connections 2024-05-22T23:48:39.778 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.011+0000 7ff94f200700 1 --2- 172.21.15.103:0/1174730503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94807c2e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:39.778 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.011+0000 7ff94f200700 1 -- 172.21.15.103:0/1174730503 >> 172.21.15.103:0/1174730503 conn(0x7ff948103190 msgr2=0x7ff9481055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:39.778 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.011+0000 7ff94f200700 1 -- 172.21.15.103:0/1174730503 shutdown_connections 2024-05-22T23:48:39.778 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.011+0000 7ff94f200700 1 -- 172.21.15.103:0/1174730503 wait complete. 2024-05-22T23:48:39.778 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.012+0000 7ff94f200700 1 Processor -- start 2024-05-22T23:48:39.779 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.012+0000 7ff94f200700 1 -- start start 2024-05-22T23:48:39.779 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.013+0000 7ff94f200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94819dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:39.779 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.013+0000 7ff94f200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff94810e220 con 0x7ff94807de80 2024-05-22T23:48:39.779 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.013+0000 7ff94de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94819dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:39.779 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.013+0000 7ff94de00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94819dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60670/0 (socket says 172.21.15.103:60670) 2024-05-22T23:48:39.779 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.013+0000 7ff94de00700 1 -- 172.21.15.103:0/1359730684 learned_addr learned my addr 172.21.15.103:0/1359730684 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:39.780 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.013+0000 7ff94de00700 1 -- 172.21.15.103:0/1359730684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff94819e2d0 con 0x7ff94807de80 2024-05-22T23:48:39.780 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.013+0000 7ff94de00700 1 --2- 172.21.15.103:0/1359730684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94819dd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff93c011b60 tx=0x7ff93c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:39.780 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.014+0000 7ff942a00700 1 -- 172.21.15.103:0/1359730684 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff93c003970 con 0x7ff94807de80 2024-05-22T23:48:39.780 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.014+0000 7ff942a00700 1 -- 172.21.15.103:0/1359730684 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff93c003b10 con 0x7ff94807de80 2024-05-22T23:48:39.780 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.014+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff94819e560 con 0x7ff94807de80 2024-05-22T23:48:39.780 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.014+0000 7ff942a00700 1 -- 172.21.15.103:0/1359730684 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff93c024610 con 0x7ff94807de80 2024-05-22T23:48:39.780 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.014+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9481a1210 con 0x7ff94807de80 2024-05-22T23:48:39.781 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.014+0000 7ff942a00700 1 -- 172.21.15.103:0/1359730684 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff93c024c50 con 0x7ff94807de80 2024-05-22T23:48:39.781 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.014+0000 7ff942a00700 1 -- 172.21.15.103:0/1359730684 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff93c011e00 con 0x7ff94807de80 2024-05-22T23:48:39.781 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.015+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff948069490 con 0x7ff94807de80 2024-05-22T23:48:39.781 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.018+0000 7ff942a00700 1 -- 172.21.15.103:0/1359730684 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff93c043dd0 con 0x7ff94807de80 2024-05-22T23:48:39.781 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.096+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff9481a1480 con 0x7ff94807de80 2024-05-22T23:48:39.781 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.097+0000 7ff942a00700 1 -- 172.21.15.103:0/1359730684 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7ff93c032030 con 0x7ff94807de80 2024-05-22T23:48:39.781 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.098+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 msgr2=0x7ff94819dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:39.782 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.098+0000 7ff94f200700 1 --2- 172.21.15.103:0/1359730684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94819dd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff93c011b60 tx=0x7ff93c011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:39.782 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.098+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 shutdown_connections 2024-05-22T23:48:39.782 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.098+0000 7ff94f200700 1 --2- 172.21.15.103:0/1359730684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff94807de80 0x7ff94819dd90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:39.782 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.099+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 >> 172.21.15.103:0/1359730684 conn(0x7ff948103190 msgr2=0x7ff948103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:39.782 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.099+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 shutdown_connections 2024-05-22T23:48:39.782 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:39.099+0000 7ff94f200700 1 -- 172.21.15.103:0/1359730684 wait complete. 2024-05-22T23:48:39.782 INFO:teuthology.orchestra.run.smithi103.stdout:mgr not available, waiting (3/15)... 2024-05-22T23:48:40.081 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:39.844+0000 7fdd0ba52200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-22T23:48:40.582 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:40 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:40.127+0000 7fdd0ba52200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-22T23:48:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: Activating manager daemon x 2024-05-22T23:48:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: mgrmap e2: x(active, starting, since 0.010597s) 2024-05-22T23:48:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-22T23:48:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-22T23:48:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-22T23:48:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-22T23:48:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-22T23:48:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-22T23:48:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:48:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:40 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-22T23:48:41.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:41 smithi103 ceph-mon[31370]: Manager daemon x is now available 2024-05-22T23:48:41.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:41 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-22T23:48:41.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:41 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-22T23:48:41.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:41 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' 2024-05-22T23:48:41.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:41 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' 2024-05-22T23:48:41.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:41 smithi103 ceph-mon[31370]: from='mgr.14100 172.21.15.103:0/2842282365' entity='mgr.x' 2024-05-22T23:48:42.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:42 smithi103 ceph-mon[31370]: mgrmap e3: x(active, since 1.04224s) 2024-05-22T23:48:43.456 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:43 smithi103 ceph-mon[31370]: mgrmap e4: x(active, since 2s) 2024-05-22T23:48:43.457 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:43 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1496862040' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-22T23:48:43.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:43.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "health": { 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 0 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "a" 2024-05-22T23:48:43.471 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-22T23:48:43.472 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-22T23:48:43.479 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-22T23:48:43.479 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:43.480 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "restful" 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ], 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-22T23:48:43.481 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "modified": "2024-05-22T23:48:21.339086+0000", 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout }, 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.361+0000 7f630be00700 1 Processor -- start 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.361+0000 7f630be00700 1 -- start start 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.362+0000 7f630be00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c10a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.362+0000 7f630be00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f630c10a800 con 0x7f630c109ea0 2024-05-22T23:48:43.482 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.362+0000 7f630aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c10a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.362+0000 7f630aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c10a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60746/0 (socket says 172.21.15.103:60746) 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.362+0000 7f630aa00700 1 -- 172.21.15.103:0/3542453543 learned_addr learned my addr 172.21.15.103:0/3542453543 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.363+0000 7f630aa00700 1 -- 172.21.15.103:0/3542453543 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630c10afb0 con 0x7f630c109ea0 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.363+0000 7f630aa00700 1 --2- 172.21.15.103:0/3542453543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c10a2c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0098a0 tx=0x7f62fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88c64f7b04014265 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.363+0000 7f6309600700 1 -- 172.21.15.103:0/3542453543 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62fc004160 con 0x7f630c109ea0 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.363+0000 7f6309600700 1 -- 172.21.15.103:0/3542453543 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f62fc004300 con 0x7f630c109ea0 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.363+0000 7f6309600700 1 -- 172.21.15.103:0/3542453543 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62fc011b50 con 0x7f630c109ea0 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.364+0000 7f630be00700 1 -- 172.21.15.103:0/3542453543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 msgr2=0x7f630c10a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.364+0000 7f630be00700 1 --2- 172.21.15.103:0/3542453543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c10a2c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0098a0 tx=0x7f62fc009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.364+0000 7f630be00700 1 -- 172.21.15.103:0/3542453543 shutdown_connections 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.364+0000 7f630be00700 1 --2- 172.21.15.103:0/3542453543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c10a2c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.364+0000 7f630be00700 1 -- 172.21.15.103:0/3542453543 >> 172.21.15.103:0/3542453543 conn(0x7f630c07d260 msgr2=0x7f630c07d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:43.483 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.364+0000 7f630be00700 1 -- 172.21.15.103:0/3542453543 shutdown_connections 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.364+0000 7f630be00700 1 -- 172.21.15.103:0/3542453543 wait complete. 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.365+0000 7f630be00700 1 Processor -- start 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.365+0000 7f630be00700 1 -- start start 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.366+0000 7f630be00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.366+0000 7f630be00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f630c10e3f0 con 0x7f630c109ea0 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.366+0000 7f630aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.366+0000 7f630aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c19dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:60752/0 (socket says 172.21.15.103:60752) 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.366+0000 7f630aa00700 1 -- 172.21.15.103:0/1496862040 learned_addr learned my addr 172.21.15.103:0/1496862040 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.366+0000 7f630aa00700 1 -- 172.21.15.103:0/1496862040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f630c19e260 con 0x7f630c109ea0 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.367+0000 7f630aa00700 1 --2- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c19dd20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f62fc011b60 tx=0x7f62fc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.367+0000 7f6303400700 1 -- 172.21.15.103:0/1496862040 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62fc003970 con 0x7f630c109ea0 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.367+0000 7f6303400700 1 -- 172.21.15.103:0/1496862040 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f62fc003b10 con 0x7f630c109ea0 2024-05-22T23:48:43.484 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.367+0000 7f6303400700 1 -- 172.21.15.103:0/1496862040 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f62fc024610 con 0x7f630c109ea0 2024-05-22T23:48:43.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.367+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f630c19e4f0 con 0x7f630c109ea0 2024-05-22T23:48:43.485 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.367+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f630c1a11d0 con 0x7f630c109ea0 2024-05-22T23:48:43.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.368+0000 7f6303400700 1 -- 172.21.15.103:0/1496862040 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f62fc01a9f0 con 0x7f630c109ea0 2024-05-22T23:48:43.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.368+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f630c069490 con 0x7f630c109ea0 2024-05-22T23:48:43.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.369+0000 7f6303400700 1 --2- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f62f0039110 0x7f62f003b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:43.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.369+0000 7f6303400700 1 -- 172.21.15.103:0/1496862040 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f62fc011e00 con 0x7f630c109ea0 2024-05-22T23:48:43.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.369+0000 7f630a000700 1 --2- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f62f0039110 0x7f62f003b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:43.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.370+0000 7f630a000700 1 --2- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f62f0039110 0x7f62f003b5d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f62f8006fd0 tx=0x7f62f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:43.487 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.375+0000 7f6303400700 1 -- 172.21.15.103:0/1496862040 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f62fc016260 con 0x7f630c109ea0 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.689+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f630c1a13a0 con 0x7f630c109ea0 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.690+0000 7f6303400700 1 -- 172.21.15.103:0/1496862040 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f630c1a13a0 con 0x7f630c109ea0 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.693+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f62f0039110 msgr2=0x7f62f003b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.693+0000 7f630be00700 1 --2- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f62f0039110 0x7f62f003b5d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f62f8006fd0 tx=0x7f62f8006e40 comp rx=0 tx=0).stop 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.693+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 msgr2=0x7f630c19dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.693+0000 7f630be00700 1 --2- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c19dd20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f62fc011b60 tx=0x7f62fc011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.693+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 shutdown_connections 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.693+0000 7f630be00700 1 --2- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f62f0039110 0x7f62f003b5d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.693+0000 7f630be00700 1 --2- 172.21.15.103:0/1496862040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f630c109ea0 0x7f630c19dd20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.693+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 >> 172.21.15.103:0/1496862040 conn(0x7f630c07d260 msgr2=0x7f630c1073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.694+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 shutdown_connections 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:42.694+0000 7f630be00700 1 -- 172.21.15.103:0/1496862040 wait complete. 2024-05-22T23:48:43.488 INFO:teuthology.orchestra.run.smithi103.stdout:mgr is available 2024-05-22T23:48:44.691 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:44 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3646546317' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-22T23:48:45.045 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:45.045 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [global] 2024-05-22T23:48:45.045 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout fsid = 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:48:45.045 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-22T23:48:45.045 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-22T23:48:45.045 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-22T23:48:45.045 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout [osd] 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-22T23:48:45.046 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-22T23:48:45.049 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-22T23:48:45.049 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.101+0000 7fec0b800700 1 Processor -- start 2024-05-22T23:48:45.049 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.101+0000 7fec0b800700 1 -- start start 2024-05-22T23:48:45.049 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.101+0000 7fec0b800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec04108220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:45.049 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.101+0000 7fec0b800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec04108760 con 0x7fec04107e00 2024-05-22T23:48:45.049 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.102+0000 7fec08e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec04108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.102+0000 7fec08e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec04108220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42874/0 (socket says 172.21.15.103:42874) 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.102+0000 7fec08e00700 1 -- 172.21.15.103:0/1071618998 learned_addr learned my addr 172.21.15.103:0/1071618998 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.102+0000 7fec08e00700 1 -- 172.21.15.103:0/1071618998 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec04108f60 con 0x7fec04107e00 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.103+0000 7fec08e00700 1 --2- 172.21.15.103:0/1071618998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec04108220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7febf40098a0 tx=0x7febf4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5021e978844a515 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.103+0000 7fec03e00700 1 -- 172.21.15.103:0/1071618998 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febf4004160 con 0x7fec04107e00 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.103+0000 7fec03e00700 1 -- 172.21.15.103:0/1071618998 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7febf4004300 con 0x7fec04107e00 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.103+0000 7fec03e00700 1 -- 172.21.15.103:0/1071618998 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febf4011b50 con 0x7fec04107e00 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.103+0000 7fec0b800700 1 -- 172.21.15.103:0/1071618998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 msgr2=0x7fec04108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.103+0000 7fec0b800700 1 --2- 172.21.15.103:0/1071618998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec04108220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7febf40098a0 tx=0x7febf4009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.104+0000 7fec0b800700 1 -- 172.21.15.103:0/1071618998 shutdown_connections 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.104+0000 7fec0b800700 1 --2- 172.21.15.103:0/1071618998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec04108220 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:45.050 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.104+0000 7fec0b800700 1 -- 172.21.15.103:0/1071618998 >> 172.21.15.103:0/1071618998 conn(0x7fec041033c0 msgr2=0x7fec041057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.104+0000 7fec0b800700 1 -- 172.21.15.103:0/1071618998 shutdown_connections 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.104+0000 7fec0b800700 1 -- 172.21.15.103:0/1071618998 wait complete. 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.105+0000 7fec0b800700 1 Processor -- start 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.105+0000 7fec0b800700 1 -- start start 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.105+0000 7fec0b800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec0407db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.105+0000 7fec0b800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec0410c3a0 con 0x7fec04107e00 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.105+0000 7fec08e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec0407db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.106+0000 7fec08e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec0407db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42878/0 (socket says 172.21.15.103:42878) 2024-05-22T23:48:45.051 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.106+0000 7fec08e00700 1 -- 172.21.15.103:0/3646546317 learned_addr learned my addr 172.21.15.103:0/3646546317 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:45.054 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.106+0000 7fec08e00700 1 -- 172.21.15.103:0/3646546317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0407e0b0 con 0x7fec04107e00 2024-05-22T23:48:45.054 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.106+0000 7fec08e00700 1 --2- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec0407db70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7febf4011b60 tx=0x7febf4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:45.054 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.106+0000 7fec02000700 1 -- 172.21.15.103:0/3646546317 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febf4003b20 con 0x7fec04107e00 2024-05-22T23:48:45.054 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.107+0000 7fec02000700 1 -- 172.21.15.103:0/3646546317 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7febf4003cc0 con 0x7fec04107e00 2024-05-22T23:48:45.054 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.107+0000 7fec02000700 1 -- 172.21.15.103:0/3646546317 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7febf4024840 con 0x7fec04107e00 2024-05-22T23:48:45.054 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.107+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0407c2a0 con 0x7fec04107e00 2024-05-22T23:48:45.054 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.107+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec0407c7e0 con 0x7fec04107e00 2024-05-22T23:48:45.054 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.108+0000 7fec02000700 1 -- 172.21.15.103:0/3646546317 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7febf4022020 con 0x7fec04107e00 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.108+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec040639c0 con 0x7fec04107e00 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.108+0000 7fec02000700 1 --2- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7febe8039110 0x7febe803b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.108+0000 7fec02000700 1 -- 172.21.15.103:0/3646546317 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7febf4011e00 con 0x7fec04107e00 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.108+0000 7febfbe00700 1 --2- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7febe8039110 0x7febe803b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.109+0000 7febfbe00700 1 --2- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7febe8039110 0x7febe803b5d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7febec006fd0 tx=0x7febec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.114+0000 7fec02000700 1 -- 172.21.15.103:0/3646546317 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7febf404cac0 con 0x7fec04107e00 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.337+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fec0407ca60 con 0x7fec04107e00 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.339+0000 7fec02000700 1 -- 172.21.15.103:0/3646546317 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7febf401a7b0 con 0x7fec04107e00 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7febe8039110 msgr2=0x7febe803b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 --2- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7febe8039110 0x7febe803b5d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7febec006fd0 tx=0x7febec006e40 comp rx=0 tx=0).stop 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 msgr2=0x7fec0407db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 --2- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec0407db70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7febf4011b60 tx=0x7febf4011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:45.055 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 shutdown_connections 2024-05-22T23:48:45.056 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 --2- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7febe8039110 0x7febe803b5d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:45.056 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 --2- 172.21.15.103:0/3646546317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec04107e00 0x7fec0407db70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:45.056 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 >> 172.21.15.103:0/3646546317 conn(0x7fec041033c0 msgr2=0x7fec04103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:45.056 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 shutdown_connections 2024-05-22T23:48:45.056 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:44.342+0000 7fec0b800700 1 -- 172.21.15.103:0/3646546317 wait complete. 2024-05-22T23:48:45.056 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling cephadm module... 2024-05-22T23:48:46.260 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:45 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/199224268' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-22T23:48:46.260 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: ignoring --setuser ceph since I am not root 2024-05-22T23:48:46.260 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: ignoring --setgroup ceph since I am not root 2024-05-22T23:48:46.581 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:46.255+0000 7f86ef4c8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-22T23:48:46.581 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:46.500+0000 7f86ef4c8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:48:46.708 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.620+0000 7f2272600700 1 Processor -- start 2024-05-22T23:48:46.708 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.621+0000 7f2272600700 1 -- start start 2024-05-22T23:48:46.708 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.621+0000 7f2272600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c108270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:46.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.621+0000 7f2272600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f226c1087b0 con 0x7f226c107e50 2024-05-22T23:48:46.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.621+0000 7f226b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:46.714 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.621+0000 7f226b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42882/0 (socket says 172.21.15.103:42882) 2024-05-22T23:48:46.714 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.621+0000 7f226b400700 1 -- 172.21.15.103:0/2844514709 learned_addr learned my addr 172.21.15.103:0/2844514709 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:46.714 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.622+0000 7f226b400700 1 -- 172.21.15.103:0/2844514709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226c108fb0 con 0x7f226c107e50 2024-05-22T23:48:46.714 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.622+0000 7f226b400700 1 --2- 172.21.15.103:0/2844514709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c108270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f225c0098a0 tx=0x7f225c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=694bb752aeb3e999 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.622+0000 7f226a000700 1 -- 172.21.15.103:0/2844514709 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f225c004160 con 0x7f226c107e50 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.622+0000 7f226a000700 1 -- 172.21.15.103:0/2844514709 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f225c004300 con 0x7f226c107e50 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.622+0000 7f226a000700 1 -- 172.21.15.103:0/2844514709 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f225c011b50 con 0x7f226c107e50 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.623+0000 7f2272600700 1 -- 172.21.15.103:0/2844514709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 msgr2=0x7f226c108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.623+0000 7f2272600700 1 --2- 172.21.15.103:0/2844514709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c108270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f225c0098a0 tx=0x7f225c009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.623+0000 7f2272600700 1 -- 172.21.15.103:0/2844514709 shutdown_connections 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.623+0000 7f2272600700 1 --2- 172.21.15.103:0/2844514709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c108270 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.623+0000 7f2272600700 1 -- 172.21.15.103:0/2844514709 >> 172.21.15.103:0/2844514709 conn(0x7f226c1033f0 msgr2=0x7f226c105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.623+0000 7f2272600700 1 -- 172.21.15.103:0/2844514709 shutdown_connections 2024-05-22T23:48:46.715 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.623+0000 7f2272600700 1 -- 172.21.15.103:0/2844514709 wait complete. 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.624+0000 7f2272600700 1 Processor -- start 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.624+0000 7f2272600700 1 -- start start 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.625+0000 7f2272600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c19de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.625+0000 7f2272600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f226c10c270 con 0x7f226c107e50 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.625+0000 7f226b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c19de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.625+0000 7f226b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c19de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42888/0 (socket says 172.21.15.103:42888) 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.625+0000 7f226b400700 1 -- 172.21.15.103:0/199224268 learned_addr learned my addr 172.21.15.103:0/199224268 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.625+0000 7f226b400700 1 -- 172.21.15.103:0/199224268 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226c19e3d0 con 0x7f226c107e50 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.626+0000 7f226b400700 1 --2- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c19de90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f225c011b60 tx=0x7f225c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.626+0000 7f2263e00700 1 -- 172.21.15.103:0/199224268 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f225c003b20 con 0x7f226c107e50 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.626+0000 7f2263e00700 1 -- 172.21.15.103:0/199224268 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f225c003cc0 con 0x7f226c107e50 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.626+0000 7f2263e00700 1 -- 172.21.15.103:0/199224268 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f225c024840 con 0x7f226c107e50 2024-05-22T23:48:46.716 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.626+0000 7f2272600700 1 -- 172.21.15.103:0/199224268 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f226c19e660 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.626+0000 7f2272600700 1 -- 172.21.15.103:0/199224268 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f226c1a1350 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.627+0000 7f2263e00700 1 -- 172.21.15.103:0/199224268 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f225c022020 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.627+0000 7f2261600700 1 -- 172.21.15.103:0/199224268 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2244005330 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.628+0000 7f2263e00700 1 --2- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f22500390c0 0x7f225003b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.628+0000 7f2263e00700 1 -- 172.21.15.103:0/199224268 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f225c011e00 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.628+0000 7f226aa00700 1 --2- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f22500390c0 0x7f225003b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.629+0000 7f226aa00700 1 --2- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f22500390c0 0x7f225003b580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f2258006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.633+0000 7f2263e00700 1 -- 172.21.15.103:0/199224268 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f225c024aa0 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.914+0000 7f2261600700 1 -- 172.21.15.103:0/199224268 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f2244005620 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.967+0000 7f2263e00700 1 -- 172.21.15.103:0/199224268 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f225c04cb60 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.970+0000 7f2263e00700 1 -- 172.21.15.103:0/199224268 <== mon.0 v2:172.21.15.103:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f225c01a950 con 0x7f226c107e50 2024-05-22T23:48:46.717 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.971+0000 7f2261600700 1 -- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f22500390c0 msgr2=0x7f225003b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.971+0000 7f2261600700 1 --2- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f22500390c0 0x7f225003b580 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2258006fd0 tx=0x7f2258006e40 comp rx=0 tx=0).stop 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.971+0000 7f2261600700 1 -- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 msgr2=0x7f226c19de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.971+0000 7f2261600700 1 --2- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c19de90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f225c011b60 tx=0x7f225c011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.972+0000 7f2261600700 1 -- 172.21.15.103:0/199224268 shutdown_connections 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.972+0000 7f2261600700 1 --2- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f22500390c0 0x7f225003b580 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.972+0000 7f2261600700 1 --2- 172.21.15.103:0/199224268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f226c107e50 0x7f226c19de90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.972+0000 7f2261600700 1 -- 172.21.15.103:0/199224268 >> 172.21.15.103:0/199224268 conn(0x7f226c1033f0 msgr2=0x7f226c103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.972+0000 7f2261600700 1 -- 172.21.15.103:0/199224268 shutdown_connections 2024-05-22T23:48:46.718 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:45.973+0000 7f2261600700 1 -- 172.21.15.103:0/199224268 wait complete. 2024-05-22T23:48:46.980 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:46.634+0000 7f86ef4c8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-22T23:48:47.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:46 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/199224268' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-22T23:48:47.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:46 smithi103 ceph-mon[31370]: mgrmap e5: x(active, since 5s) 2024-05-22T23:48:47.297 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:47 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:47.037+0000 7f86ef4c8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-22T23:48:47.297 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:47 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:47.149+0000 7f86ef4c8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-22T23:48:48.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:47 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/733260297' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-22T23:48:48.422 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-05-22T23:48:48.423 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-22T23:48:48.423 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-22T23:48:48.423 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-22T23:48:48.423 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-22T23:48:48.423 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-05-22T23:48:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.356+0000 7fe825a00700 1 Processor -- start 2024-05-22T23:48:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.356+0000 7fe825a00700 1 -- start start 2024-05-22T23:48:48.427 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.357+0000 7fe825a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82007c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.357+0000 7fe825a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe82007c890 con 0x7fe82007de60 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.357+0000 7fe81ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82007c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.357+0000 7fe81ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82007c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42924/0 (socket says 172.21.15.103:42924) 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.357+0000 7fe81ea00700 1 -- 172.21.15.103:0/2057237327 learned_addr learned my addr 172.21.15.103:0/2057237327 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.358+0000 7fe81ea00700 1 -- 172.21.15.103:0/2057237327 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82007ca10 con 0x7fe82007de60 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.358+0000 7fe81ea00700 1 --2- 172.21.15.103:0/2057237327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82007c2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe8100098a0 tx=0x7fe810009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b01fa7351bce3d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.358+0000 7fe81d600700 1 -- 172.21.15.103:0/2057237327 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe810004160 con 0x7fe82007de60 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.358+0000 7fe81d600700 1 -- 172.21.15.103:0/2057237327 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe810004300 con 0x7fe82007de60 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.358+0000 7fe81d600700 1 -- 172.21.15.103:0/2057237327 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe810011b50 con 0x7fe82007de60 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.359+0000 7fe825a00700 1 -- 172.21.15.103:0/2057237327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 msgr2=0x7fe82007c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:48.428 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.359+0000 7fe825a00700 1 --2- 172.21.15.103:0/2057237327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82007c2c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe8100098a0 tx=0x7fe810009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.359+0000 7fe825a00700 1 -- 172.21.15.103:0/2057237327 shutdown_connections 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.359+0000 7fe825a00700 1 --2- 172.21.15.103:0/2057237327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82007c2c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.359+0000 7fe825a00700 1 -- 172.21.15.103:0/2057237327 >> 172.21.15.103:0/2057237327 conn(0x7fe8201033c0 msgr2=0x7fe8201057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.360+0000 7fe825a00700 1 -- 172.21.15.103:0/2057237327 shutdown_connections 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.360+0000 7fe825a00700 1 -- 172.21.15.103:0/2057237327 wait complete. 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.361+0000 7fe825a00700 1 Processor -- start 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.361+0000 7fe825a00700 1 -- start start 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.361+0000 7fe825a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.361+0000 7fe825a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe82010e3f0 con 0x7fe82007de60 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.361+0000 7fe81ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.361+0000 7fe81ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82019df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42936/0 (socket says 172.21.15.103:42936) 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.361+0000 7fe81ea00700 1 -- 172.21.15.103:0/733260297 learned_addr learned my addr 172.21.15.103:0/733260297 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:48.429 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.362+0000 7fe81ea00700 1 -- 172.21.15.103:0/733260297 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82019e440 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.362+0000 7fe81ea00700 1 --2- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82019df00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe810011b60 tx=0x7fe810011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.362+0000 7fe817400700 1 -- 172.21.15.103:0/733260297 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe810003b20 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.362+0000 7fe817400700 1 -- 172.21.15.103:0/733260297 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe810003cc0 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.362+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe82019e6d0 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.362+0000 7fe817400700 1 -- 172.21.15.103:0/733260297 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe810024840 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.363+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8201a13c0 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.364+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8200639c0 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.364+0000 7fe817400700 1 -- 172.21.15.103:0/733260297 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7fe810022020 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.364+0000 7fe817400700 1 --2- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7fe804039160 0x7fe80403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.364+0000 7fe817400700 1 -- 172.21.15.103:0/733260297 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe810011e00 con 0x7fe82007de60 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.364+0000 7fe81e000700 1 -- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7fe804039160 msgr2=0x7fe80403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3495336128 2024-05-22T23:48:48.430 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.365+0000 7fe81e000700 1 --2- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7fe804039160 0x7fe80403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-22T23:48:48.431 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.372+0000 7fe817400700 1 -- 172.21.15.103:0/733260297 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe8100172b0 con 0x7fe82007de60 2024-05-22T23:48:48.432 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.566+0000 7fe81e000700 1 -- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7fe804039160 msgr2=0x7fe80403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3495336128 2024-05-22T23:48:48.432 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.566+0000 7fe81e000700 1 --2- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7fe804039160 0x7fe80403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-22T23:48:48.432 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.653+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fe8201a1590 con 0x7fe82007de60 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.654+0000 7fe817400700 1 -- 172.21.15.103:0/733260297 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fe8201a1590 con 0x7fe82007de60 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7fe804039160 msgr2=0x7fe80403b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 --2- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7fe804039160 0x7fe80403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 msgr2=0x7fe82019df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 --2- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82019df00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe810011b60 tx=0x7fe810011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 shutdown_connections 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 --2- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7fe804039160 0x7fe80403b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 --2- 172.21.15.103:0/733260297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe82007de60 0x7fe82019df00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 >> 172.21.15.103:0/733260297 conn(0x7fe8201033c0 msgr2=0x7fe820103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 shutdown_connections 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:47.657+0000 7fe825a00700 1 -- 172.21.15.103:0/733260297 wait complete. 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for the mgr to restart... 2024-05-22T23:48:48.433 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr epoch 5... 2024-05-22T23:48:48.994 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:48.756+0000 7f86ef4c8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-22T23:48:48.994 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:48.989+0000 7f86ef4c8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-22T23:48:49.254 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:49.204+0000 7f86ef4c8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-22T23:48:49.964 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:49.586+0000 7f86ef4c8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-22T23:48:49.965 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:49.706+0000 7f86ef4c8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-22T23:48:50.217 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:49.959+0000 7f86ef4c8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-22T23:48:50.217 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:50 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:50.211+0000 7f86ef4c8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:48:51.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:50.998+0000 7f86ef4c8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-22T23:48:51.698 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:51.559+0000 7f86ef4c8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-22T23:48:51.698 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:51.692+0000 7f86ef4c8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-22T23:48:52.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:51.965+0000 7f86ef4c8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-22T23:48:52.920 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:52 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:52.623+0000 7f86ef4c8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-22T23:48:52.920 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:52 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:52.747+0000 7f86ef4c8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-22T23:48:53.255 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:53.071+0000 7f86ef4c8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-22T23:48:53.256 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:53.249+0000 7f86ef4c8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-22T23:48:53.534 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:53.375+0000 7f86ef4c8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-22T23:48:53.534 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:53.528+0000 7f86ef4c8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-22T23:48:54.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:53.892+0000 7f86ef4c8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-22T23:48:54.927 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:54 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:54.637+0000 7f86ef4c8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-22T23:48:55.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:48:54 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:48:54.921+0000 7f86ef4c8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-22T23:48:55.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: Active manager daemon x restarted 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: Activating manager daemon x 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: osdmap e2: 0 total, 0 up, 0 in 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: mgrmap e6: x(active, starting, since 0.00485131s) 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: Manager daemon x is now available 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:48:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:54 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:48:56.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: Found migration_current of "None". Setting to last migration. 2024-05-22T23:48:56.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:48:56.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:48:56.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-22T23:48:56.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-22T23:48:56.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:48:56.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:48:56.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:48:56.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:55 smithi103 ceph-mon[31370]: mgrmap e7: x(active, since 1.01597s) 2024-05-22T23:48:56.942 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-05-22T23:48:56.954 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-22T23:48:56.954 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-22T23:48:56.954 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.040+0000 7f3216400700 1 Processor -- start 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.040+0000 7f3216400700 1 -- start start 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.040+0000 7f3216400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321007c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.040+0000 7f3216400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f321007c8b0 con 0x7f321007de80 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.041+0000 7f320f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321007c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.041+0000 7f320f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321007c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42952/0 (socket says 172.21.15.103:42952) 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.041+0000 7f320f400700 1 -- 172.21.15.103:0/2505684972 learned_addr learned my addr 172.21.15.103:0/2505684972 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.041+0000 7f320f400700 1 -- 172.21.15.103:0/2505684972 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321007ca30 con 0x7f321007de80 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.042+0000 7f320f400700 1 --2- 172.21.15.103:0/2505684972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321007c2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f32000098a0 tx=0x7f3200009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be411d7750e97b63 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.042+0000 7f320e000700 1 -- 172.21.15.103:0/2505684972 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3200004160 con 0x7f321007de80 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.042+0000 7f320e000700 1 -- 172.21.15.103:0/2505684972 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3200004300 con 0x7f321007de80 2024-05-22T23:48:56.955 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.042+0000 7f320e000700 1 -- 172.21.15.103:0/2505684972 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3200011b50 con 0x7f321007de80 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.043+0000 7f3216400700 1 -- 172.21.15.103:0/2505684972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 msgr2=0x7f321007c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.043+0000 7f3216400700 1 --2- 172.21.15.103:0/2505684972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321007c2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f32000098a0 tx=0x7f3200009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.043+0000 7f3216400700 1 -- 172.21.15.103:0/2505684972 shutdown_connections 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.043+0000 7f3216400700 1 --2- 172.21.15.103:0/2505684972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321007c2e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.043+0000 7f3216400700 1 -- 172.21.15.103:0/2505684972 >> 172.21.15.103:0/2505684972 conn(0x7f3210103320 msgr2=0x7f3210105780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.043+0000 7f3216400700 1 -- 172.21.15.103:0/2505684972 shutdown_connections 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.043+0000 7f3216400700 1 -- 172.21.15.103:0/2505684972 wait complete. 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.044+0000 7f3216400700 1 Processor -- start 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.045+0000 7f3216400700 1 -- start start 2024-05-22T23:48:56.956 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.045+0000 7f3216400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321019dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.045+0000 7f3216400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f321010e3d0 con 0x7f321007de80 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.045+0000 7f320f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321019dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.045+0000 7f320f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321019dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42956/0 (socket says 172.21.15.103:42956) 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.045+0000 7f320f400700 1 -- 172.21.15.103:0/591963044 learned_addr learned my addr 172.21.15.103:0/591963044 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.046+0000 7f320f400700 1 -- 172.21.15.103:0/591963044 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f321019e3e0 con 0x7f321007de80 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.046+0000 7f320f400700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321019dea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3200011b60 tx=0x7f3200011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.046+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3200003b20 con 0x7f321007de80 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.046+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3200003cc0 con 0x7f321007de80 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.046+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f321019e670 con 0x7f321007de80 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.046+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3200024840 con 0x7f321007de80 2024-05-22T23:48:56.957 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.047+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32101a1360 con 0x7f321007de80 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.048+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f3200022020 con 0x7f321007de80 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.048+0000 7f3207e00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 0x7f31f403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.048+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 --> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f31f403bd30 con 0x7f31f4039160 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.048+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3200011e00 con 0x7f321007de80 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.048+0000 7f320ea00700 1 -- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 msgr2=0x7f31f403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3495336128 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.048+0000 7f320ea00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 0x7f31f403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.249+0000 7f320ea00700 1 -- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 msgr2=0x7f31f403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3495336128 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.249+0000 7f320ea00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 0x7f31f403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.650+0000 7f320ea00700 1 -- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 msgr2=0x7f31f403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3495336128 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:49.650+0000 7f320ea00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 0x7f31f403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-22T23:48:56.958 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:50.450+0000 7f320ea00700 1 -- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 msgr2=0x7f31f403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3495336128 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:50.450+0000 7f320ea00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 0x7f31f403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:52.052+0000 7f320ea00700 1 -- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 msgr2=0x7f31f403b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3495336128 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:52.052+0000 7f320ea00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 0x7f31f403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:54.929+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7f3200024a20 con 0x7f321007de80 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:54.929+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 msgr2=0x7f31f403b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:54.929+0000 7f3207e00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 0x7f31f403b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.936+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7f320004eb30 con 0x7f321007de80 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.936+0000 7f3207e00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f31f403ca90 0x7f31f403ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.936+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f320004b670 con 0x7f31f403ca90 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.938+0000 7f320ea00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f31f403ca90 0x7f31f403ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:56.959 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.942+0000 7f320ea00700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f31f403ca90 0x7f31f403ee80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f31fc003a10 tx=0x7f31fc0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.944+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mgr.14120 v2:172.21.15.103:6800/3762323194 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f320004b670 con 0x7f31f403ca90 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.952+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3210069490 con 0x7f31f403ca90 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.953+0000 7f3207e00700 1 -- 172.21.15.103:0/591963044 <== mgr.14120 v2:172.21.15.103:6800/3762323194 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f3210069490 con 0x7f31f403ca90 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f31f403ca90 msgr2=0x7f31f403ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f31f403ca90 0x7f31f403ee80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f31fc003a10 tx=0x7f31fc0092b0 comp rx=0 tx=0).stop 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 msgr2=0x7f321019dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321019dea0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3200011b60 tx=0x7f3200011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 shutdown_connections 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f321007de80 0x7f321019dea0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f31f403ca90 0x7f31f403ee80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:56.960 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 --2- 172.21.15.103:0/591963044 >> [v2:172.21.15.103:6800/3495336128,v1:172.21.15.103:6801/3495336128] conn(0x7f31f4039160 0x7f31f403b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:56.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.954+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 >> 172.21.15.103:0/591963044 conn(0x7f3210103320 msgr2=0x7f3210103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:56.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.955+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 shutdown_connections 2024-05-22T23:48:56.961 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:55.955+0000 7f3216400700 1 -- 172.21.15.103:0/591963044 wait complete. 2024-05-22T23:48:56.961 INFO:teuthology.orchestra.run.smithi103.stdout:mgr epoch 5 is available 2024-05-22T23:48:56.961 INFO:teuthology.orchestra.run.smithi103.stdout:Setting orchestrator backend to cephadm... 2024-05-22T23:48:57.113 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:56 smithi103 ceph-mon[31370]: [22/May/2024:23:48:55] ENGINE Bus STARTING 2024-05-22T23:48:57.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:56 smithi103 ceph-mon[31370]: [22/May/2024:23:48:55] ENGINE Serving on https://172.21.15.103:7150 2024-05-22T23:48:57.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:56 smithi103 ceph-mon[31370]: [22/May/2024:23:48:55] ENGINE Serving on http://172.21.15.103:8765 2024-05-22T23:48:57.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:56 smithi103 ceph-mon[31370]: [22/May/2024:23:48:55] ENGINE Bus STARTED 2024-05-22T23:48:57.114 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:56 smithi103 ceph-mon[31370]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-22T23:48:57.115 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:56 smithi103 ceph-mon[31370]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-22T23:48:58.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:57 smithi103 ceph-mon[31370]: mgrmap e8: x(active, since 2s) 2024-05-22T23:48:58.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:57 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:48:58.226 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:57 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:48:58.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.582+0000 7f74d6c00700 1 Processor -- start 2024-05-22T23:48:58.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.582+0000 7f74d6c00700 1 -- start start 2024-05-22T23:48:58.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.583+0000 7f74d6c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d01082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:58.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.583+0000 7f74d6c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74d0108800 con 0x7f74d0107ea0 2024-05-22T23:48:58.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.583+0000 7f74cfe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d01082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:58.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.584+0000 7f74cfe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d01082c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45244/0 (socket says 172.21.15.103:45244) 2024-05-22T23:48:58.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.584+0000 7f74cfe00700 1 -- 172.21.15.103:0/1543228781 learned_addr learned my addr 172.21.15.103:0/1543228781 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:58.572 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.584+0000 7f74cfe00700 1 -- 172.21.15.103:0/1543228781 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d0108f60 con 0x7f74d0107ea0 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.584+0000 7f74cfe00700 1 --2- 172.21.15.103:0/1543228781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d01082c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f74c00098a0 tx=0x7f74c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8624848f792851 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.585+0000 7f74cea00700 1 -- 172.21.15.103:0/1543228781 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74c0004160 con 0x7f74d0107ea0 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.585+0000 7f74cea00700 1 -- 172.21.15.103:0/1543228781 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f74c0004300 con 0x7f74d0107ea0 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.585+0000 7f74cea00700 1 -- 172.21.15.103:0/1543228781 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74c0011b50 con 0x7f74d0107ea0 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.585+0000 7f74d6c00700 1 -- 172.21.15.103:0/1543228781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 msgr2=0x7f74d01082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.585+0000 7f74d6c00700 1 --2- 172.21.15.103:0/1543228781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d01082c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f74c00098a0 tx=0x7f74c0009bb0 comp rx=0 tx=0).stop 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.585+0000 7f74d6c00700 1 -- 172.21.15.103:0/1543228781 shutdown_connections 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.585+0000 7f74d6c00700 1 --2- 172.21.15.103:0/1543228781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d01082c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.585+0000 7f74d6c00700 1 -- 172.21.15.103:0/1543228781 >> 172.21.15.103:0/1543228781 conn(0x7f74d0103460 msgr2=0x7f74d0105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.586+0000 7f74d6c00700 1 -- 172.21.15.103:0/1543228781 shutdown_connections 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.586+0000 7f74d6c00700 1 -- 172.21.15.103:0/1543228781 wait complete. 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.587+0000 7f74d6c00700 1 Processor -- start 2024-05-22T23:48:58.573 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.587+0000 7f74d6c00700 1 -- start start 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.587+0000 7f74d6c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d019bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.587+0000 7f74d6c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74d010c3a0 con 0x7f74d0107ea0 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.587+0000 7f74cfe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d019bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.587+0000 7f74cfe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d019bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45250/0 (socket says 172.21.15.103:45250) 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.587+0000 7f74cfe00700 1 -- 172.21.15.103:0/1524023751 learned_addr learned my addr 172.21.15.103:0/1524023751 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.588+0000 7f74cfe00700 1 -- 172.21.15.103:0/1524023751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d019c1e0 con 0x7f74d0107ea0 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.588+0000 7f74cfe00700 1 --2- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d019bca0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f74c0011b60 tx=0x7f74c0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.588+0000 7f74ccc00700 1 -- 172.21.15.103:0/1524023751 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74c0003b20 con 0x7f74d0107ea0 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.588+0000 7f74ccc00700 1 -- 172.21.15.103:0/1524023751 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f74c0003cc0 con 0x7f74d0107ea0 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.588+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d019c470 con 0x7f74d0107ea0 2024-05-22T23:48:58.574 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.588+0000 7f74ccc00700 1 -- 172.21.15.103:0/1524023751 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74c0024840 con 0x7f74d0107ea0 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.589+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d019f160 con 0x7f74d0107ea0 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.590+0000 7f74ccc00700 1 -- 172.21.15.103:0/1524023751 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f74c0022020 con 0x7f74d0107ea0 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.590+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d00639c0 con 0x7f74d0107ea0 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.590+0000 7f74ccc00700 1 --2- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f74b4039160 0x7f74b403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.590+0000 7f74ccc00700 1 -- 172.21.15.103:0/1524023751 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f74c0011e00 con 0x7f74d0107ea0 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.590+0000 7f74cf400700 1 --2- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f74b4039160 0x7f74b403b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.591+0000 7f74cf400700 1 --2- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f74b4039160 0x7f74b403b620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006fd0 tx=0x7f74bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.597+0000 7f74ccc00700 1 -- 172.21.15.103:0/1524023751 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f74c000b3e0 con 0x7f74d0107ea0 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.837+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f74d01045e0 con 0x7f74b4039160 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.847+0000 7f74ccc00700 1 -- 172.21.15.103:0/1524023751 <== mgr.14120 v2:172.21.15.103:6800/3762323194 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f74d01045e0 con 0x7f74b4039160 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.850+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f74b4039160 msgr2=0x7f74b403b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:58.575 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.850+0000 7f74d6c00700 1 --2- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f74b4039160 0x7f74b403b620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006fd0 tx=0x7f74bc006e40 comp rx=0 tx=0).stop 2024-05-22T23:48:58.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.850+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 msgr2=0x7f74d019bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:48:58.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.850+0000 7f74d6c00700 1 --2- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d019bca0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f74c0011b60 tx=0x7f74c0011c40 comp rx=0 tx=0).stop 2024-05-22T23:48:58.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.851+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 shutdown_connections 2024-05-22T23:48:58.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.851+0000 7f74d6c00700 1 --2- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f74b4039160 0x7f74b403b620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:58.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.851+0000 7f74d6c00700 1 --2- 172.21.15.103:0/1524023751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74d0107ea0 0x7f74d019bca0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:48:58.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.851+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 >> 172.21.15.103:0/1524023751 conn(0x7f74d0103460 msgr2=0x7f74d0103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:48:58.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.851+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 shutdown_connections 2024-05-22T23:48:58.576 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:57.851+0000 7f74d6c00700 1 -- 172.21.15.103:0/1524023751 wait complete. 2024-05-22T23:48:59.191 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:48:58 smithi103 ceph-mon[31370]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:00.198 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-22T23:49:00.198 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.244+0000 7fd6fea00700 1 Processor -- start 2024-05-22T23:49:00.199 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.244+0000 7fd6fea00700 1 -- start start 2024-05-22T23:49:00.199 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.244+0000 7fd6fea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f8108240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:00.199 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.244+0000 7fd6fea00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f8108780 con 0x7fd6f8107e20 2024-05-22T23:49:00.199 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.245+0000 7fd6f7e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f8108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.245+0000 7fd6f7e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f8108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45260/0 (socket says 172.21.15.103:45260) 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.245+0000 7fd6f7e00700 1 -- 172.21.15.103:0/2373075688 learned_addr learned my addr 172.21.15.103:0/2373075688 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.245+0000 7fd6f7e00700 1 -- 172.21.15.103:0/2373075688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f8108f80 con 0x7fd6f8107e20 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.245+0000 7fd6f7e00700 1 --2- 172.21.15.103:0/2373075688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f8108240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80098a0 tx=0x7fd6e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6bfceeb7a68081 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.246+0000 7fd6f7400700 1 -- 172.21.15.103:0/2373075688 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e8004160 con 0x7fd6f8107e20 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.246+0000 7fd6f7400700 1 -- 172.21.15.103:0/2373075688 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd6e8004300 con 0x7fd6f8107e20 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.246+0000 7fd6f7400700 1 -- 172.21.15.103:0/2373075688 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e8011b50 con 0x7fd6f8107e20 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.246+0000 7fd6fea00700 1 -- 172.21.15.103:0/2373075688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 msgr2=0x7fd6f8108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:00.201 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.246+0000 7fd6fea00700 1 --2- 172.21.15.103:0/2373075688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f8108240 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80098a0 tx=0x7fd6e8009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:00.204 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.247+0000 7fd6fea00700 1 -- 172.21.15.103:0/2373075688 shutdown_connections 2024-05-22T23:49:00.204 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.247+0000 7fd6fea00700 1 --2- 172.21.15.103:0/2373075688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f8108240 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:00.204 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.247+0000 7fd6fea00700 1 -- 172.21.15.103:0/2373075688 >> 172.21.15.103:0/2373075688 conn(0x7fd6f81033c0 msgr2=0x7fd6f8105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:00.204 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.247+0000 7fd6fea00700 1 -- 172.21.15.103:0/2373075688 shutdown_connections 2024-05-22T23:49:00.204 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.247+0000 7fd6fea00700 1 -- 172.21.15.103:0/2373075688 wait complete. 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.248+0000 7fd6fea00700 1 Processor -- start 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.248+0000 7fd6fea00700 1 -- start start 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.249+0000 7fd6fea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.249+0000 7fd6fea00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f810c3c0 con 0x7fd6f8107e20 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.249+0000 7fd6f7e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.249+0000 7fd6f7e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45266/0 (socket says 172.21.15.103:45266) 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.249+0000 7fd6f7e00700 1 -- 172.21.15.103:0/4071196049 learned_addr learned my addr 172.21.15.103:0/4071196049 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.249+0000 7fd6f7e00700 1 -- 172.21.15.103:0/4071196049 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f819e4e0 con 0x7fd6f8107e20 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.250+0000 7fd6f7e00700 1 --2- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f819dfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8011b60 tx=0x7fd6e8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.250+0000 7fd6f5600700 1 -- 172.21.15.103:0/4071196049 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e8003970 con 0x7fd6f8107e20 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.250+0000 7fd6f5600700 1 -- 172.21.15.103:0/4071196049 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd6e8003b10 con 0x7fd6f8107e20 2024-05-22T23:49:00.205 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.250+0000 7fd6f5600700 1 -- 172.21.15.103:0/4071196049 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd6e8024610 con 0x7fd6f8107e20 2024-05-22T23:49:00.206 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.250+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f819e770 con 0x7fd6f8107e20 2024-05-22T23:49:00.206 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.250+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f81a1460 con 0x7fd6f8107e20 2024-05-22T23:49:00.206 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.251+0000 7fd6f5600700 1 -- 172.21.15.103:0/4071196049 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fd6e801a9f0 con 0x7fd6f8107e20 2024-05-22T23:49:00.206 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.251+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f8069490 con 0x7fd6f8107e20 2024-05-22T23:49:00.206 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.252+0000 7fd6f5600700 1 --2- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd6dc039160 0x7fd6dc03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.252+0000 7fd6f5600700 1 -- 172.21.15.103:0/4071196049 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd6e8011e00 con 0x7fd6f8107e20 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.252+0000 7fd6efe00700 1 --2- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd6dc039160 0x7fd6dc03b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.253+0000 7fd6efe00700 1 --2- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd6dc039160 0x7fd6dc03b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0006fd0 tx=0x7fd6e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.258+0000 7fd6f5600700 1 -- 172.21.15.103:0/4071196049 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd6e8016260 con 0x7fd6f8107e20 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.481+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fd6f81044f0 con 0x7fd6dc039160 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.483+0000 7fd6f5600700 1 -- 172.21.15.103:0/4071196049 <== mgr.14120 v2:172.21.15.103:6800/3762323194 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fd6f81044f0 con 0x7fd6dc039160 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd6dc039160 msgr2=0x7fd6dc03b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 --2- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd6dc039160 0x7fd6dc03b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0006fd0 tx=0x7fd6e0006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:00.208 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 msgr2=0x7fd6f819dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:00.209 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 --2- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f819dfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8011b60 tx=0x7fd6e8011c40 comp rx=0 tx=0).stop 2024-05-22T23:49:00.209 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 shutdown_connections 2024-05-22T23:49:00.210 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 --2- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd6dc039160 0x7fd6dc03b620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:00.210 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 --2- 172.21.15.103:0/4071196049 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f8107e20 0x7fd6f819dfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:00.210 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 >> 172.21.15.103:0/4071196049 conn(0x7fd6f81033c0 msgr2=0x7fd6f8103de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:00.210 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 shutdown_connections 2024-05-22T23:49:00.210 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:48:59.486+0000 7fd6fea00700 1 -- 172.21.15.103:0/4071196049 wait complete. 2024-05-22T23:49:00.210 INFO:teuthology.orchestra.run.smithi103.stdout:Generating ssh key... 2024-05-22T23:49:00.529 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:00 smithi103 ceph-mon[31370]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:01.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:01 smithi103 ceph-mon[31370]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:01.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:01 smithi103 ceph-mon[31370]: Generating ssh key... 2024-05-22T23:49:01.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: Generating public/private rsa key pair. 2024-05-22T23:49:01.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: Your identification has been saved in /tmp/tmpur8z3t83/key. 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: Your public key has been saved in /tmp/tmpur8z3t83/key.pub. 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: The key fingerprint is: 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: SHA256:H+rf1qlSPV/6SERSsZ7X+heroVVWR83rz7C6p+e3GDg ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: The key's randomart image is: 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: +---[RSA 3072]----+ 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | ooo| 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | . oo| 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | . o +| 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | + o+| 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | S . .=oo| 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | o .o.*=.| 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | . .E =oO=| 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | . ..=.@+*| 2024-05-22T23:49:01.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: | ...+*@++=| 2024-05-22T23:49:01.333 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: +----[SHA256]-----+ 2024-05-22T23:49:01.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.832+0000 7f5147e00700 1 Processor -- start 2024-05-22T23:49:01.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.833+0000 7f5147e00700 1 -- start start 2024-05-22T23:49:01.990 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.833+0000 7f5147e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514810a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.833+0000 7f5147e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f514810aab0 con 0x7f514810a150 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.834+0000 7f5146a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514810a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.834+0000 7f5146a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514810a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45270/0 (socket says 172.21.15.103:45270) 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.834+0000 7f5146a00700 1 -- 172.21.15.103:0/2619626159 learned_addr learned my addr 172.21.15.103:0/2619626159 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.834+0000 7f5146a00700 1 -- 172.21.15.103:0/2619626159 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514810af70 con 0x7f514810a150 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.834+0000 7f5146a00700 1 --2- 172.21.15.103:0/2619626159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514810a570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51380098a0 tx=0x7f5138009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada3f2c616a23e89 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5145600700 1 -- 172.21.15.103:0/2619626159 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5138004160 con 0x7f514810a150 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5145600700 1 -- 172.21.15.103:0/2619626159 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5138004300 con 0x7f514810a150 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5145600700 1 -- 172.21.15.103:0/2619626159 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5138011b50 con 0x7f514810a150 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5147e00700 1 -- 172.21.15.103:0/2619626159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 msgr2=0x7f514810a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5147e00700 1 --2- 172.21.15.103:0/2619626159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514810a570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f51380098a0 tx=0x7f5138009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5147e00700 1 -- 172.21.15.103:0/2619626159 shutdown_connections 2024-05-22T23:49:01.991 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5147e00700 1 --2- 172.21.15.103:0/2619626159 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514810a570 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5147e00700 1 -- 172.21.15.103:0/2619626159 >> 172.21.15.103:0/2619626159 conn(0x7f514807d230 msgr2=0x7f514807d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5147e00700 1 -- 172.21.15.103:0/2619626159 shutdown_connections 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.835+0000 7f5147e00700 1 -- 172.21.15.103:0/2619626159 wait complete. 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.836+0000 7f5147e00700 1 Processor -- start 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.837+0000 7f5147e00700 1 -- start start 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.837+0000 7f5147e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.837+0000 7f5147e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f514810e230 con 0x7f514810a150 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.837+0000 7f5146a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.837+0000 7f5146a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514819dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45278/0 (socket says 172.21.15.103:45278) 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.837+0000 7f5146a00700 1 -- 172.21.15.103:0/1363472535 learned_addr learned my addr 172.21.15.103:0/1363472535 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.838+0000 7f5146a00700 1 -- 172.21.15.103:0/1363472535 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514819e2a0 con 0x7f514810a150 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.838+0000 7f5146a00700 1 --2- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514819dd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5138009fb0 tx=0x7f5138011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:01.992 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.838+0000 7f513f400700 1 -- 172.21.15.103:0/1363472535 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51380038b0 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.838+0000 7f513f400700 1 -- 172.21.15.103:0/1363472535 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5138011e50 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.838+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514819e530 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.838+0000 7f513f400700 1 -- 172.21.15.103:0/1363472535 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5138019420 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.838+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51481a1220 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.839+0000 7f513f400700 1 -- 172.21.15.103:0/1363472535 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f5138003a50 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.839+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5148069490 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.840+0000 7f513f400700 1 --2- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f512c0390c0 0x7f512c03b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.840+0000 7f513f400700 1 -- 172.21.15.103:0/1363472535 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f513804d360 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.840+0000 7f5146000700 1 --2- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f512c0390c0 0x7f512c03b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.841+0000 7f5146000700 1 --2- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f512c0390c0 0x7f512c03b580 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5134006fd0 tx=0x7f5134006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:00.845+0000 7f513f400700 1 -- 172.21.15.103:0/1363472535 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f513804ccb0 con 0x7f514810a150 2024-05-22T23:49:01.993 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.070+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f5148108390 con 0x7f512c0390c0 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.313+0000 7f513f400700 1 -- 172.21.15.103:0/1363472535 <== mgr.14120 v2:172.21.15.103:6800/3762323194 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f5148108390 con 0x7f512c0390c0 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.316+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f512c0390c0 msgr2=0x7f512c03b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.316+0000 7f5147e00700 1 --2- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f512c0390c0 0x7f512c03b580 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5134006fd0 tx=0x7f5134006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.316+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 msgr2=0x7f514819dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.316+0000 7f5147e00700 1 --2- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514819dd60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5138009fb0 tx=0x7f5138011bf0 comp rx=0 tx=0).stop 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.316+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 shutdown_connections 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.316+0000 7f5147e00700 1 --2- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f512c0390c0 0x7f512c03b580 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.316+0000 7f5147e00700 1 --2- 172.21.15.103:0/1363472535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f514810a150 0x7f514819dd60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.316+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 >> 172.21.15.103:0/1363472535 conn(0x7f514807d230 msgr2=0x7f5148107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.317+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 shutdown_connections 2024-05-22T23:49:01.994 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:01.317+0000 7f5147e00700 1 -- 172.21.15.103:0/1363472535 wait complete. 2024-05-22T23:49:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:02 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:02 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:03.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:03 smithi103 ceph-mon[31370]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:03.606 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOFZlq/l2JR3f0+XdUUrv9nXPFJhPYpgK1ypdGY62JcbcGQZvY6mzAJ6G0gnPlgU/hb6XFDOJ7Yzy3gsxwR7N599VnU72BdhOOwovzrIeNKiesq36v+gOxZpTpN/ArQarB7xJR1SlI3hUXunGwgZb7vHpAGxyHlYGvaERqeIzMDuC3gIy+kGLDZkfA8VVWAbqcfIk3s02S8G4IdeRbkKfMyEZLjvaAMA2mg2WRDDRZ535q+jdGhDaRxpwA/j2DBWcB60gir01JR2ubGpAVfNz+vKUi3uFQfY+NUezeLy1Ao+3SyXSEug4HPHsuWd31tjQPpQScsfFTVdP+K34YxAIDJ9Op2qTYkpf7qNNaEIqrFhLiu9DifYpK3JR+vSEqvRcmznapJHtZRYwSpsVVeAZ/PAef8TZxZfIOXfdYkxJAtHviKIhi2TIll2jmRKRW1ZFM5oWjkENEFrpSKdACSEyZNjbb1cYFIISwF5wqsH/82+lr6LRMsq9TwXFw0mUTpk= ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:49:03.606 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.599+0000 7f0acb200700 1 Processor -- start 2024-05-22T23:49:03.607 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.599+0000 7f0acb200700 1 -- start start 2024-05-22T23:49:03.607 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.600+0000 7f0acb200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc10a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:03.607 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.600+0000 7f0acb200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc10aab0 con 0x7f0acc10a150 2024-05-22T23:49:03.607 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.600+0000 7f0ac9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc10a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:03.607 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.600+0000 7f0ac9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc10a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45288/0 (socket says 172.21.15.103:45288) 2024-05-22T23:49:03.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.600+0000 7f0ac9e00700 1 -- 172.21.15.103:0/1471349258 learned_addr learned my addr 172.21.15.103:0/1471349258 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:03.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.601+0000 7f0ac9e00700 1 -- 172.21.15.103:0/1471349258 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0acc10af70 con 0x7f0acc10a150 2024-05-22T23:49:03.612 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.601+0000 7f0ac9e00700 1 --2- 172.21.15.103:0/1471349258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc10a570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0098a0 tx=0x7f0abc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208607137c5d6926 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.601+0000 7f0ac8a00700 1 -- 172.21.15.103:0/1471349258 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0abc004160 con 0x7f0acc10a150 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.601+0000 7f0ac8a00700 1 -- 172.21.15.103:0/1471349258 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0abc004300 con 0x7f0acc10a150 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.601+0000 7f0ac8a00700 1 -- 172.21.15.103:0/1471349258 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0abc011b50 con 0x7f0acc10a150 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.602+0000 7f0acb200700 1 -- 172.21.15.103:0/1471349258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 msgr2=0x7f0acc10a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.602+0000 7f0acb200700 1 --2- 172.21.15.103:0/1471349258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc10a570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0098a0 tx=0x7f0abc009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.602+0000 7f0acb200700 1 -- 172.21.15.103:0/1471349258 shutdown_connections 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.602+0000 7f0acb200700 1 --2- 172.21.15.103:0/1471349258 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc10a570 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.602+0000 7f0acb200700 1 -- 172.21.15.103:0/1471349258 >> 172.21.15.103:0/1471349258 conn(0x7f0acc07d230 msgr2=0x7f0acc07d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.602+0000 7f0acb200700 1 -- 172.21.15.103:0/1471349258 shutdown_connections 2024-05-22T23:49:03.613 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.602+0000 7f0acb200700 1 -- 172.21.15.103:0/1471349258 wait complete. 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.603+0000 7f0acb200700 1 Processor -- start 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.603+0000 7f0acb200700 1 -- start start 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.604+0000 7f0acb200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.604+0000 7f0acb200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc10e230 con 0x7f0acc10a150 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.604+0000 7f0ac9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.604+0000 7f0ac9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc19dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45304/0 (socket says 172.21.15.103:45304) 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.604+0000 7f0ac9e00700 1 -- 172.21.15.103:0/2979785072 learned_addr learned my addr 172.21.15.103:0/2979785072 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.604+0000 7f0ac9e00700 1 -- 172.21.15.103:0/2979785072 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0acc19e2a0 con 0x7f0acc10a150 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.605+0000 7f0ac9e00700 1 --2- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc19dd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0abc011b60 tx=0x7f0abc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:03.614 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.605+0000 7f0ac2a00700 1 -- 172.21.15.103:0/2979785072 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0abc003970 con 0x7f0acc10a150 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.605+0000 7f0ac2a00700 1 -- 172.21.15.103:0/2979785072 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0abc003b10 con 0x7f0acc10a150 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.605+0000 7f0ac2a00700 1 -- 172.21.15.103:0/2979785072 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0abc024610 con 0x7f0acc10a150 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.605+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0acc19e530 con 0x7f0acc10a150 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.605+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0acc1a1220 con 0x7f0acc10a150 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.606+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0acc069490 con 0x7f0acc10a150 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.606+0000 7f0ac2a00700 1 -- 172.21.15.103:0/2979785072 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f0abc01a9f0 con 0x7f0acc10a150 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.607+0000 7f0ac2a00700 1 --2- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f0ab0039160 0x7f0ab003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.607+0000 7f0ac2a00700 1 -- 172.21.15.103:0/2979785072 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0abc011e00 con 0x7f0acc10a150 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.607+0000 7f0ac9400700 1 --2- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f0ab0039160 0x7f0ab003b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:03.615 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.608+0000 7f0ac9400700 1 --2- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f0ab0039160 0x7f0ab003b620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8006fd0 tx=0x7f0ab8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.612+0000 7f0ac2a00700 1 -- 172.21.15.103:0/2979785072 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0abc016260 con 0x7f0acc10a150 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.834+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f0acc108390 con 0x7f0ab0039160 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.836+0000 7f0ac2a00700 1 -- 172.21.15.103:0/2979785072 <== mgr.14120 v2:172.21.15.103:6800/3762323194 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f0acc108390 con 0x7f0ab0039160 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f0ab0039160 msgr2=0x7f0ab003b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 --2- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f0ab0039160 0x7f0ab003b620 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8006fd0 tx=0x7f0ab8006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 msgr2=0x7f0acc19dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 --2- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc19dd60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0abc011b60 tx=0x7f0abc011c40 comp rx=0 tx=0).stop 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 shutdown_connections 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 --2- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f0ab0039160 0x7f0ab003b620 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 --2- 172.21.15.103:0/2979785072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0acc10a150 0x7f0acc19dd60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:03.616 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 >> 172.21.15.103:0/2979785072 conn(0x7f0acc07d230 msgr2=0x7f0acc107c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:03.617 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 shutdown_connections 2024-05-22T23:49:03.617 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:02.839+0000 7f0acb200700 1 -- 172.21.15.103:0/2979785072 wait complete. 2024-05-22T23:49:03.617 INFO:teuthology.orchestra.run.smithi103.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-22T23:49:03.617 INFO:teuthology.orchestra.run.smithi103.stdout:Adding key to root@localhost authorized_keys... 2024-05-22T23:49:03.617 INFO:teuthology.orchestra.run.smithi103.stdout:Adding host smithi103... 2024-05-22T23:49:05.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:05 smithi103 ceph-mon[31370]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "addr": "172.21.15.103", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:06.235 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:06 smithi103 ceph-mon[31370]: Deploying cephadm binary to smithi103 2024-05-22T23:49:11.085 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Added host 'smithi103' with addr '172.21.15.103' 2024-05-22T23:49:11.085 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.246+0000 7fb4ab000700 1 Processor -- start 2024-05-22T23:49:11.085 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.246+0000 7fb4ab000700 1 -- start start 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.246+0000 7fb4ab000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a4108830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.246+0000 7fb4ab000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a4108e00 con 0x7fb4a4108410 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.247+0000 7fb4a3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a4108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.247+0000 7fb4a3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a4108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46056/0 (socket says 172.21.15.103:46056) 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.247+0000 7fb4a3e00700 1 -- 172.21.15.103:0/1250399476 learned_addr learned my addr 172.21.15.103:0/1250399476 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.247+0000 7fb4a3e00700 1 -- 172.21.15.103:0/1250399476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a4105f10 con 0x7fb4a4108410 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.248+0000 7fb4a3e00700 1 --2- 172.21.15.103:0/1250399476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a4108830 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb4940098a0 tx=0x7fb494009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6aa688580c14fa6 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.248+0000 7fb4a2a00700 1 -- 172.21.15.103:0/1250399476 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb494004160 con 0x7fb4a4108410 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.248+0000 7fb4a2a00700 1 -- 172.21.15.103:0/1250399476 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb494004300 con 0x7fb4a4108410 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.248+0000 7fb4a2a00700 1 -- 172.21.15.103:0/1250399476 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb494011b50 con 0x7fb4a4108410 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.249+0000 7fb4ab000700 1 -- 172.21.15.103:0/1250399476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 msgr2=0x7fb4a4108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:11.086 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.249+0000 7fb4ab000700 1 --2- 172.21.15.103:0/1250399476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a4108830 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb4940098a0 tx=0x7fb494009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.249+0000 7fb4ab000700 1 -- 172.21.15.103:0/1250399476 shutdown_connections 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.249+0000 7fb4ab000700 1 --2- 172.21.15.103:0/1250399476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a4108830 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.249+0000 7fb4ab000700 1 -- 172.21.15.103:0/1250399476 >> 172.21.15.103:0/1250399476 conn(0x7fb4a4101ab0 msgr2=0x7fb4a4103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.249+0000 7fb4ab000700 1 -- 172.21.15.103:0/1250399476 shutdown_connections 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.249+0000 7fb4ab000700 1 -- 172.21.15.103:0/1250399476 wait complete. 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.250+0000 7fb4ab000700 1 Processor -- start 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.250+0000 7fb4ab000700 1 -- start start 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.250+0000 7fb4ab000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a419dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.250+0000 7fb4ab000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a4106f20 con 0x7fb4a4108410 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.251+0000 7fb4a3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a419dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.251+0000 7fb4a3e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a419dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46066/0 (socket says 172.21.15.103:46066) 2024-05-22T23:49:11.087 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.251+0000 7fb4a3e00700 1 -- 172.21.15.103:0/3487131569 learned_addr learned my addr 172.21.15.103:0/3487131569 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.251+0000 7fb4a3e00700 1 -- 172.21.15.103:0/3487131569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a419e3e0 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.251+0000 7fb4a3e00700 1 --2- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a419dea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb494011b60 tx=0x7fb494011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.252+0000 7fb4a0c00700 1 -- 172.21.15.103:0/3487131569 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb494003b20 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.252+0000 7fb4a0c00700 1 -- 172.21.15.103:0/3487131569 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb494003cc0 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.252+0000 7fb4a0c00700 1 -- 172.21.15.103:0/3487131569 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb494024840 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.252+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a419e670 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.252+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a41a1360 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.253+0000 7fb4a0c00700 1 -- 172.21.15.103:0/3487131569 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb494022020 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.253+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a40639c0 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.253+0000 7fb4a0c00700 1 --2- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fb488039160 0x7fb48803b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.253+0000 7fb4a0c00700 1 -- 172.21.15.103:0/3487131569 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb494011e00 con 0x7fb4a4108410 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.254+0000 7fb4a3400700 1 --2- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fb488039160 0x7fb48803b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:11.088 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.254+0000 7fb4a3400700 1 --2- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fb488039160 0x7fb48803b620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb490006fd0 tx=0x7fb490006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.259+0000 7fb4a0c00700 1 -- 172.21.15.103:0/3487131569 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb49404cb30 con 0x7fb4a4108410 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:04.485+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi103", "addr": "172.21.15.103", "target": ["mon-mgr", ""]}) -- 0x7fb4a40627a0 con 0x7fb488039160 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.333+0000 7fb4a0c00700 1 -- 172.21.15.103:0/3487131569 <== mgr.14120 v2:172.21.15.103:6800/3762323194 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb4a40627a0 con 0x7fb488039160 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.337+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fb488039160 msgr2=0x7fb48803b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.337+0000 7fb4ab000700 1 --2- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fb488039160 0x7fb48803b620 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb490006fd0 tx=0x7fb490006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.338+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 msgr2=0x7fb4a419dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.338+0000 7fb4ab000700 1 --2- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a419dea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb494011b60 tx=0x7fb494011c40 comp rx=0 tx=0).stop 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.338+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 shutdown_connections 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.338+0000 7fb4ab000700 1 --2- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fb488039160 0x7fb48803b620 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.338+0000 7fb4ab000700 1 --2- 172.21.15.103:0/3487131569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4a4108410 0x7fb4a419dea0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.338+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 >> 172.21.15.103:0/3487131569 conn(0x7fb4a4101ab0 msgr2=0x7fb4a4194560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.338+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 shutdown_connections 2024-05-22T23:49:11.089 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:10.338+0000 7fb4ab000700 1 -- 172.21.15.103:0/3487131569 wait complete. 2024-05-22T23:49:11.090 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying unmanaged mon service... 2024-05-22T23:49:11.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:11 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:11.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:11 smithi103 ceph-mon[31370]: Added host smithi103 2024-05-22T23:49:11.595 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:11 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:49:13.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:13 smithi103 ceph-mon[31370]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:13.161 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:13 smithi103 ceph-mon[31370]: Saving service mon spec with placement count:5 2024-05-22T23:49:13.162 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:13 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.759+0000 7f1d05400700 1 Processor -- start 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.759+0000 7f1d05400700 1 -- start start 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.759+0000 7f1d05400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d00108060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.760+0000 7f1d05400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d00108630 con 0x7f1d00107c40 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.760+0000 7f1cffe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d00108060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.760+0000 7f1cffe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d00108060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46074/0 (socket says 172.21.15.103:46074) 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.760+0000 7f1cffe00700 1 -- 172.21.15.103:0/4226994849 learned_addr learned my addr 172.21.15.103:0/4226994849 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.761+0000 7f1cffe00700 1 -- 172.21.15.103:0/4226994849 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d00108e40 con 0x7f1d00107c40 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.761+0000 7f1cffe00700 1 --2- 172.21.15.103:0/4226994849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d00108060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009e80 tx=0x7f1cf000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d966529e33f1253 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:13.177 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.761+0000 7f1cff400700 1 -- 172.21.15.103:0/4226994849 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cf0017070 con 0x7f1d00107c40 2024-05-22T23:49:13.178 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.761+0000 7f1cff400700 1 -- 172.21.15.103:0/4226994849 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1cf000b8c0 con 0x7f1d00107c40 2024-05-22T23:49:13.178 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.761+0000 7f1cff400700 1 -- 172.21.15.103:0/4226994849 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cf00045c0 con 0x7f1d00107c40 2024-05-22T23:49:13.178 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.762+0000 7f1d05400700 1 -- 172.21.15.103:0/4226994849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 msgr2=0x7f1d00108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:13.178 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.762+0000 7f1d05400700 1 --2- 172.21.15.103:0/4226994849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d00108060 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009e80 tx=0x7f1cf000b330 comp rx=0 tx=0).stop 2024-05-22T23:49:13.178 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.762+0000 7f1d05400700 1 -- 172.21.15.103:0/4226994849 shutdown_connections 2024-05-22T23:49:13.178 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.762+0000 7f1d05400700 1 --2- 172.21.15.103:0/4226994849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d00108060 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.762+0000 7f1d05400700 1 -- 172.21.15.103:0/4226994849 >> 172.21.15.103:0/4226994849 conn(0x7f1d00103220 msgr2=0x7f1d00105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.762+0000 7f1d05400700 1 -- 172.21.15.103:0/4226994849 shutdown_connections 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.762+0000 7f1d05400700 1 -- 172.21.15.103:0/4226994849 wait complete. 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.763+0000 7f1d05400700 1 Processor -- start 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.763+0000 7f1d05400700 1 -- start start 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.764+0000 7f1d05400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d0019dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.764+0000 7f1d05400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0010c1f0 con 0x7f1d00107c40 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.764+0000 7f1cffe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d0019dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:13.179 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.764+0000 7f1cffe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d0019dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46088/0 (socket says 172.21.15.103:46088) 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.764+0000 7f1cffe00700 1 -- 172.21.15.103:0/1611790101 learned_addr learned my addr 172.21.15.103:0/1611790101 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.764+0000 7f1cffe00700 1 -- 172.21.15.103:0/1611790101 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d0019e2d0 con 0x7f1d00107c40 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.764+0000 7f1cffe00700 1 --2- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d0019dd90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0004210 tx=0x7f1cf000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.765+0000 7f1cfd600700 1 -- 172.21.15.103:0/1611790101 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cf0020020 con 0x7f1d00107c40 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.765+0000 7f1cfd600700 1 -- 172.21.15.103:0/1611790101 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1cf0003b50 con 0x7f1d00107c40 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.765+0000 7f1cfd600700 1 -- 172.21.15.103:0/1611790101 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1cf0017040 con 0x7f1d00107c40 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.765+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d0019e560 con 0x7f1d00107c40 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.765+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d001a1210 con 0x7f1d00107c40 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.766+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d00069490 con 0x7f1d00107c40 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.766+0000 7f1cfd600700 1 -- 172.21.15.103:0/1611790101 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f1cf0023020 con 0x7f1d00107c40 2024-05-22T23:49:13.180 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.766+0000 7f1cfd600700 1 --2- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f1ce0039160 0x7f1ce003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.766+0000 7f1cfd600700 1 -- 172.21.15.103:0/1611790101 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1cf000f050 con 0x7f1d00107c40 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.767+0000 7f1cefe00700 1 --2- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f1ce0039160 0x7f1ce003b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.767+0000 7f1cefe00700 1 --2- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f1ce0039160 0x7f1ce003b620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8006fd0 tx=0x7f1ce8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:11.772+0000 7f1cfd600700 1 -- 172.21.15.103:0/1611790101 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1cf002a020 con 0x7f1d00107c40 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.005+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f1d00104310 con 0x7f1ce0039160 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.012+0000 7f1cfd600700 1 -- 172.21.15.103:0/1611790101 <== mgr.14120 v2:172.21.15.103:6800/3762323194 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f1d00104310 con 0x7f1ce0039160 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.014+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f1ce0039160 msgr2=0x7f1ce003b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.014+0000 7f1d05400700 1 --2- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f1ce0039160 0x7f1ce003b620 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8006fd0 tx=0x7f1ce8006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.014+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 msgr2=0x7f1d0019dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.014+0000 7f1d05400700 1 --2- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d0019dd90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0004210 tx=0x7f1cf000bd50 comp rx=0 tx=0).stop 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.015+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 shutdown_connections 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.015+0000 7f1d05400700 1 --2- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f1ce0039160 0x7f1ce003b620 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:13.181 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.015+0000 7f1d05400700 1 --2- 172.21.15.103:0/1611790101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1d00107c40 0x7f1d0019dd90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:13.182 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.015+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 >> 172.21.15.103:0/1611790101 conn(0x7f1d00103220 msgr2=0x7f1d00103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:13.182 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.015+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 shutdown_connections 2024-05-22T23:49:13.182 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:12.015+0000 7f1d05400700 1 -- 172.21.15.103:0/1611790101 wait complete. 2024-05-22T23:49:13.182 INFO:teuthology.orchestra.run.smithi103.stdout:Deploying unmanaged mgr service... 2024-05-22T23:49:15.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:14 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:15.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:14 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:15.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:14 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:15.466 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-22T23:49:15.466 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.448+0000 7fd385c00700 1 Processor -- start 2024-05-22T23:49:15.466 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.448+0000 7fd385c00700 1 -- start start 2024-05-22T23:49:15.466 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.448+0000 7fd385c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd380108220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.448+0000 7fd385c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd380108760 con 0x7fd380107e00 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.449+0000 7fd37ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd380108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.449+0000 7fd37ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd380108220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49830/0 (socket says 172.21.15.103:49830) 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.449+0000 7fd37ea00700 1 -- 172.21.15.103:0/1586485850 learned_addr learned my addr 172.21.15.103:0/1586485850 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.449+0000 7fd37ea00700 1 -- 172.21.15.103:0/1586485850 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd380108f60 con 0x7fd380107e00 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.450+0000 7fd37ea00700 1 --2- 172.21.15.103:0/1586485850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd380108220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd3700098a0 tx=0x7fd370009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d055b6289500c765 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.450+0000 7fd37d600700 1 -- 172.21.15.103:0/1586485850 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd370004160 con 0x7fd380107e00 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.450+0000 7fd37d600700 1 -- 172.21.15.103:0/1586485850 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd370004300 con 0x7fd380107e00 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.450+0000 7fd37d600700 1 -- 172.21.15.103:0/1586485850 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd370011b20 con 0x7fd380107e00 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.451+0000 7fd385c00700 1 -- 172.21.15.103:0/1586485850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 msgr2=0x7fd380108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.451+0000 7fd385c00700 1 --2- 172.21.15.103:0/1586485850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd380108220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd3700098a0 tx=0x7fd370009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.451+0000 7fd385c00700 1 -- 172.21.15.103:0/1586485850 shutdown_connections 2024-05-22T23:49:15.467 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.451+0000 7fd385c00700 1 --2- 172.21.15.103:0/1586485850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd380108220 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.451+0000 7fd385c00700 1 -- 172.21.15.103:0/1586485850 >> 172.21.15.103:0/1586485850 conn(0x7fd3801033c0 msgr2=0x7fd3801057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.451+0000 7fd385c00700 1 -- 172.21.15.103:0/1586485850 shutdown_connections 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.451+0000 7fd385c00700 1 -- 172.21.15.103:0/1586485850 wait complete. 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.452+0000 7fd385c00700 1 Processor -- start 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.453+0000 7fd385c00700 1 -- start start 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.453+0000 7fd385c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd38007db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.453+0000 7fd385c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd38010c3a0 con 0x7fd380107e00 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.453+0000 7fd37ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd38007db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.453+0000 7fd37ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd38007db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49832/0 (socket says 172.21.15.103:49832) 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.453+0000 7fd37ea00700 1 -- 172.21.15.103:0/576892807 learned_addr learned my addr 172.21.15.103:0/576892807 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.454+0000 7fd37ea00700 1 -- 172.21.15.103:0/576892807 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38007e0b0 con 0x7fd380107e00 2024-05-22T23:49:15.468 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.454+0000 7fd37ea00700 1 --2- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd38007db70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd3700117e0 tx=0x7fd370011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.454+0000 7fd377400700 1 -- 172.21.15.103:0/576892807 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd370003ba0 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.454+0000 7fd377400700 1 -- 172.21.15.103:0/576892807 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd37001a910 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.454+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38007c2a0 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.454+0000 7fd377400700 1 -- 172.21.15.103:0/576892807 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd3700248f0 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.455+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38007c760 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.456+0000 7fd377400700 1 -- 172.21.15.103:0/576892807 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fd370022020 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.456+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3800639c0 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.456+0000 7fd377400700 1 --2- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd364039160 0x7fd36403b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.456+0000 7fd377400700 1 -- 172.21.15.103:0/576892807 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd37004d4c0 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.457+0000 7fd37e000700 1 --2- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd364039160 0x7fd36403b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.457+0000 7fd37e000700 1 --2- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd364039160 0x7fd36403b620 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd36c006fd0 tx=0x7fd36c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.462+0000 7fd377400700 1 -- 172.21.15.103:0/576892807 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd37001ab70 con 0x7fd380107e00 2024-05-22T23:49:15.469 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.706+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd38007cf40 con 0x7fd364039160 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.714+0000 7fd377400700 1 -- 172.21.15.103:0/576892807 <== mgr.14120 v2:172.21.15.103:6800/3762323194 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd38007cf40 con 0x7fd364039160 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd364039160 msgr2=0x7fd36403b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 --2- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd364039160 0x7fd36403b620 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd36c006fd0 tx=0x7fd36c006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 msgr2=0x7fd38007db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 --2- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd38007db70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd3700117e0 tx=0x7fd370011ba0 comp rx=0 tx=0).stop 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 shutdown_connections 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 --2- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd364039160 0x7fd36403b620 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 --2- 172.21.15.103:0/576892807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd380107e00 0x7fd38007db70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 >> 172.21.15.103:0/576892807 conn(0x7fd3801033c0 msgr2=0x7fd380103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 shutdown_connections 2024-05-22T23:49:15.470 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:14.717+0000 7fd385c00700 1 -- 172.21.15.103:0/576892807 wait complete. 2024-05-22T23:49:16.055 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:15 smithi103 ceph-mon[31370]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:16.055 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:15 smithi103 ceph-mon[31370]: Saving service mgr spec with placement count:2 2024-05-22T23:49:16.055 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:15 smithi103 ceph-mon[31370]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:49:17.066 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.049+0000 7fd5bda00700 1 Processor -- start 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.049+0000 7fd5bda00700 1 -- start start 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.050+0000 7fd5bda00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b810ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.050+0000 7fd5bda00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b807bdd0 con 0x7fd5b8108950 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.051+0000 7fd5b6a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b810ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.051+0000 7fd5b6a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b810ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49842/0 (socket says 172.21.15.103:49842) 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.051+0000 7fd5b6a00700 1 -- 172.21.15.103:0/930237012 learned_addr learned my addr 172.21.15.103:0/930237012 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.051+0000 7fd5b6a00700 1 -- 172.21.15.103:0/930237012 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b810b280 con 0x7fd5b8108950 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.052+0000 7fd5b6a00700 1 --2- 172.21.15.103:0/930237012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b810ad40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80098a0 tx=0x7fd5a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86446aae52603590 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.052+0000 7fd5b5600700 1 -- 172.21.15.103:0/930237012 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a8004160 con 0x7fd5b8108950 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.052+0000 7fd5b5600700 1 -- 172.21.15.103:0/930237012 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5a8004300 con 0x7fd5b8108950 2024-05-22T23:49:17.067 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.052+0000 7fd5b5600700 1 -- 172.21.15.103:0/930237012 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a8011b20 con 0x7fd5b8108950 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.053+0000 7fd5bda00700 1 -- 172.21.15.103:0/930237012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 msgr2=0x7fd5b810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.053+0000 7fd5bda00700 1 --2- 172.21.15.103:0/930237012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b810ad40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80098a0 tx=0x7fd5a8009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.053+0000 7fd5bda00700 1 -- 172.21.15.103:0/930237012 shutdown_connections 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.053+0000 7fd5bda00700 1 --2- 172.21.15.103:0/930237012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b810ad40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.053+0000 7fd5bda00700 1 -- 172.21.15.103:0/930237012 >> 172.21.15.103:0/930237012 conn(0x7fd5b81022a0 msgr2=0x7fd5b81046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.053+0000 7fd5bda00700 1 -- 172.21.15.103:0/930237012 shutdown_connections 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.053+0000 7fd5bda00700 1 -- 172.21.15.103:0/930237012 wait complete. 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.054+0000 7fd5bda00700 1 Processor -- start 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.055+0000 7fd5bda00700 1 -- start start 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.055+0000 7fd5bda00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b81081d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.055+0000 7fd5bda00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5b810c220 con 0x7fd5b8108950 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.055+0000 7fd5b6a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b81081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:17.068 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.055+0000 7fd5b6a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b81081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49854/0 (socket says 172.21.15.103:49854) 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.055+0000 7fd5b6a00700 1 -- 172.21.15.103:0/3465935032 learned_addr learned my addr 172.21.15.103:0/3465935032 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.056+0000 7fd5b6a00700 1 -- 172.21.15.103:0/3465935032 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5b8108710 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.056+0000 7fd5b6a00700 1 --2- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b81081d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80117e0 tx=0x7fd5a8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.056+0000 7fd5af400700 1 -- 172.21.15.103:0/3465935032 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a8003ba0 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.056+0000 7fd5af400700 1 -- 172.21.15.103:0/3465935032 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5a801a910 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.056+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5b8106900 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.056+0000 7fd5af400700 1 -- 172.21.15.103:0/3465935032 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5a80248f0 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.057+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5b8106dc0 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.058+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5b80639c0 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.058+0000 7fd5af400700 1 -- 172.21.15.103:0/3465935032 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fd5a8022020 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.058+0000 7fd5af400700 1 --2- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd59c039110 0x7fd59c03b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.059+0000 7fd5af400700 1 -- 172.21.15.103:0/3465935032 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd5a804d580 con 0x7fd5b8108950 2024-05-22T23:49:17.069 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.059+0000 7fd5b6000700 1 --2- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd59c039110 0x7fd59c03b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.060+0000 7fd5b6000700 1 --2- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd59c039110 0x7fd59c03b5d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4006fd0 tx=0x7fd5a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.065+0000 7fd5af400700 1 -- 172.21.15.103:0/3465935032 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd5a801ab70 con 0x7fd5b8108950 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.282+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fd5b8107530 con 0x7fd5b8108950 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.288+0000 7fd5af400700 1 -- 172.21.15.103:0/3465935032 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fd5a80212c0 con 0x7fd5b8108950 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd59c039110 msgr2=0x7fd59c03b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 --2- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd59c039110 0x7fd59c03b5d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4006fd0 tx=0x7fd5a4006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 msgr2=0x7fd5b81081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 --2- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b81081d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd5a80117e0 tx=0x7fd5a8011ba0 comp rx=0 tx=0).stop 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 shutdown_connections 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 --2- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fd59c039110 0x7fd59c03b5d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 --2- 172.21.15.103:0/3465935032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5b8108950 0x7fd5b81081d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:17.070 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 >> 172.21.15.103:0/3465935032 conn(0x7fd5b81022a0 msgr2=0x7fd5b8102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:17.071 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 shutdown_connections 2024-05-22T23:49:17.071 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:16.292+0000 7fd5bda00700 1 -- 172.21.15.103:0/3465935032 wait complete. 2024-05-22T23:49:17.307 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:17 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3465935032' entity='client.admin' 2024-05-22T23:49:17.307 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:17 smithi103 ceph-mon[31370]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:49:19.066 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.099+0000 7fa8d2000700 1 Processor -- start 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.099+0000 7fa8d2000700 1 -- start start 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.099+0000 7fa8d2000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc108730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.099+0000 7fa8d2000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc108d00 con 0x7fa8cc108310 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.100+0000 7fa8cb400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc108730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.100+0000 7fa8cb400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc108730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49858/0 (socket says 172.21.15.103:49858) 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.100+0000 7fa8cb400700 1 -- 172.21.15.103:0/1151533456 learned_addr learned my addr 172.21.15.103:0/1151533456 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.100+0000 7fa8cb400700 1 -- 172.21.15.103:0/1151533456 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8cc108e80 con 0x7fa8cc108310 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.100+0000 7fa8cb400700 1 --2- 172.21.15.103:0/1151533456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc108730 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0098a0 tx=0x7fa8bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43757fbf2056f601 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.101+0000 7fa8ca000700 1 -- 172.21.15.103:0/1151533456 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8bc004160 con 0x7fa8cc108310 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.101+0000 7fa8ca000700 1 -- 172.21.15.103:0/1151533456 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8bc004300 con 0x7fa8cc108310 2024-05-22T23:49:19.072 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.101+0000 7fa8ca000700 1 -- 172.21.15.103:0/1151533456 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8bc011b20 con 0x7fa8cc108310 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.101+0000 7fa8d2000700 1 -- 172.21.15.103:0/1151533456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 msgr2=0x7fa8cc108730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.101+0000 7fa8d2000700 1 --2- 172.21.15.103:0/1151533456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc108730 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0098a0 tx=0x7fa8bc009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.101+0000 7fa8d2000700 1 -- 172.21.15.103:0/1151533456 shutdown_connections 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.102+0000 7fa8d2000700 1 --2- 172.21.15.103:0/1151533456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc108730 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.102+0000 7fa8d2000700 1 -- 172.21.15.103:0/1151533456 >> 172.21.15.103:0/1151533456 conn(0x7fa8cc101a20 msgr2=0x7fa8cc103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.102+0000 7fa8d2000700 1 -- 172.21.15.103:0/1151533456 shutdown_connections 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.102+0000 7fa8d2000700 1 -- 172.21.15.103:0/1151533456 wait complete. 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.103+0000 7fa8d2000700 1 Processor -- start 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.103+0000 7fa8d2000700 1 -- start start 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.103+0000 7fa8d2000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc19de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.103+0000 7fa8d2000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa8cc106d30 con 0x7fa8cc108310 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.104+0000 7fa8cb400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc19de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.104+0000 7fa8cb400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc19de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49860/0 (socket says 172.21.15.103:49860) 2024-05-22T23:49:19.073 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.104+0000 7fa8cb400700 1 -- 172.21.15.103:0/3893678506 learned_addr learned my addr 172.21.15.103:0/3893678506 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.104+0000 7fa8cb400700 1 -- 172.21.15.103:0/3893678506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8cc19e350 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.104+0000 7fa8cb400700 1 --2- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc19de10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0117e0 tx=0x7fa8bc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.104+0000 7fa8c3e00700 1 -- 172.21.15.103:0/3893678506 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8bc003bf0 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.105+0000 7fa8c3e00700 1 -- 172.21.15.103:0/3893678506 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8bc01a910 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.105+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8cc19e640 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.105+0000 7fa8c3e00700 1 -- 172.21.15.103:0/3893678506 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa8bc0249a0 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.105+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8cc1a12d0 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.106+0000 7fa8c3e00700 1 -- 172.21.15.103:0/3893678506 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fa8bc022020 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.106+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8cc0639c0 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.106+0000 7fa8c3e00700 1 --2- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa8b00390c0 0x7fa8b003b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.106+0000 7fa8c3e00700 1 -- 172.21.15.103:0/3893678506 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa8bc04d4b0 con 0x7fa8cc108310 2024-05-22T23:49:19.074 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.107+0000 7fa8caa00700 1 --2- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa8b00390c0 0x7fa8b003b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.107+0000 7fa8caa00700 1 --2- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa8b00390c0 0x7fa8b003b580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8006fd0 tx=0x7fa8b8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.112+0000 7fa8c3e00700 1 -- 172.21.15.103:0/3893678506 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa8bc011e20 con 0x7fa8cc108310 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.336+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fa8cc1a14a0 con 0x7fa8cc108310 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.342+0000 7fa8c3e00700 1 -- 172.21.15.103:0/3893678506 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fa8cc1a14a0 con 0x7fa8cc108310 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.345+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa8b00390c0 msgr2=0x7fa8b003b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.345+0000 7fa8d2000700 1 --2- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa8b00390c0 0x7fa8b003b580 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8006fd0 tx=0x7fa8b8006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.345+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 msgr2=0x7fa8cc19de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.345+0000 7fa8d2000700 1 --2- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc19de10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc0117e0 tx=0x7fa8bc011c10 comp rx=0 tx=0).stop 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.346+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 shutdown_connections 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.346+0000 7fa8d2000700 1 --2- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa8b00390c0 0x7fa8b003b580 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.346+0000 7fa8d2000700 1 --2- 172.21.15.103:0/3893678506 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa8cc108310 0x7fa8cc19de10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.346+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 >> 172.21.15.103:0/3893678506 conn(0x7fa8cc101a20 msgr2=0x7fa8cc194450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:19.075 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.346+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 shutdown_connections 2024-05-22T23:49:19.076 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:18.346+0000 7fa8d2000700 1 -- 172.21.15.103:0/3893678506 wait complete. 2024-05-22T23:49:19.076 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling the dashboard module... 2024-05-22T23:49:19.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:18 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:19.227 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:18 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3893678506' entity='client.admin' 2024-05-22T23:49:20.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:19 smithi103 ceph-mon[31370]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:49:21.164 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:20 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2383866445' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-22T23:49:21.165 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:20 smithi103 ceph-mon[31370]: from='mgr.14120 172.21.15.103:0/1729608894' entity='mgr.x' 2024-05-22T23:49:21.165 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:21 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: ignoring --setuser ceph since I am not root 2024-05-22T23:49:21.165 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:21 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: ignoring --setgroup ceph since I am not root 2024-05-22T23:49:21.472 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:21 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:21.219+0000 7fb49591c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-22T23:49:21.831 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:21 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:21.472+0000 7fb49591c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:49:21.831 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:21 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:21.604+0000 7fb49591c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-22T23:49:22.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:21 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2383866445' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-22T23:49:22.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:21 smithi103 ceph-mon[31370]: mgrmap e9: x(active, since 25s) 2024-05-22T23:49:22.082 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:22 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:22.010+0000 7fb49591c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-22T23:49:22.399 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:22 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:22.125+0000 7fb49591c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-22T23:49:22.407 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.650+0000 7fa487600700 1 Processor -- start 2024-05-22T23:49:22.408 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.650+0000 7fa487600700 1 -- start start 2024-05-22T23:49:22.409 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.650+0000 7fa487600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48007c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:22.409 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.651+0000 7fa487600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa48007c8b0 con 0x7fa48007de80 2024-05-22T23:49:22.409 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.651+0000 7fa484c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48007c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:22.413 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.651+0000 7fa484c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48007c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49862/0 (socket says 172.21.15.103:49862) 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.651+0000 7fa484c00700 1 -- 172.21.15.103:0/2539463710 learned_addr learned my addr 172.21.15.103:0/2539463710 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.652+0000 7fa484c00700 1 -- 172.21.15.103:0/2539463710 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48007ca30 con 0x7fa48007de80 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.652+0000 7fa484c00700 1 --2- 172.21.15.103:0/2539463710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48007c2e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4740098a0 tx=0x7fa474009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db56e4365de94cbb server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.652+0000 7fa47f400700 1 -- 172.21.15.103:0/2539463710 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa474004160 con 0x7fa48007de80 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.652+0000 7fa47f400700 1 -- 172.21.15.103:0/2539463710 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa474004300 con 0x7fa48007de80 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.652+0000 7fa47f400700 1 -- 172.21.15.103:0/2539463710 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa474011b20 con 0x7fa48007de80 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.653+0000 7fa487600700 1 -- 172.21.15.103:0/2539463710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 msgr2=0x7fa48007c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.653+0000 7fa487600700 1 --2- 172.21.15.103:0/2539463710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48007c2e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa4740098a0 tx=0x7fa474009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:22.414 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.653+0000 7fa487600700 1 -- 172.21.15.103:0/2539463710 shutdown_connections 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.653+0000 7fa487600700 1 --2- 172.21.15.103:0/2539463710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48007c2e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.653+0000 7fa487600700 1 -- 172.21.15.103:0/2539463710 >> 172.21.15.103:0/2539463710 conn(0x7fa4801033c0 msgr2=0x7fa4801057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.653+0000 7fa487600700 1 -- 172.21.15.103:0/2539463710 shutdown_connections 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.653+0000 7fa487600700 1 -- 172.21.15.103:0/2539463710 wait complete. 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.654+0000 7fa487600700 1 Processor -- start 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.655+0000 7fa487600700 1 -- start start 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.655+0000 7fa487600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.655+0000 7fa487600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa48010e480 con 0x7fa48007de80 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.655+0000 7fa484c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.655+0000 7fa484c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48019de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49866/0 (socket says 172.21.15.103:49866) 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.655+0000 7fa484c00700 1 -- 172.21.15.103:0/2383866445 learned_addr learned my addr 172.21.15.103:0/2383866445 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.656+0000 7fa484c00700 1 -- 172.21.15.103:0/2383866445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48019e3b0 con 0x7fa48007de80 2024-05-22T23:49:22.415 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.656+0000 7fa484c00700 1 --2- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48019de70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa4740117e0 tx=0x7fa474011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.656+0000 7fa47d600700 1 -- 172.21.15.103:0/2383866445 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa474003ba0 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.656+0000 7fa47d600700 1 -- 172.21.15.103:0/2383866445 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa47401a910 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.656+0000 7fa47d600700 1 -- 172.21.15.103:0/2383866445 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa4740248f0 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.656+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa48019e640 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.657+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa48019eaa0 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.658+0000 7fa47d600700 1 -- 172.21.15.103:0/2383866445 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fa474022020 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.658+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4800639c0 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.658+0000 7fa47d600700 1 --2- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa468039110 0x7fa46803b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.658+0000 7fa47d600700 1 -- 172.21.15.103:0/2383866445 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa47404d6b0 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.658+0000 7fa47fe00700 1 --2- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa468039110 0x7fa46803b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.659+0000 7fa47fe00700 1 --2- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa468039110 0x7fa46803b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa46c006fd0 tx=0x7fa46c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.664+0000 7fa47d600700 1 -- 172.21.15.103:0/2383866445 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa47400b3e0 con 0x7fa48007de80 2024-05-22T23:49:22.416 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:19.963+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fa4801a1470 con 0x7fa48007de80 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.908+0000 7fa47d600700 1 -- 172.21.15.103:0/2383866445 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fa47401ab30 con 0x7fa48007de80 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.911+0000 7fa47d600700 1 -- 172.21.15.103:0/2383866445 <== mon.0 v2:172.21.15.103:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fa474018350 con 0x7fa48007de80 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.912+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa468039110 msgr2=0x7fa46803b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.912+0000 7fa487600700 1 --2- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa468039110 0x7fa46803b5d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa46c006fd0 tx=0x7fa46c006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.912+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 msgr2=0x7fa48019de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.912+0000 7fa487600700 1 --2- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48019de70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa4740117e0 tx=0x7fa474011ba0 comp rx=0 tx=0).stop 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.913+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 shutdown_connections 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.913+0000 7fa487600700 1 --2- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7fa468039110 0x7fa46803b5d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.913+0000 7fa487600700 1 --2- 172.21.15.103:0/2383866445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa48007de80 0x7fa48019de70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.913+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 >> 172.21.15.103:0/2383866445 conn(0x7fa4801033c0 msgr2=0x7fa480103d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.913+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 shutdown_connections 2024-05-22T23:49:22.417 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:20.913+0000 7fa487600700 1 -- 172.21.15.103:0/2383866445 wait complete. 2024-05-22T23:49:23.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:23 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2275116621' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-22T23:49:23.831 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:23 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:23.769+0000 7fb49591c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-22T23:49:24.323 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:24.007+0000 7fb49591c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-22T23:49:24.323 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:24.227+0000 7fb49591c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-22T23:49:24.335 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-05-22T23:49:24.335 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.240+0000 7f02d4000700 1 Processor -- start 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.241+0000 7f02d4000700 1 -- start start 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.241+0000 7f02d4000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc07c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.241+0000 7f02d4000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc07c910 con 0x7f02cc07dee0 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.242+0000 7f02d1600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc07c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.242+0000 7f02d1600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc07c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49890/0 (socket says 172.21.15.103:49890) 2024-05-22T23:49:24.336 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.242+0000 7f02d1600700 1 -- 172.21.15.103:0/154227469 learned_addr learned my addr 172.21.15.103:0/154227469 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:24.337 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.242+0000 7f02d1600700 1 -- 172.21.15.103:0/154227469 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc07ca90 con 0x7f02cc07dee0 2024-05-22T23:49:24.338 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.243+0000 7f02d1600700 1 --2- 172.21.15.103:0/154227469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc07c340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02c00098a0 tx=0x7f02c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99e70bd1b810f67 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:24.338 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.243+0000 7f02cbe00700 1 -- 172.21.15.103:0/154227469 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c0004160 con 0x7f02cc07dee0 2024-05-22T23:49:24.338 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.243+0000 7f02cbe00700 1 -- 172.21.15.103:0/154227469 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c0004300 con 0x7f02cc07dee0 2024-05-22T23:49:24.338 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.243+0000 7f02cbe00700 1 -- 172.21.15.103:0/154227469 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c0011b20 con 0x7f02cc07dee0 2024-05-22T23:49:24.338 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.244+0000 7f02d4000700 1 -- 172.21.15.103:0/154227469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 msgr2=0x7f02cc07c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.244+0000 7f02d4000700 1 --2- 172.21.15.103:0/154227469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc07c340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02c00098a0 tx=0x7f02c0009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.244+0000 7f02d4000700 1 -- 172.21.15.103:0/154227469 shutdown_connections 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.244+0000 7f02d4000700 1 --2- 172.21.15.103:0/154227469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc07c340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.244+0000 7f02d4000700 1 -- 172.21.15.103:0/154227469 >> 172.21.15.103:0/154227469 conn(0x7f02cc1033c0 msgr2=0x7f02cc1057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.244+0000 7f02d4000700 1 -- 172.21.15.103:0/154227469 shutdown_connections 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.244+0000 7f02d4000700 1 -- 172.21.15.103:0/154227469 wait complete. 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.246+0000 7f02d4000700 1 Processor -- start 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.246+0000 7f02d4000700 1 -- start start 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.246+0000 7f02d4000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc19df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.247+0000 7f02d4000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f02cc10e470 con 0x7f02cc07dee0 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.247+0000 7f02d1600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc19df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.247+0000 7f02d1600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc19df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49900/0 (socket says 172.21.15.103:49900) 2024-05-22T23:49:24.339 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.247+0000 7f02d1600700 1 -- 172.21.15.103:0/2275116621 learned_addr learned my addr 172.21.15.103:0/2275116621 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.247+0000 7f02d1600700 1 -- 172.21.15.103:0/2275116621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02cc19e440 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.248+0000 7f02d1600700 1 --2- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc19df00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02c00117e0 tx=0x7f02c0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.248+0000 7f02ca000700 1 -- 172.21.15.103:0/2275116621 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c0003bf0 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.248+0000 7f02ca000700 1 -- 172.21.15.103:0/2275116621 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02c0003d90 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.248+0000 7f02ca000700 1 -- 172.21.15.103:0/2275116621 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f02c0024800 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.248+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02cc19e6d0 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.248+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02cc19eb30 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.250+0000 7f02ca000700 1 -- 172.21.15.103:0/2275116621 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f02c0022020 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.250+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02cc0639c0 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.250+0000 7f02ca000700 1 --2- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f02b40391c0 0x7f02b403b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.250+0000 7f02ca000700 1 -- 172.21.15.103:0/2275116621 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f02c004d390 con 0x7f02cc07dee0 2024-05-22T23:49:24.340 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.250+0000 7f02d0c00700 1 -- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f02b40391c0 msgr2=0x7f02b403b680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3762323194 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.250+0000 7f02d0c00700 1 --2- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f02b40391c0 0x7f02b403b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.256+0000 7f02ca000700 1 -- 172.21.15.103:0/2275116621 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f02c004cce0 con 0x7f02cc07dee0 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.451+0000 7f02d0c00700 1 -- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f02b40391c0 msgr2=0x7f02b403b680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3762323194 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.451+0000 7f02d0c00700 1 --2- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f02b40391c0 0x7f02b403b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.551+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f02cc1a14e0 con 0x7f02cc07dee0 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.552+0000 7f02ca000700 1 -- 172.21.15.103:0/2275116621 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f02c001ac50 con 0x7f02cc07dee0 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.555+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f02b40391c0 msgr2=0x7f02b403b680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.555+0000 7f02d4000700 1 --2- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f02b40391c0 0x7f02b403b680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.555+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 msgr2=0x7f02cc19df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.555+0000 7f02d4000700 1 --2- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc19df00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02c00117e0 tx=0x7f02c0011ba0 comp rx=0 tx=0).stop 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.555+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 shutdown_connections 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.555+0000 7f02d4000700 1 --2- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f02b40391c0 0x7f02b403b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:24.341 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.555+0000 7f02d4000700 1 --2- 172.21.15.103:0/2275116621 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02cc07dee0 0x7f02cc19df00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:24.342 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.555+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 >> 172.21.15.103:0/2275116621 conn(0x7f02cc1033c0 msgr2=0x7f02cc103e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:24.342 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.556+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 shutdown_connections 2024-05-22T23:49:24.342 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:23.556+0000 7f02d4000700 1 -- 172.21.15.103:0/2275116621 wait complete. 2024-05-22T23:49:24.342 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for the mgr to restart... 2024-05-22T23:49:24.342 INFO:teuthology.orchestra.run.smithi103.stdout:Waiting for mgr epoch 9... 2024-05-22T23:49:24.893 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:24.628+0000 7fb49591c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-22T23:49:24.893 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:24.754+0000 7fb49591c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-22T23:49:25.274 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:25 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:25.015+0000 7fb49591c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-22T23:49:25.536 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:25 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:25.269+0000 7fb49591c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:49:26.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:26.058+0000 7fb49591c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-22T23:49:27.004 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:26.608+0000 7fb49591c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-22T23:49:27.004 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:26.736+0000 7fb49591c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-22T23:49:27.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:26.999+0000 7fb49591c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-22T23:49:27.939 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:27.640+0000 7fb49591c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-22T23:49:27.940 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:27.763+0000 7fb49591c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-22T23:49:28.272 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:28.087+0000 7fb49591c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-22T23:49:28.550 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:28.267+0000 7fb49591c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-22T23:49:28.551 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:28.393+0000 7fb49591c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-22T23:49:28.551 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:28.545+0000 7fb49591c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-22T23:49:29.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:28.911+0000 7fb49591c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-22T23:49:29.955 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:29.664+0000 7fb49591c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-22T23:49:30.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:49:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:49:29.950+0000 7fb49591c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-22T23:49:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: Active manager daemon x restarted 2024-05-22T23:49:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: Activating manager daemon x 2024-05-22T23:49:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: osdmap e3: 0 total, 0 up, 0 in 2024-05-22T23:49:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: mgrmap e10: x(active, starting, since 0.00888028s) 2024-05-22T23:49:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:49:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-22T23:49:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-22T23:49:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-22T23:49:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-22T23:49:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:30 smithi103 ceph-mon[31370]: Manager daemon x is now available 2024-05-22T23:49:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:49:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-22T23:49:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-22T23:49:31.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:31 smithi103 ceph-mon[31370]: mgrmap e11: x(active, since 1.00872s) 2024-05-22T23:49:31.740 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout { 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout } 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.922+0000 7f85f0000700 1 Processor -- start 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.922+0000 7f85f0000700 1 -- start start 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.922+0000 7f85f0000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e81082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.922+0000 7f85f0000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85e8108800 con 0x7f85e8107ea0 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.923+0000 7f85ed600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e81082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.923+0000 7f85ed600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e81082c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50208/0 (socket says 172.21.15.103:50208) 2024-05-22T23:49:31.741 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.923+0000 7f85ed600700 1 -- 172.21.15.103:0/2418738915 learned_addr learned my addr 172.21.15.103:0/2418738915 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:31.742 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.923+0000 7f85ed600700 1 -- 172.21.15.103:0/2418738915 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e8108f60 con 0x7f85e8107ea0 2024-05-22T23:49:31.742 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.924+0000 7f85ed600700 1 --2- 172.21.15.103:0/2418738915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e81082c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0098a0 tx=0x7f85dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6dd5baea203399 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:31.742 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.924+0000 7f85e7e00700 1 -- 172.21.15.103:0/2418738915 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85dc004160 con 0x7f85e8107ea0 2024-05-22T23:49:31.742 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.924+0000 7f85e7e00700 1 -- 172.21.15.103:0/2418738915 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85dc004300 con 0x7f85e8107ea0 2024-05-22T23:49:31.742 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.924+0000 7f85e7e00700 1 -- 172.21.15.103:0/2418738915 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85dc011b20 con 0x7f85e8107ea0 2024-05-22T23:49:31.742 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.925+0000 7f85f0000700 1 -- 172.21.15.103:0/2418738915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 msgr2=0x7f85e81082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:31.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.925+0000 7f85f0000700 1 --2- 172.21.15.103:0/2418738915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e81082c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0098a0 tx=0x7f85dc009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:31.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.925+0000 7f85f0000700 1 -- 172.21.15.103:0/2418738915 shutdown_connections 2024-05-22T23:49:31.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.925+0000 7f85f0000700 1 --2- 172.21.15.103:0/2418738915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e81082c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:31.745 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.925+0000 7f85f0000700 1 -- 172.21.15.103:0/2418738915 >> 172.21.15.103:0/2418738915 conn(0x7f85e8103460 msgr2=0x7f85e8105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.925+0000 7f85f0000700 1 -- 172.21.15.103:0/2418738915 shutdown_connections 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.925+0000 7f85f0000700 1 -- 172.21.15.103:0/2418738915 wait complete. 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.926+0000 7f85f0000700 1 Processor -- start 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.926+0000 7f85f0000700 1 -- start start 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.927+0000 7f85f0000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.927+0000 7f85f0000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85e810c3a0 con 0x7f85e8107ea0 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.927+0000 7f85ed600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.927+0000 7f85ed600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50220/0 (socket says 172.21.15.103:50220) 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.927+0000 7f85ed600700 1 -- 172.21.15.103:0/2225838732 learned_addr learned my addr 172.21.15.103:0/2225838732 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.927+0000 7f85ed600700 1 -- 172.21.15.103:0/2225838732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e819e3b0 con 0x7f85e8107ea0 2024-05-22T23:49:31.746 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.928+0000 7f85ed600700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e819de70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0117e0 tx=0x7f85dc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.928+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85dc003ba0 con 0x7f85e8107ea0 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.928+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85dc01a910 con 0x7f85e8107ea0 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.928+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f85dc0248f0 con 0x7f85e8107ea0 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.928+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e819e640 con 0x7f85e8107ea0 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.928+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e819eaa0 con 0x7f85e8107ea0 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.929+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f85dc022020 con 0x7f85e8107ea0 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.929+0000 7f85e6000700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 0x7f85d003b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.929+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 --> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f85d003bd30 con 0x7f85d0039160 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.929+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f85dc04d600 con 0x7f85e8107ea0 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.930+0000 7f85ecc00700 1 -- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 msgr2=0x7f85d003b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3762323194 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:24.930+0000 7f85ecc00700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 0x7f85d003b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:25.131+0000 7f85ecc00700 1 -- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 msgr2=0x7f85d003b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3762323194 2024-05-22T23:49:31.747 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:25.131+0000 7f85ecc00700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 0x7f85d003b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:25.532+0000 7f85ecc00700 1 -- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 msgr2=0x7f85d003b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3762323194 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:25.532+0000 7f85ecc00700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 0x7f85d003b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:26.333+0000 7f85ecc00700 1 -- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 msgr2=0x7f85d003b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3762323194 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:26.333+0000 7f85ecc00700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 0x7f85d003b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:27.935+0000 7f85ecc00700 1 -- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 msgr2=0x7f85d003b620 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/3762323194 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:27.935+0000 7f85ecc00700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 0x7f85d003b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:29.958+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f85dc02d3f0 con 0x7f85e8107ea0 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:29.959+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 msgr2=0x7f85d003b620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:29.959+0000 7f85e6000700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 0x7f85d003b620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.958+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f85dc04f660 con 0x7f85e8107ea0 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.959+0000 7f85e6000700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f85d003cc20 0x7f85d003f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:31.748 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.959+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f85dc001d30 con 0x7f85d003cc20 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.960+0000 7f85ecc00700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f85d003cc20 0x7f85d003f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.963+0000 7f85ecc00700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f85d003cc20 0x7f85d003f010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f85d4003a10 tx=0x7f85d40092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.964+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f85dc001d30 con 0x7f85d003cc20 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.972+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f85e8069490 con 0x7f85d003cc20 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.972+0000 7f85e6000700 1 -- 172.21.15.103:0/2225838732 <== mgr.14152 v2:172.21.15.103:6800/1744216601 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f85e8069490 con 0x7f85d003cc20 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f85d003cc20 msgr2=0x7f85d003f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f85d003cc20 0x7f85d003f010 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f85d4003a10 tx=0x7f85d40092b0 comp rx=0 tx=0).stop 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 msgr2=0x7f85e819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e819de70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f85dc0117e0 tx=0x7f85dc011ba0 comp rx=0 tx=0).stop 2024-05-22T23:49:31.749 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 shutdown_connections 2024-05-22T23:49:31.750 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f85d003cc20 0x7f85d003f010 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:31.750 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e8107ea0 0x7f85e819de70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:31.750 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 --2- 172.21.15.103:0/2225838732 >> [v2:172.21.15.103:6800/3762323194,v1:172.21.15.103:6801/3762323194] conn(0x7f85d0039160 0x7f85d003b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:31.750 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 >> 172.21.15.103:0/2225838732 conn(0x7f85e8103460 msgr2=0x7f85e807c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:31.750 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 shutdown_connections 2024-05-22T23:49:31.750 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:30.973+0000 7f85f0000700 1 -- 172.21.15.103:0/2225838732 wait complete. 2024-05-22T23:49:31.750 INFO:teuthology.orchestra.run.smithi103.stdout:mgr epoch 9 is available 2024-05-22T23:49:31.750 INFO:teuthology.orchestra.run.smithi103.stdout:Generating a dashboard self-signed certificate... 2024-05-22T23:49:32.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:32 smithi103 ceph-mon[31370]: [22/May/2024:23:49:30] ENGINE Bus STARTING 2024-05-22T23:49:32.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:32 smithi103 ceph-mon[31370]: [22/May/2024:23:49:30] ENGINE Serving on http://172.21.15.103:8765 2024-05-22T23:49:32.288 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:32 smithi103 ceph-mon[31370]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-22T23:49:32.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:32 smithi103 ceph-mon[31370]: [22/May/2024:23:49:30] ENGINE Serving on https://172.21.15.103:7150 2024-05-22T23:49:32.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:32 smithi103 ceph-mon[31370]: [22/May/2024:23:49:30] ENGINE Bus STARTED 2024-05-22T23:49:32.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:32 smithi103 ceph-mon[31370]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-22T23:49:33.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:33 smithi103 ceph-mon[31370]: mgrmap e12: x(active, since 2s) 2024-05-22T23:49:33.062 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:33 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:34.079 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-22T23:49:34.079 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.966+0000 7f19a1600700 1 Processor -- start 2024-05-22T23:49:34.079 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.966+0000 7f19a1600700 1 -- start start 2024-05-22T23:49:34.079 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.967+0000 7f19a1600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c108220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:34.079 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.967+0000 7f19a1600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f199c108760 con 0x7f199c107e00 2024-05-22T23:49:34.079 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.967+0000 7f199aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.967+0000 7f199aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c108220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50308/0 (socket says 172.21.15.103:50308) 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.967+0000 7f199aa00700 1 -- 172.21.15.103:0/2360300734 learned_addr learned my addr 172.21.15.103:0/2360300734 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.968+0000 7f199aa00700 1 -- 172.21.15.103:0/2360300734 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c108f60 con 0x7f199c107e00 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.968+0000 7f199aa00700 1 --2- 172.21.15.103:0/2360300734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c108220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f198c0098a0 tx=0x7f198c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f739cc860baa01 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.968+0000 7f1999600700 1 -- 172.21.15.103:0/2360300734 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f198c004160 con 0x7f199c107e00 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.969+0000 7f1999600700 1 -- 172.21.15.103:0/2360300734 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198c004300 con 0x7f199c107e00 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.969+0000 7f1999600700 1 -- 172.21.15.103:0/2360300734 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f198c011b20 con 0x7f199c107e00 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.970+0000 7f19a1600700 1 -- 172.21.15.103:0/2360300734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 msgr2=0x7f199c108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.970+0000 7f19a1600700 1 --2- 172.21.15.103:0/2360300734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c108220 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f198c0098a0 tx=0x7f198c009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.970+0000 7f19a1600700 1 -- 172.21.15.103:0/2360300734 shutdown_connections 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.970+0000 7f19a1600700 1 --2- 172.21.15.103:0/2360300734 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c108220 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.970+0000 7f19a1600700 1 -- 172.21.15.103:0/2360300734 >> 172.21.15.103:0/2360300734 conn(0x7f199c1033c0 msgr2=0x7f199c1057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:34.080 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.970+0000 7f19a1600700 1 -- 172.21.15.103:0/2360300734 shutdown_connections 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.970+0000 7f19a1600700 1 -- 172.21.15.103:0/2360300734 wait complete. 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.971+0000 7f19a1600700 1 Processor -- start 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.971+0000 7f19a1600700 1 -- start start 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.971+0000 7f19a1600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.971+0000 7f19a1600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f199c10c3a0 con 0x7f199c107e00 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.972+0000 7f199aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.972+0000 7f199aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c19de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50318/0 (socket says 172.21.15.103:50318) 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.972+0000 7f199aa00700 1 -- 172.21.15.103:0/2699738253 learned_addr learned my addr 172.21.15.103:0/2699738253 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.972+0000 7f199aa00700 1 -- 172.21.15.103:0/2699738253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c19e3a0 con 0x7f199c107e00 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.973+0000 7f199aa00700 1 --2- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c19de60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f198c0117e0 tx=0x7f198c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.973+0000 7f1993400700 1 -- 172.21.15.103:0/2699738253 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f198c003ba0 con 0x7f199c107e00 2024-05-22T23:49:34.081 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.973+0000 7f1993400700 1 -- 172.21.15.103:0/2699738253 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f198c01a910 con 0x7f199c107e00 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.973+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f199c19e630 con 0x7f199c107e00 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.973+0000 7f1993400700 1 -- 172.21.15.103:0/2699738253 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f198c0248f0 con 0x7f199c107e00 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.973+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f199c19ea90 con 0x7f199c107e00 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.974+0000 7f1993400700 1 -- 172.21.15.103:0/2699738253 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f198c022020 con 0x7f199c107e00 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.974+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f197c005360 con 0x7f199c107e00 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.975+0000 7f1993400700 1 --2- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f19800391c0 0x7f198003b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.975+0000 7f1993400700 1 -- 172.21.15.103:0/2699738253 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f198c04ea40 con 0x7f199c107e00 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.975+0000 7f199a000700 1 --2- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f19800391c0 0x7f198003b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.976+0000 7f199a000700 1 --2- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f19800391c0 0x7f198003b680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1988006fd0 tx=0x7f1988006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:32.982+0000 7f1993400700 1 -- 172.21.15.103:0/2699738253 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f198c00b3e0 con 0x7f199c107e00 2024-05-22T23:49:34.082 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.220+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f197c000bf0 con 0x7f19800391c0 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.355+0000 7f1993400700 1 -- 172.21.15.103:0/2699738253 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f197c000bf0 con 0x7f19800391c0 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f19800391c0 msgr2=0x7f198003b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 --2- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f19800391c0 0x7f198003b680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1988006fd0 tx=0x7f1988006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 msgr2=0x7f199c19de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 --2- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c19de60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f198c0117e0 tx=0x7f198c011ba0 comp rx=0 tx=0).stop 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 shutdown_connections 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 --2- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f19800391c0 0x7f198003b680 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 --2- 172.21.15.103:0/2699738253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f199c107e00 0x7f199c19de60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 >> 172.21.15.103:0/2699738253 conn(0x7f199c1033c0 msgr2=0x7f199c07c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 shutdown_connections 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:33.358+0000 7f19a1600700 1 -- 172.21.15.103:0/2699738253 wait complete. 2024-05-22T23:49:34.083 INFO:teuthology.orchestra.run.smithi103.stdout:Creating initial admin user... 2024-05-22T23:49:34.676 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:34 smithi103 ceph-mon[31370]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:34.676 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:34.676 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:35.926 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$4n2wtgR0JWOl0gbZiOfbVenEQqvWEttYsVLYmHLODYQV85vShFgny", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716421775, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.746+0000 7fc73cc00700 1 Processor -- start 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.747+0000 7fc73cc00700 1 -- start start 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.747+0000 7fc73cc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc738109840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.747+0000 7fc73cc00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc738109e10 con 0x7fc738109420 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.747+0000 7fc735e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc738109840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.747+0000 7fc735e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc738109840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42524/0 (socket says 172.21.15.103:42524) 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.747+0000 7fc735e00700 1 -- 172.21.15.103:0/4047111662 learned_addr learned my addr 172.21.15.103:0/4047111662 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.748+0000 7fc735e00700 1 -- 172.21.15.103:0/4047111662 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc738109f90 con 0x7fc738109420 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.748+0000 7fc735e00700 1 --2- 172.21.15.103:0/4047111662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc738109840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc7280098a0 tx=0x7fc728009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4052cb7df611535a server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.749+0000 7fc734a00700 1 -- 172.21.15.103:0/4047111662 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc728004160 con 0x7fc738109420 2024-05-22T23:49:35.927 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.749+0000 7fc734a00700 1 -- 172.21.15.103:0/4047111662 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc728004300 con 0x7fc738109420 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.749+0000 7fc734a00700 1 -- 172.21.15.103:0/4047111662 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc728011b20 con 0x7fc738109420 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.749+0000 7fc73cc00700 1 -- 172.21.15.103:0/4047111662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 msgr2=0x7fc738109840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.749+0000 7fc73cc00700 1 --2- 172.21.15.103:0/4047111662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc738109840 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc7280098a0 tx=0x7fc728009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.750+0000 7fc73cc00700 1 -- 172.21.15.103:0/4047111662 shutdown_connections 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.750+0000 7fc73cc00700 1 --2- 172.21.15.103:0/4047111662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc738109840 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.750+0000 7fc73cc00700 1 -- 172.21.15.103:0/4047111662 >> 172.21.15.103:0/4047111662 conn(0x7fc738101a20 msgr2=0x7fc738103e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.750+0000 7fc73cc00700 1 -- 172.21.15.103:0/4047111662 shutdown_connections 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.750+0000 7fc73cc00700 1 -- 172.21.15.103:0/4047111662 wait complete. 2024-05-22T23:49:35.928 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.751+0000 7fc73cc00700 1 Processor -- start 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.751+0000 7fc73cc00700 1 -- start start 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.752+0000 7fc73cc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc73819de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.752+0000 7fc73cc00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc73810c3f0 con 0x7fc738109420 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.752+0000 7fc735e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc73819de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.752+0000 7fc735e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc73819de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42538/0 (socket says 172.21.15.103:42538) 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.752+0000 7fc735e00700 1 -- 172.21.15.103:0/2127526188 learned_addr learned my addr 172.21.15.103:0/2127526188 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.752+0000 7fc735e00700 1 -- 172.21.15.103:0/2127526188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc73819e3b0 con 0x7fc738109420 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.753+0000 7fc735e00700 1 --2- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc73819de70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc7280117e0 tx=0x7fc728011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.753+0000 7fc72ea00700 1 -- 172.21.15.103:0/2127526188 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc728003bf0 con 0x7fc738109420 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.753+0000 7fc72ea00700 1 -- 172.21.15.103:0/2127526188 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc72801a910 con 0x7fc738109420 2024-05-22T23:49:35.929 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.753+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73819e6a0 con 0x7fc738109420 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.753+0000 7fc72ea00700 1 -- 172.21.15.103:0/2127526188 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc728024970 con 0x7fc738109420 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.753+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7381a1330 con 0x7fc738109420 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.755+0000 7fc72ea00700 1 -- 172.21.15.103:0/2127526188 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fc728022020 con 0x7fc738109420 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.755+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7380639c0 con 0x7fc738109420 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.755+0000 7fc72ea00700 1 --2- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc71c039160 0x7fc71c03b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.755+0000 7fc72ea00700 1 -- 172.21.15.103:0/2127526188 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc72804d9a0 con 0x7fc738109420 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.755+0000 7fc735400700 1 --2- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc71c039160 0x7fc71c03b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.756+0000 7fc735400700 1 --2- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc71c039160 0x7fc71c03b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc724006fd0 tx=0x7fc724006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:34.762+0000 7fc72ea00700 1 -- 172.21.15.103:0/2127526188 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc7280212c0 con 0x7fc738109420 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.004+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7fc7381a15c0 con 0x7fc71c039160 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.252+0000 7fc72ea00700 1 -- 172.21.15.103:0/2127526188 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7fc7381a15c0 con 0x7fc71c039160 2024-05-22T23:49:35.930 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.256+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc71c039160 msgr2=0x7fc71c03b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.256+0000 7fc73cc00700 1 --2- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc71c039160 0x7fc71c03b620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc724006fd0 tx=0x7fc724006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.256+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 msgr2=0x7fc73819de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.256+0000 7fc73cc00700 1 --2- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc73819de70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc7280117e0 tx=0x7fc728011ba0 comp rx=0 tx=0).stop 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.256+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 shutdown_connections 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.256+0000 7fc73cc00700 1 --2- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc71c039160 0x7fc71c03b620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.256+0000 7fc73cc00700 1 --2- 172.21.15.103:0/2127526188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc738109420 0x7fc73819de70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.256+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 >> 172.21.15.103:0/2127526188 conn(0x7fc738101a20 msgr2=0x7fc738106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.257+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 shutdown_connections 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:35.257+0000 7fc73cc00700 1 -- 172.21.15.103:0/2127526188 wait complete. 2024-05-22T23:49:35.931 INFO:teuthology.orchestra.run.smithi103.stdout:Fetching dashboard port number... 2024-05-22T23:49:36.243 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:36.244 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:35 smithi103 ceph-mon[31370]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:36.244 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:37.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:36 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2139162688' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-22T23:49:38.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:37 smithi103 ceph-mon[31370]: mgrmap e13: x(active, since 6s) 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stdout 8443 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.572+0000 7f58f7e00700 1 Processor -- start 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.573+0000 7f58f7e00700 1 -- start start 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.573+0000 7f58f7e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f810a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.573+0000 7f58f7e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58f810aab0 con 0x7f58f810a150 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.574+0000 7f58f6a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f810a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.574+0000 7f58f6a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f810a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42540/0 (socket says 172.21.15.103:42540) 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.574+0000 7f58f6a00700 1 -- 172.21.15.103:0/2833986750 learned_addr learned my addr 172.21.15.103:0/2833986750 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.574+0000 7f58f6a00700 1 -- 172.21.15.103:0/2833986750 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f810af70 con 0x7f58f810a150 2024-05-22T23:49:38.440 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.574+0000 7f58f6a00700 1 --2- 172.21.15.103:0/2833986750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f810a570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58e80098a0 tx=0x7f58e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbc3e4b824d003a server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.575+0000 7f58f5600700 1 -- 172.21.15.103:0/2833986750 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58e8004160 con 0x7f58f810a150 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.575+0000 7f58f5600700 1 -- 172.21.15.103:0/2833986750 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58e8004300 con 0x7f58f810a150 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.575+0000 7f58f5600700 1 -- 172.21.15.103:0/2833986750 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58e8011b20 con 0x7f58f810a150 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.575+0000 7f58f7e00700 1 -- 172.21.15.103:0/2833986750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 msgr2=0x7f58f810a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.575+0000 7f58f7e00700 1 --2- 172.21.15.103:0/2833986750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f810a570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58e80098a0 tx=0x7f58e8009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.576+0000 7f58f7e00700 1 -- 172.21.15.103:0/2833986750 shutdown_connections 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.576+0000 7f58f7e00700 1 --2- 172.21.15.103:0/2833986750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f810a570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.576+0000 7f58f7e00700 1 -- 172.21.15.103:0/2833986750 >> 172.21.15.103:0/2833986750 conn(0x7f58f807d230 msgr2=0x7f58f807d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:38.441 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.576+0000 7f58f7e00700 1 -- 172.21.15.103:0/2833986750 shutdown_connections 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.576+0000 7f58f7e00700 1 -- 172.21.15.103:0/2833986750 wait complete. 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.577+0000 7f58f7e00700 1 Processor -- start 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.577+0000 7f58f7e00700 1 -- start start 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.577+0000 7f58f7e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.577+0000 7f58f7e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58f810e230 con 0x7f58f810a150 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.578+0000 7f58f6a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.578+0000 7f58f6a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42548/0 (socket says 172.21.15.103:42548) 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.578+0000 7f58f6a00700 1 -- 172.21.15.103:0/2139162688 learned_addr learned my addr 172.21.15.103:0/2139162688 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.578+0000 7f58f6a00700 1 -- 172.21.15.103:0/2139162688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f819e250 con 0x7f58f810a150 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.578+0000 7f58f6a00700 1 --2- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f819dd10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58e8004660 tx=0x7f58e8011ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:38.442 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.579+0000 7f58ef400700 1 -- 172.21.15.103:0/2139162688 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58e8003ca0 con 0x7f58f810a150 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.579+0000 7f58ef400700 1 -- 172.21.15.103:0/2139162688 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58e801a910 con 0x7f58f810a150 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.579+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f819e4e0 con 0x7f58f810a150 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.579+0000 7f58ef400700 1 -- 172.21.15.103:0/2139162688 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f58e8024970 con 0x7f58f810a150 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.579+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f819e940 con 0x7f58f810a150 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.580+0000 7f58ef400700 1 -- 172.21.15.103:0/2139162688 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f58e8022020 con 0x7f58f810a150 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.580+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f8069490 con 0x7f58f810a150 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.580+0000 7f58ef400700 1 --2- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58dc039300 0x7f58dc03b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.580+0000 7f58ef400700 1 -- 172.21.15.103:0/2139162688 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f58e804d580 con 0x7f58f810a150 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.581+0000 7f58f6000700 1 --2- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58dc039300 0x7f58dc03b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.581+0000 7f58f6000700 1 --2- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58dc039300 0x7f58dc03b7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f58e4006fd0 tx=0x7f58e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:38.443 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.585+0000 7f58ef400700 1 -- 172.21.15.103:0/2139162688 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f58e800b3e0 con 0x7f58f810a150 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.820+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f58f81a13a0 con 0x7f58f810a150 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.821+0000 7f58ef400700 1 -- 172.21.15.103:0/2139162688 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f58e802ddb0 con 0x7f58f810a150 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.823+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58dc039300 msgr2=0x7f58dc03b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.823+0000 7f58f7e00700 1 --2- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58dc039300 0x7f58dc03b7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f58e4006fd0 tx=0x7f58e4006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.823+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 msgr2=0x7f58f819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.823+0000 7f58f7e00700 1 --2- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f819dd10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58e8004660 tx=0x7f58e8011ce0 comp rx=0 tx=0).stop 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.824+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 shutdown_connections 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.824+0000 7f58f7e00700 1 --2- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58dc039300 0x7f58dc03b7c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.824+0000 7f58f7e00700 1 --2- 172.21.15.103:0/2139162688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58f810a150 0x7f58f819dd10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.824+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 >> 172.21.15.103:0/2139162688 conn(0x7f58f807d230 msgr2=0x7f58f8107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.824+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 shutdown_connections 2024-05-22T23:49:38.444 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:36.824+0000 7f58f7e00700 1 -- 172.21.15.103:0/2139162688 wait complete. 2024-05-22T23:49:38.466 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-22T23:49:38.466 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout disabled 2024-05-22T23:49:38.488 INFO:teuthology.orchestra.run.smithi103.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-22T23:49:38.488 INFO:teuthology.orchestra.run.smithi103.stdout:systemctl: stdout inactive 2024-05-22T23:49:38.488 INFO:teuthology.orchestra.run.smithi103.stdout:firewalld.service is not enabled 2024-05-22T23:49:38.488 INFO:teuthology.orchestra.run.smithi103.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-22T23:49:38.496 INFO:teuthology.orchestra.run.smithi103.stdout:Ceph Dashboard is now available at: 2024-05-22T23:49:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: URL: https://smithi103.front.sepia.ceph.com:8443/ 2024-05-22T23:49:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: User: admin 2024-05-22T23:49:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: Password: j56skh9mli 2024-05-22T23:49:38.496 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:38.497 INFO:teuthology.orchestra.run.smithi103.stdout:Saving cluster configuration to /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config directory 2024-05-22T23:49:38.499 INFO:teuthology.orchestra.run.smithi103.stdout:Enabling autotune for osd_memory_target 2024-05-22T23:49:40.004 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.107+0000 7fc97a200700 1 Processor -- start 2024-05-22T23:49:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.107+0000 7fc97a200700 1 -- start start 2024-05-22T23:49:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.108+0000 7fc97a200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc974108270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.108+0000 7fc97a200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9741087b0 con 0x7fc974107e50 2024-05-22T23:49:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.108+0000 7fc973400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc974108270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.109+0000 7fc973400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc974108270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42564/0 (socket says 172.21.15.103:42564) 2024-05-22T23:49:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.109+0000 7fc973400700 1 -- 172.21.15.103:0/2038872965 learned_addr learned my addr 172.21.15.103:0/2038872965 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.109+0000 7fc973400700 1 -- 172.21.15.103:0/2038872965 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc974108fb0 con 0x7fc974107e50 2024-05-22T23:49:40.005 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.109+0000 7fc973400700 1 --2- 172.21.15.103:0/2038872965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc974108270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc9640098a0 tx=0x7fc964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=332b656b7f720b5c server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.110+0000 7fc972000700 1 -- 172.21.15.103:0/2038872965 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc964004160 con 0x7fc974107e50 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.110+0000 7fc972000700 1 -- 172.21.15.103:0/2038872965 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc964004300 con 0x7fc974107e50 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.110+0000 7fc972000700 1 -- 172.21.15.103:0/2038872965 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc964011b20 con 0x7fc974107e50 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.110+0000 7fc97a200700 1 -- 172.21.15.103:0/2038872965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 msgr2=0x7fc974108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.110+0000 7fc97a200700 1 --2- 172.21.15.103:0/2038872965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc974108270 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc9640098a0 tx=0x7fc964009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.110+0000 7fc97a200700 1 -- 172.21.15.103:0/2038872965 shutdown_connections 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.110+0000 7fc97a200700 1 --2- 172.21.15.103:0/2038872965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc974108270 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.110+0000 7fc97a200700 1 -- 172.21.15.103:0/2038872965 >> 172.21.15.103:0/2038872965 conn(0x7fc9741033f0 msgr2=0x7fc974105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:40.011 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.111+0000 7fc97a200700 1 -- 172.21.15.103:0/2038872965 shutdown_connections 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.111+0000 7fc97a200700 1 -- 172.21.15.103:0/2038872965 wait complete. 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.112+0000 7fc97a200700 1 Processor -- start 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.112+0000 7fc97a200700 1 -- start start 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.112+0000 7fc97a200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc97419bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.112+0000 7fc97a200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc97410c3f0 con 0x7fc974107e50 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.112+0000 7fc973400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc97419bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.113+0000 7fc973400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc97419bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42576/0 (socket says 172.21.15.103:42576) 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.113+0000 7fc973400700 1 -- 172.21.15.103:0/2410243443 learned_addr learned my addr 172.21.15.103:0/2410243443 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.113+0000 7fc973400700 1 -- 172.21.15.103:0/2410243443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97419c130 con 0x7fc974107e50 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.113+0000 7fc973400700 1 --2- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc97419bbf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc9640117e0 tx=0x7fc964011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.113+0000 7fc96be00700 1 -- 172.21.15.103:0/2410243443 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc964003bf0 con 0x7fc974107e50 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.113+0000 7fc96be00700 1 -- 172.21.15.103:0/2410243443 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc96401a910 con 0x7fc974107e50 2024-05-22T23:49:40.012 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.113+0000 7fc96be00700 1 -- 172.21.15.103:0/2410243443 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc964024970 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.113+0000 7fc97a200700 1 -- 172.21.15.103:0/2410243443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97419c3c0 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.114+0000 7fc97a200700 1 -- 172.21.15.103:0/2410243443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97419c820 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.115+0000 7fc96be00700 1 -- 172.21.15.103:0/2410243443 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fc964022020 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.115+0000 7fc969600700 1 -- 172.21.15.103:0/2410243443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9740639c0 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.115+0000 7fc96be00700 1 --2- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc958039210 0x7fc95803b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.115+0000 7fc96be00700 1 -- 172.21.15.103:0/2410243443 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc96404d620 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.116+0000 7fc972a00700 1 --2- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc958039210 0x7fc95803b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.116+0000 7fc972a00700 1 --2- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc958039210 0x7fc95803b6d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc960006fd0 tx=0x7fc960006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.121+0000 7fc96be00700 1 -- 172.21.15.103:0/2410243443 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc964011e20 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.341+0000 7fc969600700 1 -- 172.21.15.103:0/2410243443 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fc974063320 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.342+0000 7fc96be00700 1 -- 172.21.15.103:0/2410243443 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fc96400b3e0 con 0x7fc974107e50 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.345+0000 7fc969600700 1 -- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc958039210 msgr2=0x7fc95803b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:40.013 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.345+0000 7fc969600700 1 --2- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc958039210 0x7fc95803b6d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc960006fd0 tx=0x7fc960006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:40.014 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.345+0000 7fc969600700 1 -- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 msgr2=0x7fc97419bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:40.014 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.345+0000 7fc969600700 1 --2- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc97419bbf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc9640117e0 tx=0x7fc964011c10 comp rx=0 tx=0).stop 2024-05-22T23:49:40.014 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.345+0000 7fc969600700 1 -- 172.21.15.103:0/2410243443 shutdown_connections 2024-05-22T23:49:40.014 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.345+0000 7fc969600700 1 --2- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc958039210 0x7fc95803b6d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:40.014 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.345+0000 7fc969600700 1 --2- 172.21.15.103:0/2410243443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc974107e50 0x7fc97419bbf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:40.014 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.345+0000 7fc969600700 1 -- 172.21.15.103:0/2410243443 >> 172.21.15.103:0/2410243443 conn(0x7fc9741033f0 msgr2=0x7fc974103d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:40.015 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.346+0000 7fc969600700 1 -- 172.21.15.103:0/2410243443 shutdown_connections 2024-05-22T23:49:40.015 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:39.346+0000 7fc969600700 1 -- 172.21.15.103:0/2410243443 wait complete. 2024-05-22T23:49:41.702 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.612+0000 7f2072400700 1 Processor -- start 2024-05-22T23:49:41.702 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.612+0000 7f2072400700 1 -- start start 2024-05-22T23:49:41.702 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.613+0000 7f2072400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:41.703 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.613+0000 7f2072400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f206c07c8f0 con 0x7f206c07dec0 2024-05-22T23:49:41.703 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.613+0000 7f206b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:41.703 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.613+0000 7f206b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42582/0 (socket says 172.21.15.103:42582) 2024-05-22T23:49:41.703 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.613+0000 7f206b400700 1 -- 172.21.15.103:0/789937555 learned_addr learned my addr 172.21.15.103:0/789937555 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:41.703 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.614+0000 7f206b400700 1 -- 172.21.15.103:0/789937555 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c07ca70 con 0x7f206c07dec0 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.614+0000 7f206b400700 1 --2- 172.21.15.103:0/789937555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c07c320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f205c0098a0 tx=0x7f205c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3c6d0c8947488e server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.614+0000 7f206a000700 1 -- 172.21.15.103:0/789937555 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f205c004160 con 0x7f206c07dec0 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.614+0000 7f206a000700 1 -- 172.21.15.103:0/789937555 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f205c004300 con 0x7f206c07dec0 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.614+0000 7f206a000700 1 -- 172.21.15.103:0/789937555 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f205c011b20 con 0x7f206c07dec0 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.615+0000 7f2072400700 1 -- 172.21.15.103:0/789937555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 msgr2=0x7f206c07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.615+0000 7f2072400700 1 --2- 172.21.15.103:0/789937555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c07c320 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f205c0098a0 tx=0x7f205c009bb0 comp rx=0 tx=0).stop 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.615+0000 7f2072400700 1 -- 172.21.15.103:0/789937555 shutdown_connections 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.615+0000 7f2072400700 1 --2- 172.21.15.103:0/789937555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c07c320 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:41.709 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.615+0000 7f2072400700 1 -- 172.21.15.103:0/789937555 >> 172.21.15.103:0/789937555 conn(0x7f206c1033c0 msgr2=0x7f206c1057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:41.710 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.615+0000 7f2072400700 1 -- 172.21.15.103:0/789937555 shutdown_connections 2024-05-22T23:49:41.710 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.615+0000 7f2072400700 1 -- 172.21.15.103:0/789937555 wait complete. 2024-05-22T23:49:41.710 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.616+0000 7f2072400700 1 Processor -- start 2024-05-22T23:49:41.710 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.616+0000 7f2072400700 1 -- start start 2024-05-22T23:49:41.710 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.616+0000 7f2072400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:41.710 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.617+0000 7f2072400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f206c10e440 con 0x7f206c07dec0 2024-05-22T23:49:41.710 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.617+0000 7f206b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:41.710 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.617+0000 7f206b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c19deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:42584/0 (socket says 172.21.15.103:42584) 2024-05-22T23:49:41.711 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.617+0000 7f206b400700 1 -- 172.21.15.103:0/533695083 learned_addr learned my addr 172.21.15.103:0/533695083 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:41.711 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.617+0000 7f206b400700 1 -- 172.21.15.103:0/533695083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c19e3f0 con 0x7f206c07dec0 2024-05-22T23:49:41.711 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.617+0000 7f206b400700 1 --2- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c19deb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f205c0117e0 tx=0x7f205c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:41.711 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.618+0000 7f2063e00700 1 -- 172.21.15.103:0/533695083 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f205c003bf0 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.618+0000 7f2063e00700 1 -- 172.21.15.103:0/533695083 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f205c01a910 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.618+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c19e680 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.618+0000 7f2063e00700 1 -- 172.21.15.103:0/533695083 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f205c024970 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.618+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c19eae0 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.619+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f206c0639c0 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.619+0000 7f2063e00700 1 -- 172.21.15.103:0/533695083 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f205c022020 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.620+0000 7f2063e00700 1 --2- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2050039260 0x7f205003b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.620+0000 7f2063e00700 1 -- 172.21.15.103:0/533695083 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f205c04d640 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.620+0000 7f206aa00700 1 --2- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2050039260 0x7f205003b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.621+0000 7f206aa00700 1 --2- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2050039260 0x7f205003b720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2058006fd0 tx=0x7f2058006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.625+0000 7f2063e00700 1 -- 172.21.15.103:0/533695083 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f205c02d910 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.966+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f206c1a14b0 con 0x7f206c07dec0 2024-05-22T23:49:41.712 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.969+0000 7f2063e00700 1 -- 172.21.15.103:0/533695083 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f205c02d3f0 con 0x7f206c07dec0 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.971+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2050039260 msgr2=0x7f205003b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.971+0000 7f2072400700 1 --2- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2050039260 0x7f205003b720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2058006fd0 tx=0x7f2058006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.972+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 msgr2=0x7f206c19deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.972+0000 7f2072400700 1 --2- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c19deb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f205c0117e0 tx=0x7f205c011ba0 comp rx=0 tx=0).stop 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.972+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 shutdown_connections 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.972+0000 7f2072400700 1 --2- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2050039260 0x7f205003b720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.972+0000 7f2072400700 1 --2- 172.21.15.103:0/533695083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f206c07dec0 0x7f206c19deb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.972+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 >> 172.21.15.103:0/533695083 conn(0x7f206c1033c0 msgr2=0x7f206c103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.972+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 shutdown_connections 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr 2024-05-22T23:49:40.972+0000 7f2072400700 1 -- 172.21.15.103:0/533695083 wait complete. 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:41.713 INFO:teuthology.orchestra.run.smithi103.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout:Or, if you are only running a single cluster on this host: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: ceph telemetry on 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout:For more information see: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-22T23:49:41.714 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:49:41.715 INFO:teuthology.orchestra.run.smithi103.stdout:Bootstrap complete. 2024-05-22T23:49:41.793 INFO:tasks.cephadm:Fetching config... 2024-05-22T23:49:41.793 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:49:41.793 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-22T23:49:41.842 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-22T23:49:41.842 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:49:41.842 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-22T23:49:41.925 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-22T23:49:41.926 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:49:41.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-22T23:49:42.034 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-22T23:49:42.035 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:49:42.035 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-22T23:49:42.130 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-22T23:49:42.130 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOFZlq/l2JR3f0+XdUUrv9nXPFJhPYpgK1ypdGY62JcbcGQZvY6mzAJ6G0gnPlgU/hb6XFDOJ7Yzy3gsxwR7N599VnU72BdhOOwovzrIeNKiesq36v+gOxZpTpN/ArQarB7xJR1SlI3hUXunGwgZb7vHpAGxyHlYGvaERqeIzMDuC3gIy+kGLDZkfA8VVWAbqcfIk3s02S8G4IdeRbkKfMyEZLjvaAMA2mg2WRDDRZ535q+jdGhDaRxpwA/j2DBWcB60gir01JR2ubGpAVfNz+vKUi3uFQfY+NUezeLy1Ao+3SyXSEug4HPHsuWd31tjQPpQScsfFTVdP+K34YxAIDJ9Op2qTYkpf7qNNaEIqrFhLiu9DifYpK3JR+vSEqvRcmznapJHtZRYwSpsVVeAZ/PAef8TZxZfIOXfdYkxJAtHviKIhi2TIll2jmRKRW1ZFM5oWjkENEFrpSKdACSEyZNjbb1cYFIISwF5wqsH/82+lr6LRMsq9TwXFw0mUTpk= ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-22T23:49:42.225 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:41 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/533695083' entity='client.admin' 2024-05-22T23:49:42.240 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOFZlq/l2JR3f0+XdUUrv9nXPFJhPYpgK1ypdGY62JcbcGQZvY6mzAJ6G0gnPlgU/hb6XFDOJ7Yzy3gsxwR7N599VnU72BdhOOwovzrIeNKiesq36v+gOxZpTpN/ArQarB7xJR1SlI3hUXunGwgZb7vHpAGxyHlYGvaERqeIzMDuC3gIy+kGLDZkfA8VVWAbqcfIk3s02S8G4IdeRbkKfMyEZLjvaAMA2mg2WRDDRZ535q+jdGhDaRxpwA/j2DBWcB60gir01JR2ubGpAVfNz+vKUi3uFQfY+NUezeLy1Ao+3SyXSEug4HPHsuWd31tjQPpQScsfFTVdP+K34YxAIDJ9Op2qTYkpf7qNNaEIqrFhLiu9DifYpK3JR+vSEqvRcmznapJHtZRYwSpsVVeAZ/PAef8TZxZfIOXfdYkxJAtHviKIhi2TIll2jmRKRW1ZFM5oWjkENEFrpSKdACSEyZNjbb1cYFIISwF5wqsH/82+lr6LRMsq9TwXFw0mUTpk= ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:49:42.270 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOFZlq/l2JR3f0+XdUUrv9nXPFJhPYpgK1ypdGY62JcbcGQZvY6mzAJ6G0gnPlgU/hb6XFDOJ7Yzy3gsxwR7N599VnU72BdhOOwovzrIeNKiesq36v+gOxZpTpN/ArQarB7xJR1SlI3hUXunGwgZb7vHpAGxyHlYGvaERqeIzMDuC3gIy+kGLDZkfA8VVWAbqcfIk3s02S8G4IdeRbkKfMyEZLjvaAMA2mg2WRDDRZ535q+jdGhDaRxpwA/j2DBWcB60gir01JR2ubGpAVfNz+vKUi3uFQfY+NUezeLy1Ao+3SyXSEug4HPHsuWd31tjQPpQScsfFTVdP+K34YxAIDJ9Op2qTYkpf7qNNaEIqrFhLiu9DifYpK3JR+vSEqvRcmznapJHtZRYwSpsVVeAZ/PAef8TZxZfIOXfdYkxJAtHviKIhi2TIll2jmRKRW1ZFM5oWjkENEFrpSKdACSEyZNjbb1cYFIISwF5wqsH/82+lr6LRMsq9TwXFw0mUTpk= ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-22T23:49:42.312 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOFZlq/l2JR3f0+XdUUrv9nXPFJhPYpgK1ypdGY62JcbcGQZvY6mzAJ6G0gnPlgU/hb6XFDOJ7Yzy3gsxwR7N599VnU72BdhOOwovzrIeNKiesq36v+gOxZpTpN/ArQarB7xJR1SlI3hUXunGwgZb7vHpAGxyHlYGvaERqeIzMDuC3gIy+kGLDZkfA8VVWAbqcfIk3s02S8G4IdeRbkKfMyEZLjvaAMA2mg2WRDDRZ535q+jdGhDaRxpwA/j2DBWcB60gir01JR2ubGpAVfNz+vKUi3uFQfY+NUezeLy1Ao+3SyXSEug4HPHsuWd31tjQPpQScsfFTVdP+K34YxAIDJ9Op2qTYkpf7qNNaEIqrFhLiu9DifYpK3JR+vSEqvRcmznapJHtZRYwSpsVVeAZ/PAef8TZxZfIOXfdYkxJAtHviKIhi2TIll2jmRKRW1ZFM5oWjkENEFrpSKdACSEyZNjbb1cYFIISwF5wqsH/82+lr6LRMsq9TwXFw0mUTpk= ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:49:42.324 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOFZlq/l2JR3f0+XdUUrv9nXPFJhPYpgK1ypdGY62JcbcGQZvY6mzAJ6G0gnPlgU/hb6XFDOJ7Yzy3gsxwR7N599VnU72BdhOOwovzrIeNKiesq36v+gOxZpTpN/ArQarB7xJR1SlI3hUXunGwgZb7vHpAGxyHlYGvaERqeIzMDuC3gIy+kGLDZkfA8VVWAbqcfIk3s02S8G4IdeRbkKfMyEZLjvaAMA2mg2WRDDRZ535q+jdGhDaRxpwA/j2DBWcB60gir01JR2ubGpAVfNz+vKUi3uFQfY+NUezeLy1Ao+3SyXSEug4HPHsuWd31tjQPpQScsfFTVdP+K34YxAIDJ9Op2qTYkpf7qNNaEIqrFhLiu9DifYpK3JR+vSEqvRcmznapJHtZRYwSpsVVeAZ/PAef8TZxZfIOXfdYkxJAtHviKIhi2TIll2jmRKRW1ZFM5oWjkENEFrpSKdACSEyZNjbb1cYFIISwF5wqsH/82+lr6LRMsq9TwXFw0mUTpk= ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-22T23:49:42.366 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNOFZlq/l2JR3f0+XdUUrv9nXPFJhPYpgK1ypdGY62JcbcGQZvY6mzAJ6G0gnPlgU/hb6XFDOJ7Yzy3gsxwR7N599VnU72BdhOOwovzrIeNKiesq36v+gOxZpTpN/ArQarB7xJR1SlI3hUXunGwgZb7vHpAGxyHlYGvaERqeIzMDuC3gIy+kGLDZkfA8VVWAbqcfIk3s02S8G4IdeRbkKfMyEZLjvaAMA2mg2WRDDRZ535q+jdGhDaRxpwA/j2DBWcB60gir01JR2ubGpAVfNz+vKUi3uFQfY+NUezeLy1Ao+3SyXSEug4HPHsuWd31tjQPpQScsfFTVdP+K34YxAIDJ9Op2qTYkpf7qNNaEIqrFhLiu9DifYpK3JR+vSEqvRcmznapJHtZRYwSpsVVeAZ/PAef8TZxZfIOXfdYkxJAtHviKIhi2TIll2jmRKRW1ZFM5oWjkENEFrpSKdACSEyZNjbb1cYFIISwF5wqsH/82+lr6LRMsq9TwXFw0mUTpk= ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:49:42.378 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-22T23:49:42.804 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:49:45.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.744+0000 7f9e2be00700 1 -- 172.21.15.103:0/2854691073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 msgr2=0x7f9e2c104240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:45.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.744+0000 7f9e2be00700 1 --2- 172.21.15.103:0/2854691073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 0x7f9e2c104240 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c009910 tx=0x7f9e1c009c20 comp rx=0 tx=0).stop 2024-05-22T23:49:45.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.745+0000 7f9e2be00700 1 -- 172.21.15.103:0/2854691073 shutdown_connections 2024-05-22T23:49:45.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.745+0000 7f9e2be00700 1 --2- 172.21.15.103:0/2854691073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 0x7f9e2c104240 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:45.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.745+0000 7f9e2be00700 1 -- 172.21.15.103:0/2854691073 >> 172.21.15.103:0/2854691073 conn(0x7f9e2c0ff3c0 msgr2=0x7f9e2c101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:45.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.745+0000 7f9e2be00700 1 -- 172.21.15.103:0/2854691073 shutdown_connections 2024-05-22T23:49:45.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.745+0000 7f9e2be00700 1 -- 172.21.15.103:0/2854691073 wait complete. 2024-05-22T23:49:45.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.746+0000 7f9e2be00700 1 Processor -- start 2024-05-22T23:49:45.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.746+0000 7f9e2be00700 1 -- start start 2024-05-22T23:49:45.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.746+0000 7f9e2be00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 0x7f9e2c199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:45.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.746+0000 7f9e2be00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2c109e30 con 0x7f9e2c103e20 2024-05-22T23:49:45.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.747+0000 7f9e2aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 0x7f9e2c199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:45.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.747+0000 7f9e2aa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 0x7f9e2c199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36298/0 (socket says 172.21.15.103:36298) 2024-05-22T23:49:45.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.747+0000 7f9e2aa00700 1 -- 172.21.15.103:0/1486515815 learned_addr learned my addr 172.21.15.103:0/1486515815 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:45.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.747+0000 7f9e2aa00700 1 -- 172.21.15.103:0/1486515815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2c199a40 con 0x7f9e2c103e20 2024-05-22T23:49:45.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.747+0000 7f9e2aa00700 1 --2- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 0x7f9e2c199500 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c005950 tx=0x7f9e1c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:45.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.748+0000 7f9e23400700 1 -- 172.21.15.103:0/1486515815 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e1c026070 con 0x7f9e2c103e20 2024-05-22T23:49:45.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.748+0000 7f9e23400700 1 -- 172.21.15.103:0/1486515815 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e1c00bc30 con 0x7f9e2c103e20 2024-05-22T23:49:45.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.748+0000 7f9e23400700 1 -- 172.21.15.103:0/1486515815 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e1c003b30 con 0x7f9e2c103e20 2024-05-22T23:49:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.748+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e2c199cd0 con 0x7f9e2c103e20 2024-05-22T23:49:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.748+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e2c19c9c0 con 0x7f9e2c103e20 2024-05-22T23:49:45.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.749+0000 7f9e23400700 1 -- 172.21.15.103:0/1486515815 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f9e1c029070 con 0x7f9e2c103e20 2024-05-22T23:49:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.749+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e2c10a5b0 con 0x7f9e2c103e20 2024-05-22T23:49:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.749+0000 7f9e23400700 1 --2- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9e100390f0 0x7f9e1003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.749+0000 7f9e23400700 1 -- 172.21.15.103:0/1486515815 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9e1c01e070 con 0x7f9e2c103e20 2024-05-22T23:49:45.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.750+0000 7f9e2a000700 1 --2- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9e100390f0 0x7f9e1003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:45.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.750+0000 7f9e2a000700 1 --2- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9e100390f0 0x7f9e1003b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e18006fd0 tx=0x7f9e18006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:45.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.755+0000 7f9e23400700 1 -- 172.21.15.103:0/1486515815 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9e1c013070 con 0x7f9e2c103e20 2024-05-22T23:49:45.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.984+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f9e2c02dcd0 con 0x7f9e2c103e20 2024-05-22T23:49:45.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.988+0000 7f9e23400700 1 -- 172.21.15.103:0/1486515815 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f9e1c030020 con 0x7f9e2c103e20 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.991+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9e100390f0 msgr2=0x7f9e1003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.991+0000 7f9e2be00700 1 --2- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9e100390f0 0x7f9e1003b5b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9e18006fd0 tx=0x7f9e18006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.991+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 msgr2=0x7f9e2c199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.991+0000 7f9e2be00700 1 --2- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 0x7f9e2c199500 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c005950 tx=0x7f9e1c005030 comp rx=0 tx=0).stop 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.991+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 shutdown_connections 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.991+0000 7f9e2be00700 1 --2- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9e100390f0 0x7f9e1003b5b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.991+0000 7f9e2be00700 1 --2- 172.21.15.103:0/1486515815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e2c103e20 0x7f9e2c199500 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.991+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 >> 172.21.15.103:0/1486515815 conn(0x7f9e2c0ff3c0 msgr2=0x7f9e2c0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:45.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.992+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 shutdown_connections 2024-05-22T23:49:45.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:45.992+0000 7f9e2be00700 1 -- 172.21.15.103:0/1486515815 wait complete. 2024-05-22T23:49:46.943 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-22T23:49:46.943 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-22T23:49:47.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:46 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1486515815' entity='client.admin' 2024-05-22T23:49:47.356 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:49:49.647 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:49.648 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:49.648 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:49:49.648 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:49.648 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:49:49.648 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:50.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.151+0000 7f10b3200700 1 -- 172.21.15.103:0/1100705103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 msgr2=0x7f10b4108990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:50.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.151+0000 7f10b3200700 1 --2- 172.21.15.103:0/1100705103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 0x7f10b4108990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f10a4009910 tx=0x7f10a4009c20 comp rx=0 tx=0).stop 2024-05-22T23:49:50.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.151+0000 7f10b3200700 1 -- 172.21.15.103:0/1100705103 shutdown_connections 2024-05-22T23:49:50.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.151+0000 7f10b3200700 1 --2- 172.21.15.103:0/1100705103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 0x7f10b4108990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:50.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.151+0000 7f10b3200700 1 -- 172.21.15.103:0/1100705103 >> 172.21.15.103:0/1100705103 conn(0x7f10b4077320 msgr2=0x7f10b4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:50.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.152+0000 7f10b3200700 1 -- 172.21.15.103:0/1100705103 shutdown_connections 2024-05-22T23:49:50.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.152+0000 7f10b3200700 1 -- 172.21.15.103:0/1100705103 wait complete. 2024-05-22T23:49:50.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.153+0000 7f10b3200700 1 Processor -- start 2024-05-22T23:49:50.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.153+0000 7f10b3200700 1 -- start start 2024-05-22T23:49:50.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.153+0000 7f10b3200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 0x7f10b419d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:50.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.153+0000 7f10b3200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f10b410e290 con 0x7f10b4108570 2024-05-22T23:49:50.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.153+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 0x7f10b419d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:50.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.153+0000 7f10b1e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 0x7f10b419d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36318/0 (socket says 172.21.15.103:36318) 2024-05-22T23:49:50.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.153+0000 7f10b1e00700 1 -- 172.21.15.103:0/3612869995 learned_addr learned my addr 172.21.15.103:0/3612869995 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:50.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.154+0000 7f10b1e00700 1 -- 172.21.15.103:0/3612869995 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b419de70 con 0x7f10b4108570 2024-05-22T23:49:50.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.154+0000 7f10b1e00700 1 --2- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 0x7f10b419d930 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f10a4005850 tx=0x7f10a4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:50.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.154+0000 7f10aaa00700 1 -- 172.21.15.103:0/3612869995 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10a4026070 con 0x7f10b4108570 2024-05-22T23:49:50.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.154+0000 7f10aaa00700 1 -- 172.21.15.103:0/3612869995 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10a40044a0 con 0x7f10b4108570 2024-05-22T23:49:50.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.154+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b419e100 con 0x7f10b4108570 2024-05-22T23:49:50.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.155+0000 7f10aaa00700 1 -- 172.21.15.103:0/3612869995 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10a4003ab0 con 0x7f10b4108570 2024-05-22T23:49:50.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.155+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b41a0df0 con 0x7f10b4108570 2024-05-22T23:49:50.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.156+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b410ea10 con 0x7f10b4108570 2024-05-22T23:49:50.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.156+0000 7f10aaa00700 1 -- 172.21.15.103:0/3612869995 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f10a4029070 con 0x7f10b4108570 2024-05-22T23:49:50.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.157+0000 7f10aaa00700 1 --2- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1098039150 0x7f109803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:50.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.157+0000 7f10aaa00700 1 -- 172.21.15.103:0/3612869995 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f10a401e070 con 0x7f10b4108570 2024-05-22T23:49:50.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.157+0000 7f10b1400700 1 --2- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1098039150 0x7f109803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:50.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.158+0000 7f10b1400700 1 --2- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1098039150 0x7f109803b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f10a0006fd0 tx=0x7f10a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:50.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.162+0000 7f10aaa00700 1 -- 172.21.15.103:0/3612869995 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f10a4013070 con 0x7f10b4108570 2024-05-22T23:49:50.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.393+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f10b402dcd0 con 0x7f1098039150 2024-05-22T23:49:50.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.400+0000 7f10aaa00700 1 -- 172.21.15.103:0/3612869995 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f10b402dcd0 con 0x7f1098039150 2024-05-22T23:49:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.403+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1098039150 msgr2=0x7f109803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.403+0000 7f10b3200700 1 --2- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1098039150 0x7f109803b610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f10a0006fd0 tx=0x7f10a0006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.403+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 msgr2=0x7f10b419d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.403+0000 7f10b3200700 1 --2- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 0x7f10b419d930 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f10a4005850 tx=0x7f10a4005030 comp rx=0 tx=0).stop 2024-05-22T23:49:50.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.404+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 shutdown_connections 2024-05-22T23:49:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.404+0000 7f10b3200700 1 --2- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1098039150 0x7f109803b610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.404+0000 7f10b3200700 1 --2- 172.21.15.103:0/3612869995 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10b4108570 0x7f10b419d930 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.404+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 >> 172.21.15.103:0/3612869995 conn(0x7f10b4077320 msgr2=0x7f10b4077e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.404+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 shutdown_connections 2024-05-22T23:49:50.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:50.404+0000 7f10b3200700 1 -- 172.21.15.103:0/3612869995 wait complete. 2024-05-22T23:49:50.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:50 smithi103 ceph-mon[31370]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:49:50.793 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:51.213 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi105 2024-05-22T23:49:51.214 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:49:51.214 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.conf 2024-05-22T23:49:51.231 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:49:51.232 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:49:51.289 INFO:tasks.cephadm:Adding host smithi105 to orchestrator... 2024-05-22T23:49:51.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch host add smithi105 2024-05-22T23:49:51.696 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:49:51.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:51 smithi103 ceph-mon[31370]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:51.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:49:51.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:49:51.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:49:51.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:51 smithi103 ceph-mon[31370]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-22T23:49:52.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:52 smithi103 ceph-mon[31370]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:49:52.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:52 smithi103 ceph-mon[31370]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:49:52.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:52 smithi103 ceph-mon[31370]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:49:53.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.774+0000 7f1905800700 1 -- 172.21.15.103:0/2077496300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 msgr2=0x7f19001043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:53.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.774+0000 7f1905800700 1 --2- 172.21.15.103:0/2077496300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 0x7f19001043d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18f0009910 tx=0x7f18f0009c20 comp rx=0 tx=0).stop 2024-05-22T23:49:53.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.775+0000 7f1905800700 1 -- 172.21.15.103:0/2077496300 shutdown_connections 2024-05-22T23:49:53.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.775+0000 7f1905800700 1 --2- 172.21.15.103:0/2077496300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 0x7f19001043d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:53.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.775+0000 7f1905800700 1 -- 172.21.15.103:0/2077496300 >> 172.21.15.103:0/2077496300 conn(0x7f19000ff8a0 msgr2=0x7f1900101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:53.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.775+0000 7f1905800700 1 -- 172.21.15.103:0/2077496300 shutdown_connections 2024-05-22T23:49:53.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.775+0000 7f1905800700 1 -- 172.21.15.103:0/2077496300 wait complete. 2024-05-22T23:49:53.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.776+0000 7f1905800700 1 Processor -- start 2024-05-22T23:49:53.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.776+0000 7f1905800700 1 -- start start 2024-05-22T23:49:53.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.776+0000 7f1905800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 0x7f190019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:53.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.776+0000 7f1905800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f190010e3c0 con 0x7f1900103ff0 2024-05-22T23:49:53.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.776+0000 7f18ffe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 0x7f190019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:53.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.777+0000 7f18ffe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 0x7f190019f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36336/0 (socket says 172.21.15.103:36336) 2024-05-22T23:49:53.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.777+0000 7f18ffe00700 1 -- 172.21.15.103:0/1688801709 learned_addr learned my addr 172.21.15.103:0/1688801709 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:49:53.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.777+0000 7f18ffe00700 1 -- 172.21.15.103:0/1688801709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f190019fb90 con 0x7f1900103ff0 2024-05-22T23:49:53.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.777+0000 7f18ffe00700 1 --2- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 0x7f190019f650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18f0006010 tx=0x7f18f0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:53.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.777+0000 7f18fcc00700 1 -- 172.21.15.103:0/1688801709 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18f0026070 con 0x7f1900103ff0 2024-05-22T23:49:53.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.778+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f190019fe80 con 0x7f1900103ff0 2024-05-22T23:49:53.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.778+0000 7f18fcc00700 1 -- 172.21.15.103:0/1688801709 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18f000bbd0 con 0x7f1900103ff0 2024-05-22T23:49:53.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.778+0000 7f18fcc00700 1 -- 172.21.15.103:0/1688801709 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18f0003910 con 0x7f1900103ff0 2024-05-22T23:49:53.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.778+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19001998f0 con 0x7f1900103ff0 2024-05-22T23:49:53.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.779+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18e0005330 con 0x7f1900103ff0 2024-05-22T23:49:53.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.779+0000 7f18fcc00700 1 -- 172.21.15.103:0/1688801709 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f18f0029070 con 0x7f1900103ff0 2024-05-22T23:49:53.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.779+0000 7f18fcc00700 1 --2- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18e4039150 0x7f18e403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:49:53.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.779+0000 7f18fcc00700 1 -- 172.21.15.103:0/1688801709 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f18f001e070 con 0x7f1900103ff0 2024-05-22T23:49:53.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.780+0000 7f18ff400700 1 --2- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18e4039150 0x7f18e403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:49:53.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.780+0000 7f18ff400700 1 --2- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18e4039150 0x7f18e403b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f18ec006fd0 tx=0x7f18ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:49:53.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:53.784+0000 7f18fcc00700 1 -- 172.21.15.103:0/1688801709 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f18f0013070 con 0x7f1900103ff0 2024-05-22T23:49:54.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:54.001+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}) -- 0x7f18e0000bc0 con 0x7f18e4039150 2024-05-22T23:49:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:53 smithi103 ceph-mon[31370]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-22T23:49:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:54.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:54.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:55.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:54 smithi103 ceph-mon[31370]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:49:55.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:54 smithi103 ceph-mon[31370]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:49:56.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:55 smithi103 ceph-mon[31370]: Deploying cephadm binary to smithi105 2024-05-22T23:49:57.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:56 smithi103 ceph-mon[31370]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:49:58.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.663+0000 7f18fcc00700 1 -- 172.21.15.103:0/1688801709 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f18e0000bc0 con 0x7f18e4039150 2024-05-22T23:49:58.670 INFO:teuthology.orchestra.run.smithi103.stdout:Added host 'smithi105' with addr '172.21.15.105' 2024-05-22T23:49:58.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.667+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18e4039150 msgr2=0x7f18e403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:58.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.667+0000 7f1905800700 1 --2- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18e4039150 0x7f18e403b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f18ec006fd0 tx=0x7f18ec006e40 comp rx=0 tx=0).stop 2024-05-22T23:49:58.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.667+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 msgr2=0x7f190019f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:49:58.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.667+0000 7f1905800700 1 --2- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 0x7f190019f650 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18f0006010 tx=0x7f18f0005030 comp rx=0 tx=0).stop 2024-05-22T23:49:58.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.667+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 shutdown_connections 2024-05-22T23:49:58.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.667+0000 7f1905800700 1 --2- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18e4039150 0x7f18e403b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:58.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.667+0000 7f1905800700 1 --2- 172.21.15.103:0/1688801709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1900103ff0 0x7f190019f650 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:49:58.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.667+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 >> 172.21.15.103:0/1688801709 conn(0x7f19000ff8a0 msgr2=0x7f19001059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:49:58.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.668+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 shutdown_connections 2024-05-22T23:49:58.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:49:58.668+0000 7f1905800700 1 -- 172.21.15.103:0/1688801709 wait complete. 2024-05-22T23:49:59.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:59 smithi103 ceph-mon[31370]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:49:59.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:59 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:49:59.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:49:59 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:49:59.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch host ls --format=json 2024-05-22T23:49:59.848 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:50:00.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:00 smithi103 ceph-mon[31370]: Added host smithi105 2024-05-22T23:50:00.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:00 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:00.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:00 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:01.218 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:01 smithi103 ceph-mon[31370]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:02.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.996+0000 7f407ec00700 1 -- 172.21.15.103:0/493839314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 msgr2=0x7f40781043d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:02.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.996+0000 7f407ec00700 1 --2- 172.21.15.103:0/493839314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 0x7f40781043d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f406c009910 tx=0x7f406c009c20 comp rx=0 tx=0).stop 2024-05-22T23:50:02.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.996+0000 7f407ec00700 1 -- 172.21.15.103:0/493839314 shutdown_connections 2024-05-22T23:50:02.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.996+0000 7f407ec00700 1 --2- 172.21.15.103:0/493839314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 0x7f40781043d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:02.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.996+0000 7f407ec00700 1 -- 172.21.15.103:0/493839314 >> 172.21.15.103:0/493839314 conn(0x7f40780ff8a0 msgr2=0x7f4078101cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:02.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.996+0000 7f407ec00700 1 -- 172.21.15.103:0/493839314 shutdown_connections 2024-05-22T23:50:02.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.996+0000 7f407ec00700 1 -- 172.21.15.103:0/493839314 wait complete. 2024-05-22T23:50:02.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.997+0000 7f407ec00700 1 Processor -- start 2024-05-22T23:50:02.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.997+0000 7f407ec00700 1 -- start start 2024-05-22T23:50:02.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.997+0000 7f407ec00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 0x7f407819f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:02.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.997+0000 7f407ec00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f407810e3c0 con 0x7f4078103ff0 2024-05-22T23:50:02.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.998+0000 7f407d800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 0x7f407819f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:02.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.998+0000 7f407d800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 0x7f407819f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53684/0 (socket says 172.21.15.103:53684) 2024-05-22T23:50:02.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.998+0000 7f407d800700 1 -- 172.21.15.103:0/883670116 learned_addr learned my addr 172.21.15.103:0/883670116 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:50:02.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.998+0000 7f407d800700 1 -- 172.21.15.103:0/883670116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f407819fb60 con 0x7f4078103ff0 2024-05-22T23:50:02.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.998+0000 7f407d800700 1 --2- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 0x7f407819f620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f406c006010 tx=0x7f406c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:02.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.999+0000 7f4076000700 1 -- 172.21.15.103:0/883670116 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f406c026070 con 0x7f4078103ff0 2024-05-22T23:50:02.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.999+0000 7f4076000700 1 -- 172.21.15.103:0/883670116 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f406c00bbd0 con 0x7f4078103ff0 2024-05-22T23:50:02.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.999+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f407819fe50 con 0x7f4078103ff0 2024-05-22T23:50:02.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.999+0000 7f4076000700 1 -- 172.21.15.103:0/883670116 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f406c003910 con 0x7f4078103ff0 2024-05-22T23:50:02.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:01.999+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40781998c0 con 0x7f4078103ff0 2024-05-22T23:50:02.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.000+0000 7f4076000700 1 -- 172.21.15.103:0/883670116 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f406c029070 con 0x7f4078103ff0 2024-05-22T23:50:02.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.000+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40781054d0 con 0x7f4078103ff0 2024-05-22T23:50:02.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.000+0000 7f4076000700 1 --2- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f40600390f0 0x7f406003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:02.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.000+0000 7f4076000700 1 -- 172.21.15.103:0/883670116 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f406c01e070 con 0x7f4078103ff0 2024-05-22T23:50:02.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.001+0000 7f407ce00700 1 --2- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f40600390f0 0x7f406003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:02.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.001+0000 7f407ce00700 1 --2- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f40600390f0 0x7f406003b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4064006fd0 tx=0x7f4064006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:02.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.006+0000 7f4076000700 1 -- 172.21.15.103:0/883670116 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f406c013070 con 0x7f4078103ff0 2024-05-22T23:50:02.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.237+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f407819acc0 con 0x7f40600390f0 2024-05-22T23:50:02.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.240+0000 7f4076000700 1 -- 172.21.15.103:0/883670116 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7f407819acc0 con 0x7f40600390f0 2024-05-22T23:50:02.246 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:50:02.247 INFO:teuthology.orchestra.run.smithi103.stdout:[{"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}] 2024-05-22T23:50:02.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.243+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f40600390f0 msgr2=0x7f406003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:02.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 --2- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f40600390f0 0x7f406003b5b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4064006fd0 tx=0x7f4064006e40 comp rx=0 tx=0).stop 2024-05-22T23:50:02.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 msgr2=0x7f407819f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:02.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 --2- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 0x7f407819f620 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f406c006010 tx=0x7f406c005030 comp rx=0 tx=0).stop 2024-05-22T23:50:02.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 shutdown_connections 2024-05-22T23:50:02.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 --2- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f40600390f0 0x7f406003b5b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:02.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 --2- 172.21.15.103:0/883670116 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4078103ff0 0x7f407819f620 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:02.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 >> 172.21.15.103:0/883670116 conn(0x7f40780ff8a0 msgr2=0x7f40781059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:02.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 shutdown_connections 2024-05-22T23:50:02.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:02.244+0000 7f407ec00700 1 -- 172.21.15.103:0/883670116 wait complete. 2024-05-22T23:50:02.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:02 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:02.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:02 smithi103 ceph-mon[31370]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:03.034 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi174 2024-05-22T23:50:03.034 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:50:03.034 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.conf 2024-05-22T23:50:03.053 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:50:03.053 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:03.110 INFO:tasks.cephadm:Adding host smithi174 to orchestrator... 2024-05-22T23:50:03.110 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch host add smithi174 2024-05-22T23:50:03.554 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:50:03.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:03 smithi103 ceph-mon[31370]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:50:04.728 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:04 smithi103 ceph-mon[31370]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:05.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.558+0000 7f829a000700 1 -- 172.21.15.103:0/579773384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 msgr2=0x7f8294104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:05.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.558+0000 7f829a000700 1 --2- 172.21.15.103:0/579773384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 0x7f8294104590 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8284009910 tx=0x7f8284009c20 comp rx=0 tx=0).stop 2024-05-22T23:50:05.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.558+0000 7f829a000700 1 -- 172.21.15.103:0/579773384 shutdown_connections 2024-05-22T23:50:05.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.558+0000 7f829a000700 1 --2- 172.21.15.103:0/579773384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 0x7f8294104590 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:05.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.558+0000 7f829a000700 1 -- 172.21.15.103:0/579773384 >> 172.21.15.103:0/579773384 conn(0x7f82940ffa80 msgr2=0x7f8294101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:05.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.558+0000 7f829a000700 1 -- 172.21.15.103:0/579773384 shutdown_connections 2024-05-22T23:50:05.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.558+0000 7f829a000700 1 -- 172.21.15.103:0/579773384 wait complete. 2024-05-22T23:50:05.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.560+0000 7f829a000700 1 Processor -- start 2024-05-22T23:50:05.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.560+0000 7f829a000700 1 -- start start 2024-05-22T23:50:05.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.560+0000 7f829a000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 0x7f829419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:05.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.560+0000 7f829a000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f829410e580 con 0x7f82941041b0 2024-05-22T23:50:05.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.561+0000 7f8293400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 0x7f829419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:05.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.561+0000 7f8293400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 0x7f829419f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46456/0 (socket says 172.21.15.103:46456) 2024-05-22T23:50:05.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.561+0000 7f8293400700 1 -- 172.21.15.103:0/836260396 learned_addr learned my addr 172.21.15.103:0/836260396 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:50:05.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.561+0000 7f8293400700 1 -- 172.21.15.103:0/836260396 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829419fcd0 con 0x7f82941041b0 2024-05-22T23:50:05.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.561+0000 7f8293400700 1 --2- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 0x7f829419f790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8284005850 tx=0x7f8284005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:05.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.562+0000 7f828be00700 1 -- 172.21.15.103:0/836260396 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8284026070 con 0x7f82941041b0 2024-05-22T23:50:05.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.562+0000 7f828be00700 1 -- 172.21.15.103:0/836260396 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f828400bbd0 con 0x7f82941041b0 2024-05-22T23:50:05.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.562+0000 7f828be00700 1 -- 172.21.15.103:0/836260396 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8284003970 con 0x7f82941041b0 2024-05-22T23:50:05.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.562+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f829419ffc0 con 0x7f82941041b0 2024-05-22T23:50:05.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.562+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8294199950 con 0x7f82941041b0 2024-05-22T23:50:05.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.563+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8294105690 con 0x7f82941041b0 2024-05-22T23:50:05.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.563+0000 7f828be00700 1 -- 172.21.15.103:0/836260396 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f8284029070 con 0x7f82941041b0 2024-05-22T23:50:05.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.564+0000 7f828be00700 1 --2- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8278039150 0x7f827803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:05.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.564+0000 7f828be00700 1 -- 172.21.15.103:0/836260396 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f828401e070 con 0x7f82941041b0 2024-05-22T23:50:05.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.564+0000 7f8292a00700 1 --2- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8278039150 0x7f827803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:05.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.564+0000 7f8292a00700 1 --2- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8278039150 0x7f827803b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8280006fd0 tx=0x7f8280006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:05.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.569+0000 7f828be00700 1 -- 172.21.15.103:0/836260396 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8284013070 con 0x7f82941041b0 2024-05-22T23:50:05.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:05.798+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}) -- 0x7f829419ae80 con 0x7f8278039150 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-22T23:50:05.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:05 smithi103 ceph-mon[31370]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:06 smithi103 ceph-mon[31370]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:06 smithi103 ceph-mon[31370]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-22T23:50:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:06 smithi103 ceph-mon[31370]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:06 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:06 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:06 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:06 smithi103 ceph-mon[31370]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:08.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:07 smithi103 ceph-mon[31370]: Deploying cephadm binary to smithi174 2024-05-22T23:50:09.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:08 smithi103 ceph-mon[31370]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:10.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.520+0000 7f828be00700 1 -- 172.21.15.103:0/836260396 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f829419ae80 con 0x7f8278039150 2024-05-22T23:50:10.525 INFO:teuthology.orchestra.run.smithi103.stdout:Added host 'smithi174' with addr '172.21.15.174' 2024-05-22T23:50:10.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8278039150 msgr2=0x7f827803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 --2- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8278039150 0x7f827803b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8280006fd0 tx=0x7f8280006e40 comp rx=0 tx=0).stop 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 msgr2=0x7f829419f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 --2- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 0x7f829419f790 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8284005850 tx=0x7f8284005030 comp rx=0 tx=0).stop 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 shutdown_connections 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 --2- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8278039150 0x7f827803b610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 --2- 172.21.15.103:0/836260396 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f82941041b0 0x7f829419f790 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 >> 172.21.15.103:0/836260396 conn(0x7f82940ffa80 msgr2=0x7f8294106fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 shutdown_connections 2024-05-22T23:50:10.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:10.524+0000 7f829a000700 1 -- 172.21.15.103:0/836260396 wait complete. 2024-05-22T23:50:11.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:11 smithi103 ceph-mon[31370]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:11.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:11.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:11.412 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch host ls --format=json 2024-05-22T23:50:11.843 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:50:12.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:12 smithi103 ceph-mon[31370]: Added host smithi174 2024-05-22T23:50:12.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:12.294 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:13.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:13 smithi103 ceph-mon[31370]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.985+0000 7feca0c00700 1 -- 172.21.15.103:0/1785792946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 msgr2=0x7fec9c1021e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.985+0000 7feca0c00700 1 --2- 172.21.15.103:0/1785792946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 0x7fec9c1021e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fec8c009910 tx=0x7fec8c009c20 comp rx=0 tx=0).stop 2024-05-22T23:50:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.985+0000 7feca0c00700 1 -- 172.21.15.103:0/1785792946 shutdown_connections 2024-05-22T23:50:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.985+0000 7feca0c00700 1 --2- 172.21.15.103:0/1785792946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 0x7fec9c1021e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.985+0000 7feca0c00700 1 -- 172.21.15.103:0/1785792946 >> 172.21.15.103:0/1785792946 conn(0x7fec9c0fd6f0 msgr2=0x7fec9c0ffb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.985+0000 7feca0c00700 1 -- 172.21.15.103:0/1785792946 shutdown_connections 2024-05-22T23:50:13.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.985+0000 7feca0c00700 1 -- 172.21.15.103:0/1785792946 wait complete. 2024-05-22T23:50:13.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.986+0000 7feca0c00700 1 Processor -- start 2024-05-22T23:50:13.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.986+0000 7feca0c00700 1 -- start start 2024-05-22T23:50:13.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.987+0000 7feca0c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 0x7fec9c19f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:13.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.987+0000 7feca0c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fec9c10c1d0 con 0x7fec9c101e00 2024-05-22T23:50:13.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.987+0000 7fec9b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 0x7fec9c19f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:13.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.987+0000 7fec9b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 0x7fec9c19f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45104/0 (socket says 172.21.15.103:45104) 2024-05-22T23:50:13.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.987+0000 7fec9b400700 1 -- 172.21.15.103:0/52160299 learned_addr learned my addr 172.21.15.103:0/52160299 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:50:13.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.987+0000 7fec9b400700 1 -- 172.21.15.103:0/52160299 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9c19fb90 con 0x7fec9c101e00 2024-05-22T23:50:13.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.988+0000 7fec9b400700 1 --2- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 0x7fec9c19f650 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec8c006010 tx=0x7fec8c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:13.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.988+0000 7fec93e00700 1 -- 172.21.15.103:0/52160299 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec8c026070 con 0x7fec9c101e00 2024-05-22T23:50:13.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.988+0000 7fec93e00700 1 -- 172.21.15.103:0/52160299 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fec8c00bbd0 con 0x7fec9c101e00 2024-05-22T23:50:13.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.988+0000 7fec93e00700 1 -- 172.21.15.103:0/52160299 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec8c003910 con 0x7fec9c101e00 2024-05-22T23:50:13.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.988+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9c19fe80 con 0x7fec9c101e00 2024-05-22T23:50:13.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.988+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec9c1998f0 con 0x7fec9c101e00 2024-05-22T23:50:13.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.989+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec9c1032e0 con 0x7fec9c101e00 2024-05-22T23:50:13.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.989+0000 7fec93e00700 1 -- 172.21.15.103:0/52160299 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fec8c029070 con 0x7fec9c101e00 2024-05-22T23:50:13.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.990+0000 7fec93e00700 1 --2- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fec80039150 0x7fec8003b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:13.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.990+0000 7fec93e00700 1 -- 172.21.15.103:0/52160299 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fec8c01e070 con 0x7fec9c101e00 2024-05-22T23:50:13.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.990+0000 7fec9aa00700 1 --2- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fec80039150 0x7fec8003b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:13.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.991+0000 7fec9aa00700 1 --2- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fec80039150 0x7fec8003b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fec88006fd0 tx=0x7fec88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:13.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:13.995+0000 7fec93e00700 1 -- 172.21.15.103:0/52160299 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fec8c050030 con 0x7fec9c101e00 2024-05-22T23:50:14.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.224+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fec9c19ac50 con 0x7fec80039150 2024-05-22T23:50:14.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.226+0000 7fec93e00700 1 -- 172.21.15.103:0/52160299 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7fec9c19ac50 con 0x7fec80039150 2024-05-22T23:50:14.230 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:50:14.230 INFO:teuthology.orchestra.run.smithi103.stdout:[{"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}] 2024-05-22T23:50:14.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fec80039150 msgr2=0x7fec8003b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:14.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 --2- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fec80039150 0x7fec8003b610 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fec88006fd0 tx=0x7fec88006e40 comp rx=0 tx=0).stop 2024-05-22T23:50:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 msgr2=0x7fec9c19f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 --2- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 0x7fec9c19f650 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec8c006010 tx=0x7fec8c005030 comp rx=0 tx=0).stop 2024-05-22T23:50:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 shutdown_connections 2024-05-22T23:50:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 --2- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fec80039150 0x7fec8003b610 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 --2- 172.21.15.103:0/52160299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fec9c101e00 0x7fec9c19f650 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 >> 172.21.15.103:0/52160299 conn(0x7fec9c0fd6f0 msgr2=0x7fec9c074590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 shutdown_connections 2024-05-22T23:50:14.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:14.229+0000 7feca0c00700 1 -- 172.21.15.103:0/52160299 wait complete. 2024-05-22T23:50:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:14 smithi103 ceph-mon[31370]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:14.998 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-22T23:50:14.998 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd crush tunables default 2024-05-22T23:50:15.410 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:50:15.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:15 smithi103 ceph-mon[31370]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:50:16.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:16 smithi103 ceph-mon[31370]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:16.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:16 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:16.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:16 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:16.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:16 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:16.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:16 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:17.450 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:50:17.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:17.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:17.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:17 smithi103 ceph-mon[31370]: Updating smithi174:/etc/ceph/ceph.conf 2024-05-22T23:50:17.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:17 smithi103 ceph-mon[31370]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:17.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:17 smithi103 ceph-mon[31370]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:17.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.616+0000 7fe848a00700 1 -- 172.21.15.103:0/2846140867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 msgr2=0x7fe844074b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:17.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.616+0000 7fe848a00700 1 --2- 172.21.15.103:0/2846140867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 0x7fe844074b10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe834009910 tx=0x7fe834009c20 comp rx=0 tx=0).stop 2024-05-22T23:50:17.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.616+0000 7fe848a00700 1 -- 172.21.15.103:0/2846140867 shutdown_connections 2024-05-22T23:50:17.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.616+0000 7fe848a00700 1 --2- 172.21.15.103:0/2846140867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 0x7fe844074b10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:17.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.616+0000 7fe848a00700 1 -- 172.21.15.103:0/2846140867 >> 172.21.15.103:0/2846140867 conn(0x7fe8440fd880 msgr2=0x7fe8440ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:17.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.616+0000 7fe848a00700 1 -- 172.21.15.103:0/2846140867 shutdown_connections 2024-05-22T23:50:17.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.616+0000 7fe848a00700 1 -- 172.21.15.103:0/2846140867 wait complete. 2024-05-22T23:50:17.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.617+0000 7fe848a00700 1 Processor -- start 2024-05-22T23:50:17.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.617+0000 7fe848a00700 1 -- start start 2024-05-22T23:50:17.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.618+0000 7fe848a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 0x7fe84419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:17.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.618+0000 7fe848a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe84410e5f0 con 0x7fe844074730 2024-05-22T23:50:17.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.618+0000 7fe841e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 0x7fe84419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:17.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.618+0000 7fe841e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 0x7fe84419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45128/0 (socket says 172.21.15.103:45128) 2024-05-22T23:50:17.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.618+0000 7fe841e00700 1 -- 172.21.15.103:0/3195254633 learned_addr learned my addr 172.21.15.103:0/3195254633 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:50:17.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.619+0000 7fe841e00700 1 -- 172.21.15.103:0/3195254633 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84419fc80 con 0x7fe844074730 2024-05-22T23:50:17.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.619+0000 7fe841e00700 1 --2- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 0x7fe84419f740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe834006010 tx=0x7fe834005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:17.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.619+0000 7fe83aa00700 1 -- 172.21.15.103:0/3195254633 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe834026070 con 0x7fe844074730 2024-05-22T23:50:17.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.619+0000 7fe83aa00700 1 -- 172.21.15.103:0/3195254633 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe83400bc30 con 0x7fe844074730 2024-05-22T23:50:17.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.619+0000 7fe848a00700 1 -- 172.21.15.103:0/3195254633 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84419ff10 con 0x7fe844074730 2024-05-22T23:50:17.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.619+0000 7fe83aa00700 1 -- 172.21.15.103:0/3195254633 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe8340039a0 con 0x7fe844074730 2024-05-22T23:50:17.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.619+0000 7fe848a00700 1 -- 172.21.15.103:0/3195254633 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe844199900 con 0x7fe844074730 2024-05-22T23:50:17.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.621+0000 7fe83aa00700 1 -- 172.21.15.103:0/3195254633 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fe834029070 con 0x7fe844074730 2024-05-22T23:50:17.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.621+0000 7fe82fe00700 1 -- 172.21.15.103:0/3195254633 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe844075de0 con 0x7fe844074730 2024-05-22T23:50:17.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.621+0000 7fe83aa00700 1 --2- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe8280390f0 0x7fe82803b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:17.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.621+0000 7fe83aa00700 1 -- 172.21.15.103:0/3195254633 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe83401e070 con 0x7fe844074730 2024-05-22T23:50:17.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.621+0000 7fe841400700 1 --2- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe8280390f0 0x7fe82803b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:17.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.622+0000 7fe841400700 1 --2- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe8280390f0 0x7fe82803b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe830006fd0 tx=0x7fe830006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:17.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.627+0000 7fe83aa00700 1 -- 172.21.15.103:0/3195254633 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe834013070 con 0x7fe844074730 2024-05-22T23:50:17.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:17.869+0000 7fe82fe00700 1 -- 172.21.15.103:0/3195254633 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fe8440639c0 con 0x7fe844074730 2024-05-22T23:50:18.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.514+0000 7fe83aa00700 1 -- 172.21.15.103:0/3195254633 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fe834030090 con 0x7fe844074730 2024-05-22T23:50:18.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 -- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe8280390f0 msgr2=0x7fe82803b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:18.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 --2- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe8280390f0 0x7fe82803b5b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe830006fd0 tx=0x7fe830006e40 comp rx=0 tx=0).stop 2024-05-22T23:50:18.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 -- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 msgr2=0x7fe84419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:18.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 --2- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 0x7fe84419f740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe834006010 tx=0x7fe834005030 comp rx=0 tx=0).stop 2024-05-22T23:50:18.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 -- 172.21.15.103:0/3195254633 shutdown_connections 2024-05-22T23:50:18.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 --2- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe8280390f0 0x7fe82803b5b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:18.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 --2- 172.21.15.103:0/3195254633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe844074730 0x7fe84419f740 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:18.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 -- 172.21.15.103:0/3195254633 >> 172.21.15.103:0/3195254633 conn(0x7fe8440fd880 msgr2=0x7fe844106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:18.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 -- 172.21.15.103:0/3195254633 shutdown_connections 2024-05-22T23:50:18.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:18.519+0000 7fe82fe00700 1 -- 172.21.15.103:0/3195254633 wait complete. 2024-05-22T23:50:18.526 INFO:teuthology.orchestra.run.smithi103.stderr:adjusted tunables profile to default 2024-05-22T23:50:18.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:18 smithi103 ceph-mon[31370]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-22T23:50:18.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:18.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:18.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:18.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:18 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3195254633' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-22T23:50:18.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:18 smithi103 ceph-mon[31370]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:19.325 INFO:tasks.cephadm:Adding mon.a on smithi103 2024-05-22T23:50:19.325 INFO:tasks.cephadm:Adding mon.b on smithi105 2024-05-22T23:50:19.325 INFO:tasks.cephadm:Adding mon.c on smithi174 2024-05-22T23:50:19.326 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch apply mon '3;smithi103:172.21.15.103=a;smithi105:172.21.15.105=b;smithi174:172.21.15.174=c' 2024-05-22T23:50:19.561 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:19.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:19 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3195254633' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-22T23:50:19.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:19 smithi103 ceph-mon[31370]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:20.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:20 smithi103 ceph-mon[31370]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:20.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.965+0000 7f0bee23c700 1 -- 172.21.15.174:0/3248707385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 msgr2=0x7f0be8102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:20.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.965+0000 7f0bee23c700 1 --2- 172.21.15.174:0/3248707385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 0x7f0be8102390 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009910 tx=0x7f0bd8009c20 comp rx=0 tx=0).stop 2024-05-22T23:50:20.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.965+0000 7f0bee23c700 1 -- 172.21.15.174:0/3248707385 shutdown_connections 2024-05-22T23:50:20.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.965+0000 7f0bee23c700 1 --2- 172.21.15.174:0/3248707385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 0x7f0be8102390 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:20.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.965+0000 7f0bee23c700 1 -- 172.21.15.174:0/3248707385 >> 172.21.15.174:0/3248707385 conn(0x7f0be80fd880 msgr2=0x7f0be80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:20.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.965+0000 7f0bee23c700 1 -- 172.21.15.174:0/3248707385 shutdown_connections 2024-05-22T23:50:20.970 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.965+0000 7f0bee23c700 1 -- 172.21.15.174:0/3248707385 wait complete. 2024-05-22T23:50:20.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.966+0000 7f0bee23c700 1 Processor -- start 2024-05-22T23:50:20.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.966+0000 7f0bee23c700 1 -- start start 2024-05-22T23:50:20.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.966+0000 7f0bee23c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 0x7f0be819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:20.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.966+0000 7f0bee23c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0be810c380 con 0x7f0be8101fb0 2024-05-22T23:50:20.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.966+0000 7f0be77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 0x7f0be819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:20.971 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.966+0000 7f0be77fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 0x7f0be819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.174:49922/0 (socket says 172.21.15.174:49922) 2024-05-22T23:50:20.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.966+0000 7f0be77fe700 1 -- 172.21.15.174:0/3874609484 learned_addr learned my addr 172.21.15.174:0/3874609484 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:50:20.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.967+0000 7f0be77fe700 1 -- 172.21.15.174:0/3874609484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be819db30 con 0x7f0be8101fb0 2024-05-22T23:50:20.972 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.967+0000 7f0be77fe700 1 --2- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 0x7f0be819d5f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8005850 tx=0x7f0bd8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:20.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.968+0000 7f0be4ff9700 1 -- 172.21.15.174:0/3874609484 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd8026070 con 0x7f0be8101fb0 2024-05-22T23:50:20.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.968+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0be819de20 con 0x7f0be8101fb0 2024-05-22T23:50:20.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.968+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0be8197890 con 0x7f0be8101fb0 2024-05-22T23:50:20.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.968+0000 7f0be4ff9700 1 -- 172.21.15.174:0/3874609484 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0bd800b6f0 con 0x7f0be8101fb0 2024-05-22T23:50:20.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.968+0000 7f0be4ff9700 1 -- 172.21.15.174:0/3874609484 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0bd8030060 con 0x7f0be8101fb0 2024-05-22T23:50:20.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.969+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd4005330 con 0x7f0be8101fb0 2024-05-22T23:50:20.973 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.969+0000 7f0be4ff9700 1 -- 172.21.15.174:0/3874609484 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f0bd8029070 con 0x7f0be8101fb0 2024-05-22T23:50:20.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.969+0000 7f0be4ff9700 1 --2- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f0bd00390f0 0x7f0bd003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:20.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.969+0000 7f0be4ff9700 1 -- 172.21.15.174:0/3874609484 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0bd801e070 con 0x7f0be8101fb0 2024-05-22T23:50:20.974 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.970+0000 7f0be6ffd700 1 --2- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f0bd00390f0 0x7f0bd003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:20.975 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.971+0000 7f0be6ffd700 1 --2- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f0bd00390f0 0x7f0bd003b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc006fd0 tx=0x7f0bdc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:20.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:20.985+0000 7f0be4ff9700 1 -- 172.21.15.174:0/3874609484 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0bd8013070 con 0x7f0be8101fb0 2024-05-22T23:50:21.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.192+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi103:172.21.15.103=a;smithi105:172.21.15.105=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}) -- 0x7f0bd4000c80 con 0x7f0bd00390f0 2024-05-22T23:50:21.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.207+0000 7f0be4ff9700 1 -- 172.21.15.174:0/3874609484 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0bd4000c80 con 0x7f0bd00390f0 2024-05-22T23:50:21.212 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mon update... 2024-05-22T23:50:21.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f0bd00390f0 msgr2=0x7f0bd003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:21.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 --2- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f0bd00390f0 0x7f0bd003b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc006fd0 tx=0x7f0bdc006e40 comp rx=0 tx=0).stop 2024-05-22T23:50:21.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 msgr2=0x7f0be819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:21.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 --2- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 0x7f0be819d5f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8005850 tx=0x7f0bd8005030 comp rx=0 tx=0).stop 2024-05-22T23:50:21.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 shutdown_connections 2024-05-22T23:50:21.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 --2- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f0bd00390f0 0x7f0bd003b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:21.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 --2- 172.21.15.174:0/3874609484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0be8101fb0 0x7f0be819d5f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:21.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 >> 172.21.15.174:0/3874609484 conn(0x7f0be80fd880 msgr2=0x7f0be8103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:21.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 shutdown_connections 2024-05-22T23:50:21.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:21.209+0000 7f0bee23c700 1 -- 172.21.15.174:0/3874609484 wait complete. 2024-05-22T23:50:21.816 DEBUG:teuthology.orchestra.run.smithi105:mon.b> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.b.service 2024-05-22T23:50:21.819 DEBUG:teuthology.orchestra.run.smithi174:mon.c> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.c.service 2024-05-22T23:50:21.821 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-22T23:50:21.821 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-22T23:50:22.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi103:172.21.15.103=a;smithi105:172.21.15.105=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:22.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: Saving service mon spec with placement smithi103:172.21.15.103=a;smithi105:172.21.15.105=b;smithi174:172.21.15.174=c;count:3 2024-05-22T23:50:22.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:22.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:22.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:22.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:22.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:22.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:22.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:22.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: Deploying daemon mon.c on smithi174 2024-05-22T23:50:22.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:22 smithi103 ceph-mon[31370]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:22.841 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:25 smithi103 ceph-mon[31370]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:25.571 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 -- 172.21.15.174:0/3852873097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 msgr2=0x7f89b8106840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:25.571 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 --2- 172.21.15.174:0/3852873097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 0x7f89b8106840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f89a800aa20 tx=0x7f89a800ad30 comp rx=0 tx=0).stop 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 -- 172.21.15.174:0/3852873097 shutdown_connections 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 --2- 172.21.15.174:0/3852873097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 0x7f89b8106840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 -- 172.21.15.174:0/3852873097 >> 172.21.15.174:0/3852873097 conn(0x7f89b8077310 msgr2=0x7f89b8077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 -- 172.21.15.174:0/3852873097 shutdown_connections 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 -- 172.21.15.174:0/3852873097 wait complete. 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 Processor -- start 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.566+0000 7f89be42f700 1 -- start start 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.567+0000 7f89be42f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 0x7f89b81954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:25.572 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.567+0000 7f89be42f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89b810a190 con 0x7f89b8106460 2024-05-22T23:50:25.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.567+0000 7f89b7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 0x7f89b81954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:25.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.567+0000 7f89b7fff700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 0x7f89b81954c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.174:49940/0 (socket says 172.21.15.174:49940) 2024-05-22T23:50:25.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.567+0000 7f89b7fff700 1 -- 172.21.15.174:0/3271910314 learned_addr learned my addr 172.21.15.174:0/3271910314 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:50:25.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.568+0000 7f89b7fff700 1 -- 172.21.15.174:0/3271910314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b81973b0 con 0x7f89b8106460 2024-05-22T23:50:25.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.568+0000 7f89b7fff700 1 --2- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 0x7f89b81954c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f89a8004c90 tx=0x7f89a8004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.569+0000 7f89b57fa700 1 -- 172.21.15.174:0/3271910314 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89a8005310 con 0x7f89b8106460 2024-05-22T23:50:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.569+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b8195a60 con 0x7f89b8106460 2024-05-22T23:50:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.569+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b8195f20 con 0x7f89b8106460 2024-05-22T23:50:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.569+0000 7f89b57fa700 1 -- 172.21.15.174:0/3271910314 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89a80054b0 con 0x7f89b8106460 2024-05-22T23:50:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.569+0000 7f89b57fa700 1 -- 172.21.15.174:0/3271910314 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f89a800e740 con 0x7f89b8106460 2024-05-22T23:50:25.574 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.570+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a4005330 con 0x7f89b8106460 2024-05-22T23:50:25.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.570+0000 7f89b57fa700 1 -- 172.21.15.174:0/3271910314 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f89a800e9b0 con 0x7f89b8106460 2024-05-22T23:50:25.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.570+0000 7f89b57fa700 1 --2- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89a00390f0 0x7f89a003b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:25.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.570+0000 7f89b57fa700 1 -- 172.21.15.174:0/3271910314 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f89a801e070 con 0x7f89b8106460 2024-05-22T23:50:25.575 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.571+0000 7f89b77fe700 1 --2- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89a00390f0 0x7f89a003b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:25.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.572+0000 7f89b77fe700 1 --2- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89a00390f0 0x7f89a003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f89ac006fd0 tx=0x7f89ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:25.590 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.586+0000 7f89b57fa700 1 -- 172.21.15.174:0/3271910314 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f89a800edf0 con 0x7f89b8106460 2024-05-22T23:50:25.877 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.872+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f89a40058f0 con 0x7f89b8106460 2024-05-22T23:50:25.878 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.874+0000 7f89b57fa700 1 -- 172.21.15.174:0/3271910314 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f89a8030070 con 0x7f89b8106460 2024-05-22T23:50:25.879 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:50:25.879 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":1,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","modified":"2024-05-22T23:48:16.842516Z","created":"2024-05-22T23:48:16.842516Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89a00390f0 msgr2=0x7f89a003b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 --2- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89a00390f0 0x7f89a003b5b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f89ac006fd0 tx=0x7f89ac006e40 comp rx=0 tx=0).stop 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 msgr2=0x7f89b81954c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 --2- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 0x7f89b81954c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f89a8004c90 tx=0x7f89a8004d70 comp rx=0 tx=0).stop 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 shutdown_connections 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 --2- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89a00390f0 0x7f89a003b5b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 --2- 172.21.15.174:0/3271910314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89b8106460 0x7f89b81954c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 >> 172.21.15.174:0/3271910314 conn(0x7f89b8077310 msgr2=0x7f89b80fd300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 shutdown_connections 2024-05-22T23:50:25.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:25.876+0000 7f89be42f700 1 -- 172.21.15.174:0/3271910314 wait complete. 2024-05-22T23:50:25.882 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 1 2024-05-22T23:50:25.956 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 podman[36147]: 2024-05-22 23:50:25.928775357 +0000 UTC m=+0.346178065 container start bf01ac9e5176755fce6e0611148b74411c56235d93fa5855dc6a865623eaa459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-c, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-22T23:50:25.957 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: set uid:gid to 167:167 (ceph:ceph) 2024-05-22T23:50:25.957 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: ceph version 19.0.0-2765-g2e1f7622 (2e1f76222e88afe0f1cda37775b2d45d88289794) squid (rc), process ceph-mon, pid 2 2024-05-22T23:50:25.957 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pidfile_write: ignore empty --pid-file 2024-05-22T23:50:26.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: load: jerasure load: lrc 2024-05-22T23:50:26.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: RocksDB version: 7.9.2 2024-05-22T23:50:26.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Git sha 0 2024-05-22T23:50:26.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Compile date 2024-05-22 16:19:42 2024-05-22T23:50:26.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: DB SUMMARY 2024-05-22T23:50:26.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: DB Session ID: 8D46AEA2AW4NRX4VCZ0B 2024-05-22T23:50:26.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: CURRENT file: CURRENT 2024-05-22T23:50:26.375 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: IDENTITY file: IDENTITY 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.error_if_exists: 0 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.create_if_missing: 0 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.paranoid_checks: 1 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-22T23:50:26.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.env: 0x563d4ecf0c60 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.fs: PosixFileSystem 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.info_log: 0x563d50b27ae0 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_file_opening_threads: 16 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.statistics: (nil) 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.use_fsync: 0 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_log_file_size: 0 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.keep_log_file_num: 1000 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.recycle_log_file_num: 0 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.allow_fallocate: 1 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.allow_mmap_reads: 0 2024-05-22T23:50:26.377 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.allow_mmap_writes: 0 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.use_direct_reads: 0 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.create_missing_column_families: 0 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.db_log_dir: 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.wal_dir: 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-22T23:50:26.378 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-22T23:50:26.381 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-22T23:50:26.381 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-22T23:50:26.381 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.advise_random_on_open: 1 2024-05-22T23:50:26.381 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.db_write_buffer_size: 0 2024-05-22T23:50:26.381 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.write_buffer_manager: 0x563d50af81e0 2024-05-22T23:50:26.381 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-22T23:50:26.381 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-22T23:50:26.381 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.rate_limiter: (nil) 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.wal_recovery_mode: 2 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.enable_thread_tracking: 0 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.enable_pipelined_write: 0 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.unordered_write: 0 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-22T23:50:26.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.row_cache: None 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.wal_filter: None 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.allow_ingest_behind: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.two_write_queues: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.manual_wal_flush: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.wal_compression: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.atomic_flush: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-22T23:50:26.383 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.log_readahead_size: 0 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.best_efforts_recovery: 0 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.allow_data_in_errors: 0 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.db_host_id: __hostname__ 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_background_jobs: 2 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_background_compactions: -1 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_subcompactions: 1 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-22T23:50:26.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_total_wal_size: 0 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_open_files: -1 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bytes_per_sync: 0 2024-05-22T23:50:26.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_readahead_size: 0 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_background_flushes: -1 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Compression algorithms supported: 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: kZSTD supported: 0 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: kXpressCompression supported: 0 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: kLZ4HCCompression supported: 1 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: kZlibCompression supported: 1 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: kSnappyCompression supported: 1 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: kLZ4Compression supported: 1 2024-05-22T23:50:26.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: kBZip2Compression supported: 0 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.merge_operator: 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_filter: None 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_filter_factory: None 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.sst_partitioner_factory: None 2024-05-22T23:50:26.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563d50b27be0) 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: cache_index_and_filter_blocks: 1 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: pin_top_level_index_and_filter: 1 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_type: 0 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: data_block_index_type: 0 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_shortening: 1 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: checksum: 4 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: no_block_cache: 0 2024-05-22T23:50:26.388 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache: 0x563d50b29090 2024-05-22T23:50:26.389 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_name: BinnedLRUCache 2024-05-22T23:50:26.389 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_options: 2024-05-22T23:50:26.389 INFO:journalctl@ceph.mon.c.smithi174.stdout: capacity : 536870912 2024-05-22T23:50:26.389 INFO:journalctl@ceph.mon.c.smithi174.stdout: num_shard_bits : 4 2024-05-22T23:50:26.389 INFO:journalctl@ceph.mon.c.smithi174.stdout: strict_capacity_limit : 0 2024-05-22T23:50:26.389 INFO:journalctl@ceph.mon.c.smithi174.stdout: high_pri_pool_ratio: 0.000 2024-05-22T23:50:26.389 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_compressed: (nil) 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: persistent_cache: (nil) 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_size: 4096 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_size_deviation: 10 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_restart_interval: 16 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_block_restart_interval: 1 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: metadata_block_size: 4096 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: partition_filters: 0 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: use_delta_encoding: 1 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: filter_policy: bloomfilter 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: whole_key_filtering: 1 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: verify_compression: 0 2024-05-22T23:50:26.390 INFO:journalctl@ceph.mon.c.smithi174.stdout: read_amp_bytes_per_bit: 0 2024-05-22T23:50:26.391 INFO:journalctl@ceph.mon.c.smithi174.stdout: format_version: 5 2024-05-22T23:50:26.391 INFO:journalctl@ceph.mon.c.smithi174.stdout: enable_index_compression: 1 2024-05-22T23:50:26.391 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_align: 0 2024-05-22T23:50:26.391 INFO:journalctl@ceph.mon.c.smithi174.stdout: max_auto_readahead_size: 262144 2024-05-22T23:50:26.391 INFO:journalctl@ceph.mon.c.smithi174.stdout: prepopulate_block_cache: 0 2024-05-22T23:50:26.392 INFO:journalctl@ceph.mon.c.smithi174.stdout: initial_auto_readahead_size: 8192 2024-05-22T23:50:26.392 INFO:journalctl@ceph.mon.c.smithi174.stdout: num_file_reads_for_auto_readahead: 2 2024-05-22T23:50:26.392 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.write_buffer_size: 33554432 2024-05-22T23:50:26.392 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_write_buffer_number: 2 2024-05-22T23:50:26.392 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression: NoCompression 2024-05-22T23:50:26.392 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression: Disabled 2024-05-22T23:50:26.392 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.prefix_extractor: nullptr 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.num_levels: 7 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-22T23:50:26.393 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.level: 32767 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.strategy: 0 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.enabled: false 2024-05-22T23:50:26.394 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-22T23:50:26.395 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-22T23:50:26.395 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-22T23:50:26.395 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-22T23:50:26.395 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.target_file_size_base: 67108864 2024-05-22T23:50:26.395 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-22T23:50:26.395 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-22T23:50:26.395 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.arena_block_size: 1048576 2024-05-22T23:50:26.396 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.disable_auto_compactions: 0 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-22T23:50:26.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.table_properties_collectors: 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.inplace_update_support: 0 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.bloom_locality: 0 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.max_successive_merges: 0 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-22T23:50:26.398 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.paranoid_file_checks: 0 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.force_consistency_checks: 1 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.report_bg_io_stats: 0 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.ttl: 2592000 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.enable_blob_files: false 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.min_blob_size: 0 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.blob_file_size: 268435456 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-22T23:50:26.399 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.blob_file_starting_level: 0 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 56e00ef6-c092-4414-a516-4d8834f6a877 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716421825958501, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716421825958915, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716421825, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "56e00ef6-c092-4414-a516-4d8834f6a877", "db_session_id": "8D46AEA2AW4NRX4VCZ0B", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-22T23:50:26.400 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716421825959003, "job": 1, "event": "recovery_finished"} 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563d50c18000 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: DB pointer 0x563d50c02000 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** DB Stats ** 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-22T23:50:26.401 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** Compaction Stats [default] ** 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-05-22T23:50:26.402 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** Compaction Stats [default] ** 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-22T23:50:26.403 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout: Block cache BinnedLRUCache@0x563d50b29090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-22T23:50:26.404 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: using public_addr v2:172.21.15.174:0/0 -> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: starting mon.c rank -1 at public addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] at bind addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(???) e0 preinit fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).mds e1 new map 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout: e1 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout: legacy client fscid: -1 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-05-22T23:50:26.405 INFO:journalctl@ceph.mon.c.smithi174.stdout: No filesystems configured 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-22T23:50:26.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:26.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:26.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:26.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:26.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-22T23:50:26.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:26.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:26.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.407 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Deploying cephadm binary to smithi105 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Added host smithi105 2024-05-22T23:50:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.409 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:26.410 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Deploying cephadm binary to smithi174 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Added host smithi174 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.411 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.412 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:50:26.412 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.412 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.412 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.412 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.412 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi174:/etc/ceph/ceph.conf 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3195254633' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.413 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3195254633' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi103:172.21.15.103=a;smithi105:172.21.15.105=b;smithi174:172.21.15.174=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Saving service mon spec with placement smithi103:172.21.15.103=a;smithi105:172.21.15.105=b;smithi174:172.21.15.174=c;count:3 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: Deploying daemon mon.c on smithi174 2024-05-22T23:50:26.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.415 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:26.415 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:25 smithi174 ceph-mon[36179]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-22T23:50:26.415 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:26 smithi174 bash[36147]: bf01ac9e5176755fce6e0611148b74411c56235d93fa5855dc6a865623eaa459 2024-05-22T23:50:26.415 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:26 smithi174 systemd[1]: Started Ceph mon.c for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:50:27.585 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-22T23:50:27.586 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-22T23:50:27.825 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:50:31.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:31.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:50:31.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: mon.a calling monitor election 2024-05-22T23:50:31.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:31.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: mon.c calling monitor election 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: monmap epoch 2 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: last_changed 2024-05-22T23:50:25.998235+0000 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: min_mon_release 19 (squid) 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: election_strategy: 1 2024-05-22T23:50:31.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: fsmap 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: mgrmap e13: x(active, since 61s) 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: overall HEALTH_OK 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:31.372 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: mon.a calling monitor election 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: mon.c calling monitor election 2024-05-22T23:50:31.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: monmap epoch 2 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: last_changed 2024-05-22T23:50:25.998235+0000 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:50:31.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: min_mon_release 19 (squid) 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: election_strategy: 1 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: fsmap 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: mgrmap e13: x(active, since 61s) 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: overall HEALTH_OK 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:31.585 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:32.192 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.187+0000 7fb553541700 1 -- 172.21.15.174:0/217241673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 msgr2=0x7fb538005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:32.192 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.187+0000 7fb553541700 1 --2- 172.21.15.174:0/217241673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 0x7fb538005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb54c100c80 tx=0x7fb548009e80 comp rx=0 tx=0).stop 2024-05-22T23:50:32.192 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.187+0000 7fb553541700 1 -- 172.21.15.174:0/217241673 shutdown_connections 2024-05-22T23:50:32.192 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.187+0000 7fb553541700 1 --2- 172.21.15.174:0/217241673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 0x7fb538005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:32.193 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.187+0000 7fb553541700 1 -- 172.21.15.174:0/217241673 >> 172.21.15.174:0/217241673 conn(0x7fb54c0fc840 msgr2=0x7fb54c0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:32.193 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.188+0000 7fb553541700 1 -- 172.21.15.174:0/217241673 shutdown_connections 2024-05-22T23:50:32.193 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.188+0000 7fb553541700 1 -- 172.21.15.174:0/217241673 wait complete. 2024-05-22T23:50:32.193 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.188+0000 7fb553541700 1 Processor -- start 2024-05-22T23:50:32.193 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb553541700 1 -- start start 2024-05-22T23:50:32.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb553541700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 0x7fb54c19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:32.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb553541700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 0x7fb54c199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:32.197 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb553541700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c108de0 con 0x7fb5380035c0 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb553541700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb54c108c60 con 0x7fb54c19fdc0 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb550adc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 0x7fb54c199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb550adc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 0x7fb54c199a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:50896/0 (socket says 172.21.15.174:50896) 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb550adc700 1 -- 172.21.15.174:0/3362417652 learned_addr learned my addr 172.21.15.174:0/3362417652 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb550adc700 1 -- 172.21.15.174:0/3362417652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 msgr2=0x7fb54c199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb550adc700 1 -- 172.21.15.174:0/3362417652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 msgr2=0x7fb54c199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb550adc700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 0x7fb54c199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb550adc700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 0x7fb54c199a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.189+0000 7fb5512dd700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 0x7fb54c19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.190+0000 7fb5512dd700 1 -- 172.21.15.174:0/3362417652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 msgr2=0x7fb54c199a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.190+0000 7fb5512dd700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 0x7fb54c199a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:32.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.190+0000 7fb5512dd700 1 -- 172.21.15.174:0/3362417652 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb54c1a0400 con 0x7fb5380035c0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.190+0000 7fb5512dd700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 0x7fb54c19f880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb548002230 tx=0x7fb548017750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.191+0000 7fb5427fc700 1 -- 172.21.15.174:0/3362417652 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb54802a610 con 0x7fb5380035c0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.191+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb54c199f80 con 0x7fb5380035c0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.191+0000 7fb5427fc700 1 -- 172.21.15.174:0/3362417652 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb548025070 con 0x7fb5380035c0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.191+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb54c19a510 con 0x7fb5380035c0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.191+0000 7fb5427fc700 1 -- 172.21.15.174:0/3362417652 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fb5480208f0 con 0x7fb5380035c0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.192+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb54c19a880 con 0x7fb5380035c0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.193+0000 7fb5427fc700 1 -- 172.21.15.174:0/3362417652 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fb54801e070 con 0x7fb5380035c0 2024-05-22T23:50:32.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.193+0000 7fb5427fc700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb530039050 0x7fb53003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:32.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.193+0000 7fb5427fc700 1 -- 172.21.15.174:0/3362417652 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb54801c070 con 0x7fb5380035c0 2024-05-22T23:50:32.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.193+0000 7fb550adc700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb530039050 0x7fb53003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:32.202 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.194+0000 7fb550adc700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb530039050 0x7fb53003b510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb53c009180 tx=0x7fb53c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:32.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.208+0000 7fb5427fc700 1 -- 172.21.15.174:0/3362417652 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb54801ec20 con 0x7fb5380035c0 2024-05-22T23:50:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:32 smithi174 ceph-mon[36179]: Deploying daemon mon.b on smithi105 2024-05-22T23:50:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:32 smithi174 ceph-mon[36179]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:32.491 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.487+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fb54c19af70 con 0x7fb5380035c0 2024-05-22T23:50:32.493 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.488+0000 7fb5427fc700 1 -- 172.21.15.174:0/3362417652 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7fb548020e10 con 0x7fb5380035c0 2024-05-22T23:50:32.493 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:50:32.493 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":2,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","modified":"2024-05-22T23:50:25.998235Z","created":"2024-05-22T23:48:16.842516Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb530039050 msgr2=0x7fb53003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb530039050 0x7fb53003b510 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb53c009180 tx=0x7fb53c008040 comp rx=0 tx=0).stop 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 msgr2=0x7fb54c19f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 0x7fb54c19f880 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb548002230 tx=0x7fb548017750 comp rx=0 tx=0).stop 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 shutdown_connections 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb530039050 0x7fb53003b510 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5380035c0 0x7fb54c19f880 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 --2- 172.21.15.174:0/3362417652 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb54c19fdc0 0x7fb54c199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 >> 172.21.15.174:0/3362417652 conn(0x7fb54c0fc840 msgr2=0x7fb54c06a570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:32.495 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 shutdown_connections 2024-05-22T23:50:32.496 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:32.490+0000 7fb553541700 1 -- 172.21.15.174:0/3362417652 wait complete. 2024-05-22T23:50:32.496 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 2 2024-05-22T23:50:32.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:32 smithi103 ceph-mon[31370]: Deploying daemon mon.b on smithi105 2024-05-22T23:50:32.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:32 smithi103 ceph-mon[31370]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:32.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:33.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:33 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/3362417652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-22T23:50:33.332 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:50:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:50:32.999+0000 7fb451e00700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-22T23:50:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:33 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/3362417652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-22T23:50:34.096 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-22T23:50:34.096 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph mon dump -f json 2024-05-22T23:50:34.341 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:50:34.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:34 smithi174 ceph-mon[36179]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:34.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:34 smithi103 ceph-mon[31370]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:35.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:35 smithi105 systemd[1]: Started Ceph mon.b for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:50:40.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.203+0000 7f81e8553700 1 -- 172.21.15.174:0/3003743369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc0035c0 msgr2=0x7f81cc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:40.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.203+0000 7f81e8553700 1 --2- 172.21.15.174:0/3003743369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc0035c0 0x7f81cc005a50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f81e0101db0 tx=0x7f81dc00fe60 comp rx=0 tx=0).stop 2024-05-22T23:50:40.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.203+0000 7f81e8553700 1 -- 172.21.15.174:0/3003743369 shutdown_connections 2024-05-22T23:50:40.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.203+0000 7f81e8553700 1 --2- 172.21.15.174:0/3003743369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81cc0035c0 0x7f81cc005a50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:40.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.203+0000 7f81e8553700 1 -- 172.21.15.174:0/3003743369 >> 172.21.15.174:0/3003743369 conn(0x7f81e00fd970 msgr2=0x7f81e00ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:40.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.204+0000 7f81e8553700 1 -- 172.21.15.174:0/3003743369 shutdown_connections 2024-05-22T23:50:40.209 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.204+0000 7f81e8553700 1 -- 172.21.15.174:0/3003743369 wait complete. 2024-05-22T23:50:40.211 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.204+0000 7f81e8553700 1 Processor -- start 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e8553700 1 -- start start 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e8553700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 0x7f81e0199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e8553700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 0x7f81e019e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e8553700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e019eb80 0x7f81e019f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e8553700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81e0109ec0 con 0x7f81e019eb80 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e8553700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f81e0109d40 con 0x7f81e019a2b0 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e8553700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81e010a040 con 0x7f81cc0035c0 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e62ef700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 0x7f81e0199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e62ef700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 0x7f81e0199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:40642/0 (socket says 172.21.15.174:40642) 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e62ef700 1 -- 172.21.15.174:0/3368084069 learned_addr learned my addr 172.21.15.174:0/3368084069 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:50:40.212 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e62ef700 1 -- 172.21.15.174:0/3368084069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 msgr2=0x7f81e0199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e62ef700 1 -- 172.21.15.174:0/3368084069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 msgr2=0x7f81e0199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e62ef700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 0x7f81e0199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e62ef700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 0x7f81e0199d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e5aee700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 0x7f81e019e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e6af0700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e019eb80 0x7f81e019f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e5aee700 1 -- 172.21.15.174:0/3368084069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 msgr2=0x7f81e019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e5aee700 1 -- 172.21.15.174:0/3368084069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 msgr2=0x7f81e019e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e5aee700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 0x7f81e019e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.205+0000 7f81e5aee700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 0x7f81e019e640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.206+0000 7f81e6af0700 1 -- 172.21.15.174:0/3368084069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 msgr2=0x7f81e0199d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.206+0000 7f81e6af0700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 0x7f81e0199d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.206+0000 7f81e6af0700 1 -- 172.21.15.174:0/3368084069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 msgr2=0x7f81e019e640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-22T23:50:40.213 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.206+0000 7f81e6af0700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 0x7f81e019e640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:40.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.206+0000 7f81e6af0700 1 -- 172.21.15.174:0/3368084069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e01a68f0 con 0x7f81e019eb80 2024-05-22T23:50:40.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.206+0000 7f81e6af0700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e019eb80 0x7f81e019f030 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f81d800bd50 tx=0x7f81d800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:40.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.207+0000 7f81d77fe700 1 -- 172.21.15.174:0/3368084069 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d8019070 con 0x7f81e019eb80 2024-05-22T23:50:40.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.207+0000 7f81d77fe700 1 -- 172.21.15.174:0/3368084069 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81d800cd10 con 0x7f81e019eb80 2024-05-22T23:50:40.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.207+0000 7f81d77fe700 1 -- 172.21.15.174:0/3368084069 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d80127e0 con 0x7f81e019eb80 2024-05-22T23:50:40.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.207+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e01a6b20 con 0x7f81e019eb80 2024-05-22T23:50:40.214 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.207+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e01a7010 con 0x7f81e019eb80 2024-05-22T23:50:40.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.208+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e00639c0 con 0x7f81e019eb80 2024-05-22T23:50:40.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.208+0000 7f81d77fe700 1 -- 172.21.15.174:0/3368084069 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f81d801c070 con 0x7f81e019eb80 2024-05-22T23:50:40.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.208+0000 7f81d77fe700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81c4039580 0x7f81c403ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:40.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.208+0000 7f81d77fe700 1 -- 172.21.15.174:0/3368084069 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f81d804ba30 con 0x7f81e019eb80 2024-05-22T23:50:40.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.209+0000 7f81e62ef700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81c4039580 0x7f81c403ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:40.215 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.209+0000 7f81e62ef700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81c4039580 0x7f81c403ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f81dc001eb0 tx=0x7f81dc005de0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:40.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.217+0000 7f81d77fe700 1 -- 172.21.15.174:0/3368084069 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f81d8012980 con 0x7f81e019eb80 2024-05-22T23:50:40.504 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.500+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f81e01a7330 con 0x7f81e019eb80 2024-05-22T23:50:40.505 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: mon.c calling monitor election 2024-05-22T23:50:40.505 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:50:40.505 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: mon.a calling monitor election 2024-05-22T23:50:40.505 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.505 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:40.505 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:40.505 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: monmap epoch 3 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: last_changed 2024-05-22T23:50:35.083121+0000 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:50:40.506 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: min_mon_release 19 (squid) 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: election_strategy: 1 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: fsmap 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: mgrmap e13: x(active, since 70s) 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: overall HEALTH_OK 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:40.507 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:40.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.500+0000 7f81d77fe700 1 -- 172.21.15.174:0/3368084069 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1287 (secure 0 0 0) 0x7f81d8014440 con 0x7f81e019eb80 2024-05-22T23:50:40.508 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-05-22T23:50:40.508 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":3,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","modified":"2024-05-22T23:50:35.083121Z","created":"2024-05-22T23:48:16.842516Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.502+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81c4039580 msgr2=0x7f81c403ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.502+0000 7f81e8553700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81c4039580 0x7f81c403ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f81dc001eb0 tx=0x7f81dc005de0 comp rx=0 tx=0).stop 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.502+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e019eb80 msgr2=0x7f81e019f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.502+0000 7f81e8553700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e019eb80 0x7f81e019f030 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f81d800bd50 tx=0x7f81d800be30 comp rx=0 tx=0).stop 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.503+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 shutdown_connections 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.503+0000 7f81e8553700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81c4039580 0x7f81c403ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.503+0000 7f81e8553700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81e019eb80 0x7f81e019f030 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.503+0000 7f81e8553700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81cc0035c0 0x7f81e0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.503+0000 7f81e8553700 1 --2- 172.21.15.174:0/3368084069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81e019a2b0 0x7f81e019e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.503+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 >> 172.21.15.174:0/3368084069 conn(0x7f81e00fd970 msgr2=0x7f81e0103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.503+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 shutdown_connections 2024-05-22T23:50:40.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:40.503+0000 7f81e8553700 1 -- 172.21.15.174:0/3368084069 wait complete. 2024-05-22T23:50:40.510 INFO:teuthology.orchestra.run.smithi174.stderr:dumped monmap epoch 3 2024-05-22T23:50:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: mon.c calling monitor election 2024-05-22T23:50:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:50:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: mon.a calling monitor election 2024-05-22T23:50:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:40.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: monmap epoch 3 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: last_changed 2024-05-22T23:50:35.083121+0000 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: min_mon_release 19 (squid) 2024-05-22T23:50:40.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: election_strategy: 1 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: fsmap 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: mgrmap e13: x(active, since 70s) 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: overall HEALTH_OK 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:40.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:41.205 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-22T23:50:41.206 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph config generate-minimal-conf 2024-05-22T23:50:41.540 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:41.540 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:41.540 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:41.540 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:41 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/3368084069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-22T23:50:41.540 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:41.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:41.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:41.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:41.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:41 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/3368084069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-22T23:50:41.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:41.642 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:50:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mon.c calling monitor election 2024-05-22T23:50:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:50:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mon.a calling monitor election 2024-05-22T23:50:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: monmap epoch 3 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: last_changed 2024-05-22T23:50:35.083121+0000 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: min_mon_release 19 (squid) 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: election_strategy: 1 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: fsmap 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mgrmap e13: x(active, since 70s) 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: overall HEALTH_OK 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:42.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:42.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:42.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:42.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/3368084069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-22T23:50:42.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:43.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: mon.b calling monitor election 2024-05-22T23:50:43.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: mon.b calling monitor election 2024-05-22T23:50:43.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: mon.c calling monitor election 2024-05-22T23:50:43.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: mon.a calling monitor election 2024-05-22T23:50:43.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-22T23:50:43.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: monmap epoch 3 2024-05-22T23:50:43.154 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: last_changed 2024-05-22T23:50:35.083121+0000 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: min_mon_release 19 (squid) 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: election_strategy: 1 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: fsmap 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: mgrmap e13: x(active, since 72s) 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: overall HEALTH_OK 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.155 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.156 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mon.b calling monitor election 2024-05-22T23:50:43.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mon.b calling monitor election 2024-05-22T23:50:43.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mon.c calling monitor election 2024-05-22T23:50:43.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mon.a calling monitor election 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: monmap epoch 3 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: last_changed 2024-05-22T23:50:35.083121+0000 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: min_mon_release 19 (squid) 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: election_strategy: 1 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: fsmap 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: mgrmap e13: x(active, since 72s) 2024-05-22T23:50:43.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: overall HEALTH_OK 2024-05-22T23:50:43.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: mon.b calling monitor election 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: mon.b calling monitor election 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: mon.c calling monitor election 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: mon.a calling monitor election 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: monmap epoch 3 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: last_changed 2024-05-22T23:50:35.083121+0000 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: created 2024-05-22T23:48:16.842516+0000 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: min_mon_release 19 (squid) 2024-05-22T23:50:43.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: election_strategy: 1 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: 0: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.a 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: 1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.c 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: 2: [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon.b 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: fsmap 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: osdmap e4: 0 total, 0 up, 0 in 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: mgrmap e13: x(active, since 72s) 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: overall HEALTH_OK 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:43.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.667+0000 7fc6b2c00700 1 -- 172.21.15.103:0/825414678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac074790 msgr2=0x7fc6ac074b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:43.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.667+0000 7fc6b2c00700 1 --2- 172.21.15.103:0/825414678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac074790 0x7fc6ac074b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009910 tx=0x7fc69c009c20 comp rx=0 tx=0).stop 2024-05-22T23:50:43.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.667+0000 7fc6b2c00700 1 -- 172.21.15.103:0/825414678 shutdown_connections 2024-05-22T23:50:43.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.667+0000 7fc6b2c00700 1 --2- 172.21.15.103:0/825414678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac074790 0x7fc6ac074b70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:43.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.667+0000 7fc6b2c00700 1 -- 172.21.15.103:0/825414678 >> 172.21.15.103:0/825414678 conn(0x7fc6ac0fd850 msgr2=0x7fc6ac0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:43.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.667+0000 7fc6b2c00700 1 -- 172.21.15.103:0/825414678 shutdown_connections 2024-05-22T23:50:43.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.667+0000 7fc6b2c00700 1 -- 172.21.15.103:0/825414678 wait complete. 2024-05-22T23:50:43.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.668+0000 7fc6b2c00700 1 Processor -- start 2024-05-22T23:50:43.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.668+0000 7fc6b2c00700 1 -- start start 2024-05-22T23:50:43.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6b2c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac074790 0x7fc6ac199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:43.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6b2c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac19a290 0x7fc6ac19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:43.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6b2c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ac19eb60 0x7fc6ac19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:43.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6b2c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac10e5a0 con 0x7fc6ac19a290 2024-05-22T23:50:43.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6b2c00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac10e420 con 0x7fc6ac074790 2024-05-22T23:50:43.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6b2c00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ac10e720 con 0x7fc6ac19eb60 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6ab400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac19a290 0x7fc6ac19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6abe00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac074790 0x7fc6ac199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6abe00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac074790 0x7fc6ac199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:32880/0 (socket says 172.21.15.103:32880) 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6abe00700 1 -- 172.21.15.103:0/2440024811 learned_addr learned my addr 172.21.15.103:0/2440024811 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.669+0000 7fc6b0c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ac19eb60 0x7fc6ac19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6abe00700 1 -- 172.21.15.103:0/2440024811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ac19eb60 msgr2=0x7fc6ac19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6abe00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ac19eb60 0x7fc6ac19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6abe00700 1 -- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac19a290 msgr2=0x7fc6ac19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6abe00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac19a290 0x7fc6ac19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6abe00700 1 -- 172.21.15.103:0/2440024811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ac1a68d0 con 0x7fc6ac074790 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6ab400700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac19a290 0x7fc6ac19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:50:43.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6b0c00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ac19eb60 0x7fc6ac19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:50:43.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6abe00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac074790 0x7fc6ac199d50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009fd0 tx=0x7fc69c00bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:43.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6a8c00700 1 -- 172.21.15.103:0/2440024811 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c026070 con 0x7fc6ac074790 2024-05-22T23:50:43.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.670+0000 7fc6a8c00700 1 -- 172.21.15.103:0/2440024811 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc69c005da0 con 0x7fc6ac074790 2024-05-22T23:50:43.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.671+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ac1a6aa0 con 0x7fc6ac074790 2024-05-22T23:50:43.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.671+0000 7fc6a8c00700 1 -- 172.21.15.103:0/2440024811 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69c0038e0 con 0x7fc6ac074790 2024-05-22T23:50:43.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.671+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ac1a6fd0 con 0x7fc6ac074790 2024-05-22T23:50:43.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.672+0000 7fc6a8c00700 1 -- 172.21.15.103:0/2440024811 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fc69c029070 con 0x7fc6ac074790 2024-05-22T23:50:43.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.672+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ac0639c0 con 0x7fc6ac074790 2024-05-22T23:50:43.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.672+0000 7fc6a8c00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc690039580 0x7fc69003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:43.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.672+0000 7fc6a8c00700 1 -- 172.21.15.103:0/2440024811 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc69c04cdd0 con 0x7fc6ac074790 2024-05-22T23:50:43.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.673+0000 7fc6ab400700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc690039580 0x7fc69003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:43.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.673+0000 7fc6ab400700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc690039580 0x7fc69003ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc698009ec0 tx=0x7fc698009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:43.683 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.678+0000 7fc6a8c00700 1 -- 172.21.15.103:0/2440024811 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc69c013070 con 0x7fc6ac074790 2024-05-22T23:50:43.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.902+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fc6ac1a71d0 con 0x7fc6ac074790 2024-05-22T23:50:43.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.903+0000 7fc6a8c00700 1 -- 172.21.15.103:0/2440024811 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7fc69c030020 con 0x7fc6ac074790 2024-05-22T23:50:43.908 INFO:teuthology.orchestra.run.smithi103.stdout:# minimal ceph.conf for 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:43.909 INFO:teuthology.orchestra.run.smithi103.stdout:[global] 2024-05-22T23:50:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: fsid = 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-22T23:50:43.909 INFO:teuthology.orchestra.run.smithi103.stdout: mon_host = [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc690039580 msgr2=0x7fc69003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc690039580 0x7fc69003ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc698009ec0 tx=0x7fc698009210 comp rx=0 tx=0).stop 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac074790 msgr2=0x7fc6ac199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac074790 0x7fc6ac199d50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009fd0 tx=0x7fc69c00bde0 comp rx=0 tx=0).stop 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 shutdown_connections 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac074790 0x7fc6ac199d50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc690039580 0x7fc69003ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6ac19eb60 0x7fc6ac19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 --2- 172.21.15.103:0/2440024811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6ac19a290 0x7fc6ac19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 >> 172.21.15.103:0/2440024811 conn(0x7fc6ac0fd850 msgr2=0x7fc6ac107ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:43.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 shutdown_connections 2024-05-22T23:50:43.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:43.906+0000 7fc6b2c00700 1 -- 172.21.15.103:0/2440024811 wait complete. 2024-05-22T23:50:44.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:44.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:44.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:43 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2440024811' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:44.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:44.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:44.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:43 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2440024811' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-22T23:50:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:50:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:44.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:44.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:44.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:44.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:43 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2440024811' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:44.699 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-22T23:50:44.700 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:50:44.700 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-05-22T23:50:44.777 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:50:44.778 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:44.898 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:50:44.898 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-05-22T23:50:44.926 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:50:44.927 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:44.991 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:50:44.991 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-05-22T23:50:45.020 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:50:45.020 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-22T23:50:45.085 INFO:tasks.cephadm:Adding mgr.x on smithi103 2024-05-22T23:50:45.085 INFO:tasks.cephadm:Adding mgr.y on smithi105 2024-05-22T23:50:45.085 INFO:tasks.cephadm:Adding mgr.z on smithi174 2024-05-22T23:50:45.085 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch apply mgr '3;smithi103=x;smithi105=y;smithi174=z' 2024-05-22T23:50:45.222 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:44 smithi103 ceph-mon[31370]: Reconfiguring mon.a (unknown last config time)... 2024-05-22T23:50:45.222 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:44 smithi103 ceph-mon[31370]: Reconfiguring daemon mon.a on smithi103 2024-05-22T23:50:45.222 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:44 smithi103 ceph-mon[31370]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:45.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:44 smithi105 ceph-mon[36426]: Reconfiguring mon.a (unknown last config time)... 2024-05-22T23:50:45.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:44 smithi105 ceph-mon[36426]: Reconfiguring daemon mon.a on smithi103 2024-05-22T23:50:45.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:44 smithi105 ceph-mon[36426]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:45.352 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:50:45.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:44 smithi174 ceph-mon[36179]: Reconfiguring mon.a (unknown last config time)... 2024-05-22T23:50:45.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:44 smithi174 ceph-mon[36179]: Reconfiguring daemon mon.a on smithi103 2024-05-22T23:50:45.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:44 smithi174 ceph-mon[36179]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:46.302 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.302 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.302 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.302 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:46.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:46.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:46.720 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.715+0000 7fb86de22700 1 -- 172.21.15.174:0/1790168794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb868100dc0 msgr2=0x7fb8681011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:46.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.715+0000 7fb86de22700 1 --2- 172.21.15.174:0/1790168794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb868100dc0 0x7fb8681011e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb858009910 tx=0x7fb858009c20 comp rx=0 tx=0).stop 2024-05-22T23:50:46.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.715+0000 7fb86de22700 1 -- 172.21.15.174:0/1790168794 shutdown_connections 2024-05-22T23:50:46.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.715+0000 7fb86de22700 1 --2- 172.21.15.174:0/1790168794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb868100dc0 0x7fb8681011e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:46.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.715+0000 7fb86de22700 1 -- 172.21.15.174:0/1790168794 >> 172.21.15.174:0/1790168794 conn(0x7fb8680fc360 msgr2=0x7fb8680fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:46.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.715+0000 7fb86de22700 1 -- 172.21.15.174:0/1790168794 shutdown_connections 2024-05-22T23:50:46.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.715+0000 7fb86de22700 1 -- 172.21.15.174:0/1790168794 wait complete. 2024-05-22T23:50:46.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.716+0000 7fb86de22700 1 Processor -- start 2024-05-22T23:50:46.721 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.716+0000 7fb86de22700 1 -- start start 2024-05-22T23:50:46.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.716+0000 7fb86de22700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868100dc0 0x7fb8681976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:46.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.716+0000 7fb86de22700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb868197c20 0x7fb86819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:46.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.716+0000 7fb86de22700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8681980a0 0x7fb86819f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:46.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.716+0000 7fb86de22700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb868106e00 con 0x7fb868197c20 2024-05-22T23:50:46.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.716+0000 7fb86de22700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb868106c80 con 0x7fb868100dc0 2024-05-22T23:50:46.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.716+0000 7fb86de22700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb868106f80 con 0x7fb8681980a0 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8681980a0 0x7fb86819f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8681980a0 0x7fb86819f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:34408/0 (socket says 172.21.15.174:34408) 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 -- 172.21.15.174:0/1852534128 learned_addr learned my addr 172.21.15.174:0/1852534128 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb8677fe700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868100dc0 0x7fb8681976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb866ffd700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb868197c20 0x7fb86819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 -- 172.21.15.174:0/1852534128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868100dc0 msgr2=0x7fb8681976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868100dc0 0x7fb8681976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 -- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb868197c20 msgr2=0x7fb86819cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb868197c20 0x7fb86819cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 -- 172.21.15.174:0/1852534128 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86819faf0 con 0x7fb8681980a0 2024-05-22T23:50:46.724 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb867fff700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8681980a0 0x7fb86819f220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb86000ba50 tx=0x7fb86000bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb864ff9700 1 -- 172.21.15.174:0/1852534128 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb860019070 con 0x7fb8681980a0 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb864ff9700 1 -- 172.21.15.174:0/1852534128 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb860014440 con 0x7fb8681980a0 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86819fde0 con 0x7fb8681980a0 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb864ff9700 1 -- 172.21.15.174:0/1852534128 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8600075b0 con 0x7fb8681980a0 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.717+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8681a0310 con 0x7fb8681980a0 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.718+0000 7fb864ff9700 1 -- 172.21.15.174:0/1852534128 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7fb86001c070 con 0x7fb8681980a0 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.718+0000 7fb864ff9700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb850039580 0x7fb85003ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.718+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8680639c0 con 0x7fb8681980a0 2024-05-22T23:50:46.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.718+0000 7fb864ff9700 1 -- 172.21.15.174:0/1852534128 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb86004b790 con 0x7fb8681980a0 2024-05-22T23:50:46.726 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.719+0000 7fb8677fe700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb850039580 0x7fb85003ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:46.726 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.720+0000 7fb8677fe700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb850039580 0x7fb85003ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb858005200 tx=0x7fb85801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:46.728 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.724+0000 7fb864ff9700 1 -- 172.21.15.174:0/1852534128 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb86000e0a0 con 0x7fb8681980a0 2024-05-22T23:50:46.932 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.928+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi103=x;smithi105=y;smithi174=z", "target": ["mon-mgr", ""]}) -- 0x7fb868062830 con 0x7fb850039580 2024-05-22T23:50:46.947 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.943+0000 7fb864ff9700 1 -- 172.21.15.174:0/1852534128 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb868062830 con 0x7fb850039580 2024-05-22T23:50:46.947 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mgr update... 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb850039580 msgr2=0x7fb85003ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb850039580 0x7fb85003ba40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb858005200 tx=0x7fb85801a040 comp rx=0 tx=0).stop 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8681980a0 msgr2=0x7fb86819f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8681980a0 0x7fb86819f220 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb86000ba50 tx=0x7fb86000bd60 comp rx=0 tx=0).stop 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 shutdown_connections 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb868100dc0 0x7fb8681976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb850039580 0x7fb85003ba40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8681980a0 0x7fb86819f220 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 --2- 172.21.15.174:0/1852534128 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb868197c20 0x7fb86819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 >> 172.21.15.174:0/1852534128 conn(0x7fb8680fc360 msgr2=0x7fb86806a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 shutdown_connections 2024-05-22T23:50:46.950 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:50:46.945+0000 7fb86de22700 1 -- 172.21.15.174:0/1852534128 wait complete. 2024-05-22T23:50:47.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:47 smithi105 ceph-mon[36426]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:47.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:47 smithi105 ceph-mon[36426]: Reconfiguring mon.b (monmap changed)... 2024-05-22T23:50:47.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:47 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:47.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:47 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:47.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:47 smithi105 ceph-mon[36426]: Reconfiguring daemon mon.b on smithi105 2024-05-22T23:50:47.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:47 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:47.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:47 smithi103 ceph-mon[31370]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:47 smithi103 ceph-mon[31370]: Reconfiguring mon.b (monmap changed)... 2024-05-22T23:50:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:47 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:47 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:47 smithi103 ceph-mon[31370]: Reconfiguring daemon mon.b on smithi105 2024-05-22T23:50:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:47 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:47 smithi174 ceph-mon[36179]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:47 smithi174 ceph-mon[36179]: Reconfiguring mon.b (monmap changed)... 2024-05-22T23:50:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:47 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:47 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:47 smithi174 ceph-mon[36179]: Reconfiguring daemon mon.b on smithi105 2024-05-22T23:50:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:47 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:47.546 DEBUG:teuthology.orchestra.run.smithi105:mgr.y> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.y.service 2024-05-22T23:50:47.549 DEBUG:teuthology.orchestra.run.smithi174:mgr.z> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.z.service 2024-05-22T23:50:47.552 INFO:tasks.cephadm:Deploying OSDs... 2024-05-22T23:50:47.552 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:50:47.552 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-05-22T23:50:47.610 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-22T23:50:47.610 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-05-22T23:50:47.659 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-22T23:50:47.660 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:47.660 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 632 Links: 1 2024-05-22T23:50:47.660 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:47.660 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:47.660 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-22 23:49:47.147914795 +0000 2024-05-22T23:50:47.660 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-22 23:44:19.239960734 +0000 2024-05-22T23:50:47.660 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-22 23:44:19.239960734 +0000 2024-05-22T23:50:47.660 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-22 23:44:19.239960734 +0000 2024-05-22T23:50:47.660 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-22T23:50:47.734 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-05-22T23:50:47.734 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-05-22T23:50:47.734 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000363254 s, 1.4 MB/s 2024-05-22T23:50:47.738 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-22T23:50:47.795 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 653 Links: 1 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-22 23:49:47.214913531 +0000 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-22 23:44:19.835949264 +0000 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-22 23:44:19.835949264 +0000 2024-05-22T23:50:47.846 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-22 23:44:19.835949264 +0000 2024-05-22T23:50:47.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-22T23:50:47.926 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-05-22T23:50:47.926 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-05-22T23:50:47.926 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.00042588 s, 1.2 MB/s 2024-05-22T23:50:47.930 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-22T23:50:47.989 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-05-22T23:50:48.044 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-22T23:50:48.044 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 669 Links: 1 2024-05-22T23:50:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-22 23:49:47.269912494 +0000 2024-05-22T23:50:48.044 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-22 23:44:20.407938256 +0000 2024-05-22T23:50:48.045 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-22 23:44:20.407938256 +0000 2024-05-22T23:50:48.045 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-22 23:44:20.407938256 +0000 2024-05-22T23:50:48.045 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-22T23:50:48.121 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-05-22T23:50:48.121 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-05-22T23:50:48.121 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000344068 s, 1.5 MB/s 2024-05-22T23:50:48.124 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-22T23:50:48.180 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-05-22T23:50:48.230 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-22T23:50:48.231 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:48.231 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-05-22T23:50:48.231 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:48.231 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:48.231 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-22 23:49:47.325911438 +0000 2024-05-22T23:50:48.231 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-22 23:44:20.982927190 +0000 2024-05-22T23:50:48.231 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-22 23:44:20.982927190 +0000 2024-05-22T23:50:48.231 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-22 23:44:20.982927190 +0000 2024-05-22T23:50:48.231 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-22T23:50:48.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:48 smithi103 ceph-mon[31370]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi103=x;smithi105=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:48.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:48 smithi103 ceph-mon[31370]: Saving service mgr spec with placement smithi103=x;smithi105=y;smithi174=z;count:3 2024-05-22T23:50:48.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:48.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:48.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:48.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:48.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:48.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:48 smithi105 ceph-mon[36426]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi103=x;smithi105=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:48.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:48 smithi105 ceph-mon[36426]: Saving service mgr spec with placement smithi103=x;smithi105=y;smithi174=z;count:3 2024-05-22T23:50:48.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:48.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:48.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:48.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:48.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:48.311 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-05-22T23:50:48.311 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-05-22T23:50:48.311 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000365228 s, 1.4 MB/s 2024-05-22T23:50:48.313 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-22T23:50:48.335 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:48 smithi174 ceph-mon[36179]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi103=x;smithi105=y;smithi174=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:48.335 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:48 smithi174 ceph-mon[36179]: Saving service mgr spec with placement smithi103=x;smithi105=y;smithi174=z;count:3 2024-05-22T23:50:48.335 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:48.335 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:48.335 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-22T23:50:48.335 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-22T23:50:48.335 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:48.369 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-05-22T23:50:48.369 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-05-22T23:50:48.388 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-22T23:50:48.388 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-05-22T23:50:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-22T23:50:48.446 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:48.446 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-22T23:50:48.446 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:48.446 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:48.447 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-22 23:50:34.524993584 +0000 2024-05-22T23:50:48.447 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-22 23:42:51.412540644 +0000 2024-05-22T23:50:48.447 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-22 23:42:51.412540644 +0000 2024-05-22T23:50:48.447 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-22 23:42:51.412540644 +0000 2024-05-22T23:50:48.453 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-22T23:50:48.513 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-22T23:50:48.513 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-22T23:50:48.513 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000304958 s, 1.7 MB/s 2024-05-22T23:50:48.514 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-22T23:50:48.573 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-05-22T23:50:48.629 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-22T23:50:48.630 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:48.630 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-05-22T23:50:48.630 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:48.630 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:48.630 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-22 23:50:34.524993584 +0000 2024-05-22T23:50:48.631 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-22 23:42:51.748533721 +0000 2024-05-22T23:50:48.631 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-22 23:42:51.748533721 +0000 2024-05-22T23:50:48.631 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-22 23:42:51.748533721 +0000 2024-05-22T23:50:48.631 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-22T23:50:48.696 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-22T23:50:48.696 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-22T23:50:48.696 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000209518 s, 2.4 MB/s 2024-05-22T23:50:48.696 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-22T23:50:48.753 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-05-22T23:50:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-22T23:50:48.809 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:48.809 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-22T23:50:48.810 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:48.810 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:48.810 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-22 23:50:34.524993584 +0000 2024-05-22T23:50:48.810 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-22 23:42:52.082526840 +0000 2024-05-22T23:50:48.810 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-22 23:42:52.082526840 +0000 2024-05-22T23:50:48.810 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-22 23:42:52.082526840 +0000 2024-05-22T23:50:48.810 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-22T23:50:48.874 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-22T23:50:48.874 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-22T23:50:48.874 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000187173 s, 2.7 MB/s 2024-05-22T23:50:48.875 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-22T23:50:48.932 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-05-22T23:50:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-22T23:50:48.988 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:48.989 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-22T23:50:48.989 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:48.989 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:48.989 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-05-22 23:50:34.525993564 +0000 2024-05-22T23:50:48.989 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-05-22 23:42:52.404520207 +0000 2024-05-22T23:50:48.989 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-05-22 23:42:52.404520207 +0000 2024-05-22T23:50:48.989 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-05-22 23:42:52.404520207 +0000 2024-05-22T23:50:48.989 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-22T23:50:49.050 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-05-22T23:50:49.050 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-05-22T23:50:49.050 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.00019214 s, 2.7 MB/s 2024-05-22T23:50:49.051 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-22T23:50:49.107 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-05-22T23:50:49.108 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-05-22T23:50:49.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:49 smithi174 ceph-mon[36179]: Reconfiguring mon.c (monmap changed)... 2024-05-22T23:50:49.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:49 smithi174 ceph-mon[36179]: Reconfiguring daemon mon.c on smithi174 2024-05-22T23:50:49.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:49 smithi174 ceph-mon[36179]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:49.125 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-22T23:50:49.125 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-05-22T23:50:49.182 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-22T23:50:49.182 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:49.182 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-05-22T23:50:49.182 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:49.182 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:49.182 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-05-22 23:50:24.712315254 +0000 2024-05-22T23:50:49.182 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-05-22 23:43:02.717900563 +0000 2024-05-22T23:50:49.183 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-05-22 23:43:02.717900563 +0000 2024-05-22T23:50:49.183 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-05-22 23:43:02.717900563 +0000 2024-05-22T23:50:49.183 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-22T23:50:49.246 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-05-22T23:50:49.246 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-05-22T23:50:49.246 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000246407 s, 2.1 MB/s 2024-05-22T23:50:49.247 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-22T23:50:49.302 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-05-22T23:50:49.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:49 smithi105 ceph-mon[36426]: Reconfiguring mon.c (monmap changed)... 2024-05-22T23:50:49.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:49 smithi105 ceph-mon[36426]: Reconfiguring daemon mon.c on smithi174 2024-05-22T23:50:49.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:49 smithi105 ceph-mon[36426]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:49 smithi103 ceph-mon[31370]: Reconfiguring mon.c (monmap changed)... 2024-05-22T23:50:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:49 smithi103 ceph-mon[31370]: Reconfiguring daemon mon.c on smithi174 2024-05-22T23:50:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:49 smithi103 ceph-mon[31370]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:49.358 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-22T23:50:49.359 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:49.359 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-22T23:50:49.359 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:49.359 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:49.359 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-05-22 23:50:24.713315242 +0000 2024-05-22T23:50:49.359 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-05-22 23:43:03.074895838 +0000 2024-05-22T23:50:49.359 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-05-22 23:43:03.074895838 +0000 2024-05-22T23:50:49.359 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-05-22 23:43:03.074895838 +0000 2024-05-22T23:50:49.360 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-22T23:50:49.420 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-05-22T23:50:49.421 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-05-22T23:50:49.421 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000189071 s, 2.7 MB/s 2024-05-22T23:50:49.422 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-22T23:50:49.478 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-05-22T23:50:49.533 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-22T23:50:49.533 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:49.533 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-22T23:50:49.534 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:49.534 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:49.534 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-05-22 23:50:24.713315242 +0000 2024-05-22T23:50:49.534 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-05-22 23:43:03.391891642 +0000 2024-05-22T23:50:49.534 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-05-22 23:43:03.391891642 +0000 2024-05-22T23:50:49.534 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-05-22 23:43:03.391891642 +0000 2024-05-22T23:50:49.535 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-22T23:50:49.596 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-05-22T23:50:49.596 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-05-22T23:50:49.596 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000221886 s, 2.3 MB/s 2024-05-22T23:50:49.597 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-22T23:50:49.654 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-05-22T23:50:49.709 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-22T23:50:49.709 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-22T23:50:49.709 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-22T23:50:49.710 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-22T23:50:49.710 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-05-22T23:50:49.710 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-05-22 23:50:24.713315242 +0000 2024-05-22T23:50:49.710 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-05-22 23:43:03.739887035 +0000 2024-05-22T23:50:49.710 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-05-22 23:43:03.739887035 +0000 2024-05-22T23:50:49.710 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-05-22 23:43:03.739887035 +0000 2024-05-22T23:50:49.710 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-22T23:50:49.772 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-05-22T23:50:49.773 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-05-22T23:50:49.773 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000204435 s, 2.5 MB/s 2024-05-22T23:50:49.774 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-22T23:50:49.830 INFO:tasks.cephadm:Deploying osd.0 on smithi103 with /dev/vg_nvme/lv_4... 2024-05-22T23:50:49.830 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-22T23:50:50.260 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:50:50.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:50.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:50.501 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:50.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:50.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-22T23:50:50.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:50.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:50.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: Deploying daemon mgr.z on smithi174 2024-05-22T23:50:50.502 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:50 smithi103 ceph-mon[31370]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-22T23:50:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:50.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:50.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: Deploying daemon mgr.z on smithi174 2024-05-22T23:50:50.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:50 smithi105 ceph-mon[36426]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:50.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:50.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-22T23:50:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: Deploying daemon mgr.z on smithi174 2024-05-22T23:50:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:50 smithi174 ceph-mon[36179]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:51.341 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 systemd[1]: Starting Ceph mgr.z for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:50:51.619 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 podman[38023]: 2024-05-22 23:50:51.336997363 +0000 UTC m=+0.094819905 container create 9e6c9e4eb7105b88b06289536178d9a9fe45d1f6d54730763bd641745702bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-05-22T23:50:51.619 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 podman[38023]: 2024-05-22 23:50:51.259325662 +0000 UTC m=+0.017148219 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:50:51.619 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 podman[38023]: 2024-05-22 23:50:51.522530462 +0000 UTC m=+0.280353004 container init 9e6c9e4eb7105b88b06289536178d9a9fe45d1f6d54730763bd641745702bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762) 2024-05-22T23:50:51.619 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 podman[38023]: 2024-05-22 23:50:51.525324162 +0000 UTC m=+0.283146705 container start 9e6c9e4eb7105b88b06289536178d9a9fe45d1f6d54730763bd641745702bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-05-22T23:50:51.620 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 ceph-mgr[38043]: -- 172.21.15.174:0/1067017543 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c58742a5a0 con 0x55c5873fe800 2024-05-22T23:50:51.620 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 bash[38023]: 9e6c9e4eb7105b88b06289536178d9a9fe45d1f6d54730763bd641745702bd66 2024-05-22T23:50:51.620 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 systemd[1]: Started Ceph mgr.z for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:50:51.970 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:51.678+0000 7fe6f83c2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-22T23:50:51.970 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:51.862+0000 7fe6f83c2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:50:51.970 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:51.964+0000 7fe6f83c2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-22T23:50:52.619 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:52.272+0000 7fe6f83c2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-22T23:50:52.619 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:52.357+0000 7fe6f83c2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-22T23:50:53.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:53.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-22T23:50:53.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:53.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:53.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: Deploying daemon mgr.y on smithi105 2024-05-22T23:50:53.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:52 smithi105 ceph-mon[36426]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:53.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:53.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-22T23:50:53.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:53.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:53.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: Deploying daemon mgr.y on smithi105 2024-05-22T23:50:53.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:52 smithi103 ceph-mon[31370]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:53.091 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-22T23:50:53.092 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-22T23:50:53.092 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-05-22T23:50:53.092 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-05-22T23:50:53.092 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0368403 s, 285 MB/s 2024-05-22T23:50:53.092 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-05-22T23:50:53.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:53.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:53.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-22T23:50:53.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:53.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:53.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: Deploying daemon mgr.y on smithi105 2024-05-22T23:50:53.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:52 smithi174 ceph-mon[36179]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:53.810 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:53 smithi105 systemd[1]: Starting Ceph mgr.y for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:50:53.865 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-05-22T23:50:53.869 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:53.548+0000 7fe6f83c2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-22T23:50:53.869 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:53.724+0000 7fe6f83c2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-22T23:50:54.122 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:53 smithi105 podman[37787]: 2024-05-22 23:50:53.760298264 +0000 UTC m=+0.017196588 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:50:54.122 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:53 smithi105 podman[37787]: 2024-05-22 23:50:53.891207395 +0000 UTC m=+0.148105706 container create 4c2a1c3611d0c422576116421d8454c60a8d988d4938776202ed15849742c9c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-22T23:50:54.123 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 podman[37787]: 2024-05-22 23:50:54.118367805 +0000 UTC m=+0.375266109 container init 4c2a1c3611d0c422576116421d8454c60a8d988d4938776202ed15849742c9c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-22T23:50:54.184 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:53.894+0000 7fe6f83c2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-22T23:50:54.184 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:54.179+0000 7fe6f83c2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-22T23:50:54.308 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:50:54.480 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 podman[37787]: 2024-05-22 23:50:54.121206863 +0000 UTC m=+0.378105178 container start 4c2a1c3611d0c422576116421d8454c60a8d988d4938776202ed15849742c9c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-05-22T23:50:54.480 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 ceph-mgr[37806]: -- 172.21.15.105:0/793112597 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562ed60fa5a0 con 0x562ed60cf000 2024-05-22T23:50:54.481 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 bash[37787]: 4c2a1c3611d0c422576116421d8454c60a8d988d4938776202ed15849742c9c1 2024-05-22T23:50:54.481 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 systemd[1]: Started Ceph mgr.y for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:50:54.481 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:54.289+0000 7fbc824fd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-22T23:50:54.481 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:54.475+0000 7fbc824fd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:50:54.481 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:54.271+0000 7fe6f83c2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-22T23:50:54.810 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:54.578+0000 7fbc824fd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-22T23:50:54.869 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:54.476+0000 7fe6f83c2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-22T23:50:54.869 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:54.676+0000 7fe6f83c2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:50:55.310 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:54.902+0000 7fbc824fd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-22T23:50:55.310 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:54.990+0000 7fbc824fd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-22T23:50:55.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:55 smithi105 ceph-mon[36426]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:55.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:55 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:55 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:55 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:55 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:55 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:55.322 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:55.317+0000 7fe6f83c2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-22T23:50:55.322 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:55 smithi174 ceph-mon[36179]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:55.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:55 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:55 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:55 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:55 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:55 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:55 smithi103 ceph-mon[31370]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:55 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:55 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:55 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:55 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:55 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:56.056 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:55.750+0000 7fe6f83c2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-22T23:50:56.057 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:55.861+0000 7fe6f83c2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-22T23:50:56.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:56 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:56 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:56 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:56 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:56 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:56 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:56.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:56 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:56.190 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:56 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.310 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:56.242+0000 7fbc824fd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-22T23:50:56.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:56 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:56 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:56 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:56 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:56 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:56 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:56.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:56 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:56.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:56 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.358+0000 7f92f8c00700 1 -- 172.21.15.103:0/217579574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f410e9b0 msgr2=0x7f92f4110da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:56.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.358+0000 7f92f8c00700 1 --2- 172.21.15.103:0/217579574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f410e9b0 0x7f92f4110da0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f92ec00b0a0 tx=0x7f92ec00b3b0 comp rx=0 tx=0).stop 2024-05-22T23:50:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.359+0000 7f92f8c00700 1 -- 172.21.15.103:0/217579574 shutdown_connections 2024-05-22T23:50:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.359+0000 7f92f8c00700 1 --2- 172.21.15.103:0/217579574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f410e9b0 0x7f92f4110da0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.359+0000 7f92f8c00700 1 --2- 172.21.15.103:0/217579574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92f4076390 0x7f92f40747f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.359+0000 7f92f8c00700 1 --2- 172.21.15.103:0/217579574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f4074dc0 0x7f92f4075240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.359+0000 7f92f8c00700 1 -- 172.21.15.103:0/217579574 >> 172.21.15.103:0/217579574 conn(0x7f92f40fd930 msgr2=0x7f92f40ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:50:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.359+0000 7f92f8c00700 1 -- 172.21.15.103:0/217579574 shutdown_connections 2024-05-22T23:50:56.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.359+0000 7f92f8c00700 1 -- 172.21.15.103:0/217579574 wait complete. 2024-05-22T23:50:56.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.360+0000 7f92f8c00700 1 Processor -- start 2024-05-22T23:50:56.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.360+0000 7f92f8c00700 1 -- start start 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f8c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f4074dc0 0x7f92f419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f8c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92f4076390 0x7f92f419f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f8c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f410e9b0 0x7f92f41a6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f8c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f92f4114600 con 0x7f92f4076390 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f8c00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f92f4114480 con 0x7f92f4074dc0 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f8c00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f92f4114780 con 0x7f92f410e9b0 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f4074dc0 0x7f92f419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f4074dc0 0x7f92f419eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:41984/0 (socket says 172.21.15.103:41984) 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1e00700 1 -- 172.21.15.103:0/2919982382 learned_addr learned my addr 172.21.15.103:0/2919982382 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1400700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92f4076390 0x7f92f419f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f2800700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f410e9b0 0x7f92f41a6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:56.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1e00700 1 -- 172.21.15.103:0/2919982382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f410e9b0 msgr2=0x7f92f41a6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1e00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f410e9b0 0x7f92f41a6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1e00700 1 -- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92f4076390 msgr2=0x7f92f419f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1e00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92f4076390 0x7f92f419f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f1e00700 1 -- 172.21.15.103:0/2919982382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f41a6800 con 0x7f92f4074dc0 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.361+0000 7f92f2800700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f410e9b0 0x7f92f41a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.362+0000 7f92f1400700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92f4076390 0x7f92f419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.362+0000 7f92f1e00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f4074dc0 0x7f92f419eba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d950 tx=0x7f92e400dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.362+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e4016070 con 0x7f92f4074dc0 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.362+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92e400de30 con 0x7f92f4074dc0 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.362+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e4005420 con 0x7f92f4074dc0 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.362+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f41a6af0 con 0x7f92f4074dc0 2024-05-22T23:50:56.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.362+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f41a6fe0 con 0x7f92f4074dc0 2024-05-22T23:50:56.370 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:56.063+0000 7fe6f83c2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-22T23:50:56.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:56 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:56 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:56 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:56 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:56 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:56 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:56.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:56 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:56.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:56 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.363+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f4069050 con 0x7f92f4074dc0 2024-05-22T23:50:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.363+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) ==== 46344+0+0 (secure 0 0 0) 0x7f92e4004750 con 0x7f92f4074dc0 2024-05-22T23:50:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.364+0000 7f92eaa00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92d8039330 0x7f92d803b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:50:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.364+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f92e404bd00 con 0x7f92f4074dc0 2024-05-22T23:50:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.364+0000 7f92f1400700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92d8039330 0x7f92d803b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:50:56.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.365+0000 7f92f1400700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92d8039330 0x7f92d803b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92f40760c0 tx=0x7f92e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:50:56.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.369+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f92e401f070 con 0x7f92f4074dc0 2024-05-22T23:50:56.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:56.607+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f92f40627a0 con 0x7f92d8039330 2024-05-22T23:50:56.810 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:56.411+0000 7fbc824fd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-22T23:50:56.810 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:56.579+0000 7fbc824fd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-22T23:50:56.869 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:56.587+0000 7fe6f83c2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-22T23:50:56.869 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:56.679+0000 7fe6f83c2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-22T23:50:57.166 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:56.866+0000 7fbc824fd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-22T23:50:57.166 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:56.958+0000 7fbc824fd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-22T23:50:57.166 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:57.160+0000 7fbc824fd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-22T23:50:57.166 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:57 smithi105 ceph-mon[36426]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:57.166 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:57 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:57.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:57 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:57.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:57 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:57.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:57 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:50:57.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:57 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:50:57.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:57 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:57.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:57 smithi103 ceph-mon[31370]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:57.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:57 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:57.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:57 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:57.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:57 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:57.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:57 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:50:57.219 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:57 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:50:57.220 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:57 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:57.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:57 smithi174 ceph-mon[36179]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:57.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:57 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-22T23:50:57.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:57 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-22T23:50:57.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:57 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:57.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:57 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:50:57.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:57 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:50:57.272 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:57 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:57.273 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:56.922+0000 7fe6f83c2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-22T23:50:57.273 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:57.056+0000 7fe6f83c2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-22T23:50:57.273 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:57.152+0000 7fe6f83c2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-22T23:50:57.273 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:57.267+0000 7fe6f83c2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-22T23:50:57.560 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:57.352+0000 7fbc824fd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-22T23:50:57.869 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:57.580+0000 7fe6f83c2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-22T23:50:58.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:58 smithi105 ceph-mon[36426]: Reconfiguring mgr.x (unknown last config time)... 2024-05-22T23:50:58.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:58 smithi105 ceph-mon[36426]: Reconfiguring daemon mgr.x on smithi103 2024-05-22T23:50:58.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:58 smithi105 ceph-mon[36426]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:58.311 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:57.957+0000 7fbc824fd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-22T23:50:58.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:58 smithi103 ceph-mon[31370]: Reconfiguring mgr.x (unknown last config time)... 2024-05-22T23:50:58.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:58 smithi103 ceph-mon[31370]: Reconfiguring daemon mgr.x on smithi103 2024-05-22T23:50:58.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:58 smithi103 ceph-mon[31370]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:58.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:58 smithi174 ceph-mon[36179]: Reconfiguring mgr.x (unknown last config time)... 2024-05-22T23:50:58.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:58 smithi174 ceph-mon[36179]: Reconfiguring daemon mgr.x on smithi103 2024-05-22T23:50:58.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:58 smithi174 ceph-mon[36179]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:50:58.369 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:58.177+0000 7fe6f83c2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-22T23:50:58.687 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:58.384+0000 7fbc824fd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-22T23:50:58.687 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:58.483+0000 7fbc824fd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-22T23:50:58.688 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:58.682+0000 7fbc824fd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-22T23:50:58.869 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 22 23:50:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-22T23:50:58.393+0000 7fe6f83c2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-22T23:50:59.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:50:59.250+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 14) ==== 92061+0+0 (secure 0 0 0) 0x7f92e4010460 con 0x7f92f4074dc0 2024-05-22T23:50:59.259 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:59.259 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.259 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.259 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:59.259 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:59.259 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:59.259 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.259 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-22T23:50:59.260 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-22T23:50:59.260 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: Standby manager daemon z started 2024-05-22T23:50:59.260 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-22T23:50:59.260 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:50:59 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-22T23:50:59.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:59.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:59.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:59.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:59.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-22T23:50:59.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-22T23:50:59.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: Standby manager daemon z started 2024-05-22T23:50:59.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-22T23:50:59.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:50:59 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-22T23:50:59.541 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:59.202+0000 7fbc824fd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-22T23:50:59.541 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:59.294+0000 7fbc824fd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-22T23:50:59.542 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:50:59.542 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.542 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.542 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:50:59.543 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:50:59.543 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:50:59.543 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:50:59.543 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-22T23:50:59.543 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-22T23:50:59.544 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: Standby manager daemon z started 2024-05-22T23:50:59.544 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-22T23:50:59.544 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:50:59 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.174:0/3983704744' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-22T23:50:59.815 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:59.535+0000 7fbc824fd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-22T23:50:59.816 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:59.668+0000 7fbc824fd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-22T23:50:59.816 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:59.762+0000 7fbc824fd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-22T23:51:00.181 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:50:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:50:59.877+0000 7fbc824fd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-22T23:51:00.181 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:51:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:51:00.175+0000 7fbc824fd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-22T23:51:00.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:00 smithi105 ceph-mon[36426]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-22T23:51:00.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:00 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-22T23:51:00.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:00 smithi105 ceph-mon[36426]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:00.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:00 smithi103 ceph-mon[31370]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-22T23:51:00.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:00 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-22T23:51:00.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:00 smithi103 ceph-mon[31370]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:00.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:00 smithi174 ceph-mon[36179]: mgrmap e14: x(active, since 89s), standbys: z 2024-05-22T23:51:00.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:00 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-22T23:51:00.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:00 smithi174 ceph-mon[36179]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:01.060 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:51:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:51:00.770+0000 7fbc824fd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-22T23:51:01.060 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 22 23:51:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-22T23:51:00.985+0000 7fbc824fd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-22T23:51:01.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:01.278+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mon.2 v2:172.21.15.105:3300/0 8 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f92e4004a40 con 0x7f92f4074dc0 2024-05-22T23:51:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: Standby manager daemon y started 2024-05-22T23:51:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-22T23:51:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-22T23:51:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-22T23:51:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-22T23:51:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3371579937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]: dispatch 2024-05-22T23:51:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]: dispatch 2024-05-22T23:51:01.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]': finished 2024-05-22T23:51:01.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: osdmap e5: 1 total, 0 up, 1 in 2024-05-22T23:51:01.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:01 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: Standby manager daemon y started 2024-05-22T23:51:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-22T23:51:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-22T23:51:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-22T23:51:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-22T23:51:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3371579937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]: dispatch 2024-05-22T23:51:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]: dispatch 2024-05-22T23:51:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]': finished 2024-05-22T23:51:01.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: osdmap e5: 1 total, 0 up, 1 in 2024-05-22T23:51:01.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:01 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:01.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: Standby manager daemon y started 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.105:0/3993229975' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3371579937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]: dispatch 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]: dispatch 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2976753d-f629-42a4-ab40-e6c3bee4d852"}]': finished 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: osdmap e5: 1 total, 0 up, 1 in 2024-05-22T23:51:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:01 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:02 smithi105 ceph-mon[36426]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-22T23:51:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:02 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-22T23:51:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:02 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2714332611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:02 smithi105 ceph-mon[36426]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:02 smithi103 ceph-mon[31370]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-22T23:51:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:02 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-22T23:51:02.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:02 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2714332611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:02.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:02 smithi103 ceph-mon[31370]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:02.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:02 smithi174 ceph-mon[36179]: mgrmap e15: x(active, since 91s), standbys: z, y 2024-05-22T23:51:02.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:02 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-22T23:51:02.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:02 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2714332611' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:02.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:02 smithi174 ceph-mon[36179]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:05.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:05 smithi105 ceph-mon[36426]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:05.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:05 smithi103 ceph-mon[31370]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:05.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:05 smithi174 ceph-mon[36179]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:07.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:07 smithi105 ceph-mon[36426]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:07 smithi103 ceph-mon[31370]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:07.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:07 smithi174 ceph-mon[36179]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:09 smithi103 ceph-mon[31370]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:09.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:09 smithi105 ceph-mon[36426]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:09.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:09 smithi174 ceph-mon[36179]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:11.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:11 smithi105 ceph-mon[36426]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:11.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:11 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-22T23:51:11.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:11 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:11.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:11 smithi103 ceph-mon[31370]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:11.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-22T23:51:11.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:11.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:11 smithi174 ceph-mon[36179]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:11.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:11 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-22T23:51:11.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:11 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:12.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:12 smithi103 ceph-mon[31370]: Deploying daemon osd.0 on smithi103 2024-05-22T23:51:12.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:12 smithi174 ceph-mon[36179]: Deploying daemon osd.0 on smithi103 2024-05-22T23:51:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:12 smithi105 ceph-mon[36426]: Deploying daemon osd.0 on smithi103 2024-05-22T23:51:13.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:13 smithi103 ceph-mon[31370]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:13.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:13 smithi174 ceph-mon[36179]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:13.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:13 smithi105 ceph-mon[36426]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:14.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:14 smithi174 ceph-mon[36179]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:14.492 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:14 smithi103 ceph-mon[31370]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:14 smithi105 ceph-mon[36426]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:17.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:17 smithi105 ceph-mon[36426]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:17.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:17 smithi103 ceph-mon[31370]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:17.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:17 smithi174 ceph-mon[36179]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:19.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:19 smithi105 ceph-mon[36426]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:19.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:19.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:19.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:19.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:19 smithi103 ceph-mon[31370]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:19.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:19.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:19.316 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:19.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:19 smithi174 ceph-mon[36179]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:19.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:19.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:19.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:21.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:21 smithi105 ceph-mon[36426]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:21.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:21 smithi103 ceph-mon[31370]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:21.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:21 smithi174 ceph-mon[36179]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:23.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:23.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:23.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:22 smithi105 ceph-mon[36426]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:23.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:22 smithi105 ceph-mon[36426]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-22T23:51:23.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:23.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:23.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:22 smithi103 ceph-mon[31370]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:23.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:22 smithi103 ceph-mon[31370]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-22T23:51:23.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:23.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:23.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:23.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:22 smithi174 ceph-mon[36179]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:23.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:22 smithi174 ceph-mon[36179]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-22T23:51:23.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.408+0000 7f92eaa00700 1 -- 172.21.15.103:0/2919982382 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f92f40627a0 con 0x7f92d8039330 2024-05-22T23:51:23.413 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 0 on host 'smithi103' 2024-05-22T23:51:23.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92d8039330 msgr2=0x7f92d803b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:23.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92d8039330 0x7f92d803b7f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f92f40760c0 tx=0x7f92e0009260 comp rx=0 tx=0).stop 2024-05-22T23:51:23.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f4074dc0 msgr2=0x7f92f419eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:23.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f4074dc0 0x7f92f419eba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d950 tx=0x7f92e400dd10 comp rx=0 tx=0).stop 2024-05-22T23:51:23.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 shutdown_connections 2024-05-22T23:51:23.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f92f4074dc0 0x7f92f419eba0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:23.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92d8039330 0x7f92d803b7f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:23.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92f410e9b0 0x7f92f41a6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:23.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 --2- 172.21.15.103:0/2919982382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f92f4076390 0x7f92f419f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:23.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 >> 172.21.15.103:0/2919982382 conn(0x7f92f40fd930 msgr2=0x7f92f4101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:51:23.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.412+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 shutdown_connections 2024-05-22T23:51:23.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:51:23.413+0000 7f92f8c00700 1 -- 172.21.15.103:0/2919982382 wait complete. 2024-05-22T23:51:24.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:23 smithi105 ceph-mon[36426]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-22T23:51:24.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:23 smithi105 ceph-mon[36426]: osdmap e6: 1 total, 0 up, 1 in 2024-05-22T23:51:24.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:23 smithi105 ceph-mon[36426]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:51:24.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:23 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:24.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:23 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:24.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:23 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:24.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:23 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:24.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:23 smithi103 ceph-mon[31370]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-22T23:51:24.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:23 smithi103 ceph-mon[31370]: osdmap e6: 1 total, 0 up, 1 in 2024-05-22T23:51:24.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:23 smithi103 ceph-mon[31370]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:51:24.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:23 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:24.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:23 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:24.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:23 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:24.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:23 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:24.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:23 smithi174 ceph-mon[36179]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-22T23:51:24.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:23 smithi174 ceph-mon[36179]: osdmap e6: 1 total, 0 up, 1 in 2024-05-22T23:51:24.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:23 smithi174 ceph-mon[36179]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:51:24.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:23 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:24.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:23 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:24.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:23 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:24.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:23 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:24.228 DEBUG:teuthology.orchestra.run.smithi103:osd.0> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.0.service 2024-05-22T23:51:24.232 INFO:tasks.cephadm:Deploying osd.1 on smithi105 with /dev/vg_nvme/lv_4... 2024-05-22T23:51:24.232 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-22T23:51:24.463 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-22T23:51:24.902 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:24 smithi105 ceph-mon[36426]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:51:24.903 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:24 smithi105 ceph-mon[36426]: osdmap e7: 1 total, 0 up, 1 in 2024-05-22T23:51:24.903 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:24 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:24.903 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:24 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:24.903 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:24 smithi105 ceph-mon[36426]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:25.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:24 smithi103 ceph-mon[31370]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:51:25.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:24 smithi103 ceph-mon[31370]: osdmap e7: 1 total, 0 up, 1 in 2024-05-22T23:51:25.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:24 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:25.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:24 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:25.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:24 smithi103 ceph-mon[31370]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:25.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:24 smithi174 ceph-mon[36179]: from='osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:51:25.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:24 smithi174 ceph-mon[36179]: osdmap e7: 1 total, 0 up, 1 in 2024-05-22T23:51:25.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:24 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:25.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:24 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:25.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:24 smithi174 ceph-mon[36179]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-22T23:51:26.024 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:25 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:51:26.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:25 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:51:26.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:25 smithi103 ceph-mon[31370]: osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] boot 2024-05-22T23:51:26.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:25 smithi103 ceph-mon[31370]: osdmap e8: 1 total, 1 up, 1 in 2024-05-22T23:51:26.025 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:26.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:25 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:51:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:25 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:51:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:25 smithi105 ceph-mon[36426]: osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] boot 2024-05-22T23:51:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:25 smithi105 ceph-mon[36426]: osdmap e8: 1 total, 1 up, 1 in 2024-05-22T23:51:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:26.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:25 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:51:26.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:25 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:51:26.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:25 smithi174 ceph-mon[36179]: osd.0 [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] boot 2024-05-22T23:51:26.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:25 smithi174 ceph-mon[36179]: osdmap e8: 1 total, 1 up, 1 in 2024-05-22T23:51:26.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-22T23:51:26.297 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-22T23:51:26.298 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-22T23:51:26.298 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-22T23:51:26.298 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-22T23:51:26.298 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113358 s, 925 MB/s 2024-05-22T23:51:26.298 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-22T23:51:26.886 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:26 smithi105 ceph-mon[36426]: osdmap e9: 1 total, 1 up, 1 in 2024-05-22T23:51:26.887 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:26 smithi105 ceph-mon[36426]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:26.913 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2024-05-22T23:51:27.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:26 smithi174 ceph-mon[36179]: osdmap e9: 1 total, 1 up, 1 in 2024-05-22T23:51:27.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:26 smithi174 ceph-mon[36179]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:27.142 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-22T23:51:27.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:26 smithi103 ceph-mon[31370]: osdmap e9: 1 total, 1 up, 1 in 2024-05-22T23:51:27.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:26 smithi103 ceph-mon[31370]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:28.575 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/115359913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 msgr2=0x7f4c44101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:28.575 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/115359913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 0x7f4c44101700 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4c340099c0 tx=0x7f4c34009cd0 comp rx=0 tx=0).stop 2024-05-22T23:51:28.575 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/115359913 shutdown_connections 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/115359913 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 0x7f4c44101700 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/115359913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c44102e80 0x7f4c44109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/115359913 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c441024e0 0x7f4c44102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/115359913 >> 172.21.15.105:0/115359913 conn(0x7f4c440fc8c0 msgr2=0x7f4c440fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/115359913 shutdown_connections 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.567+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/115359913 wait complete. 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c4a8dc700 1 Processor -- start 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c4a8dc700 1 -- start start 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c4a8dc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 0x7f4c441975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c4a8dc700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441024e0 0x7f4c44197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c4a8dc700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c44102e80 0x7f4c4419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:51:28.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c4a8dc700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410cfa0 con 0x7f4c441024e0 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c4a8dc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410ce20 con 0x7f4c441012e0 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c4a8dc700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4410d120 con 0x7f4c44102e80 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.568+0000 7f4c43fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 0x7f4c441975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c43fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 0x7f4c441975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49396/0 (socket says 172.21.15.105:49396) 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c43fff700 1 -- 172.21.15.105:0/4210173398 learned_addr learned my addr 172.21.15.105:0/4210173398 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c48e79700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c44102e80 0x7f4c4419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c43fff700 1 -- 172.21.15.105:0/4210173398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c44102e80 msgr2=0x7f4c4419eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c43fff700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c44102e80 0x7f4c4419eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c43fff700 1 -- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441024e0 msgr2=0x7f4c44197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c43fff700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441024e0 0x7f4c44197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c43fff700 1 -- 172.21.15.105:0/4210173398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4419f210 con 0x7f4c441012e0 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c43fff700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 0x7f4c441975b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4c340098e0 tx=0x7f4c34009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:51:28.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c417fa700 1 -- 172.21.15.105:0/4210173398 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c34027070 con 0x7f4c441012e0 2024-05-22T23:51:28.578 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4419f4a0 con 0x7f4c441012e0 2024-05-22T23:51:28.578 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c417fa700 1 -- 172.21.15.105:0/4210173398 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c34004150 con 0x7f4c441012e0 2024-05-22T23:51:28.578 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c417fa700 1 -- 172.21.15.105:0/4210173398 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c34022e60 con 0x7f4c441012e0 2024-05-22T23:51:28.578 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.569+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4419f970 con 0x7f4c441012e0 2024-05-22T23:51:28.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.570+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c44069050 con 0x7f4c441012e0 2024-05-22T23:51:28.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.571+0000 7f4c417fa700 1 -- 172.21.15.105:0/4210173398 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f4c3402a070 con 0x7f4c441012e0 2024-05-22T23:51:28.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.572+0000 7f4c417fa700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4c2c0a2b90 0x7f4c2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:51:28.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.572+0000 7f4c417fa700 1 -- 172.21.15.105:0/4210173398 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f4c340ce250 con 0x7f4c441012e0 2024-05-22T23:51:28.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.572+0000 7f4c437fe700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4c2c0a2b90 0x7f4c2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:51:28.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.574+0000 7f4c437fe700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4c2c0a2b90 0x7f4c2c0a5050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4c38005950 tx=0x7f4c380058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:51:28.582 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.577+0000 7f4c417fa700 1 -- 172.21.15.105:0/4210173398 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4c34013070 con 0x7f4c441012e0 2024-05-22T23:51:28.783 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:28.778+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4c440627a0 con 0x7f4c2c0a2b90 2024-05-22T23:51:29.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:29 smithi105 ceph-mon[36426]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:29.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:51:29.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:51:29.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:29.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:29 smithi103 ceph-mon[31370]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:29.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:51:29.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:51:29.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:29.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:29 smithi174 ceph-mon[36179]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:29.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:51:29.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:51:29.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:30.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:51:30.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi103 2024-05-22T23:51:30.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:51:30.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi103 to 17089M 2024-05-22T23:51:30.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:30.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:30.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi103 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi103 to 17089M 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:30.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:51:30.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi103 2024-05-22T23:51:30.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:51:30.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi103 to 17089M 2024-05-22T23:51:30.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:30.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:30.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:30.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:31.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:31 smithi105 ceph-mon[36426]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:31 smithi103 ceph-mon[31370]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:31.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:31 smithi174 ceph-mon[36179]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:32.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:32 smithi105 ceph-mon[36426]: from='client.? 172.21.15.105:0/1847500760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]: dispatch 2024-05-22T23:51:32.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:32 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]: dispatch 2024-05-22T23:51:32.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:32 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]': finished 2024-05-22T23:51:32.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:32 smithi105 ceph-mon[36426]: osdmap e10: 2 total, 1 up, 2 in 2024-05-22T23:51:32.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:32.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:32 smithi105 ceph-mon[36426]: from='client.? 172.21.15.105:0/2612729345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:32.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:32 smithi103 ceph-mon[31370]: from='client.? 172.21.15.105:0/1847500760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]: dispatch 2024-05-22T23:51:32.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:32 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]: dispatch 2024-05-22T23:51:32.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:32 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]': finished 2024-05-22T23:51:32.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:32 smithi103 ceph-mon[31370]: osdmap e10: 2 total, 1 up, 2 in 2024-05-22T23:51:32.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:32.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:32 smithi103 ceph-mon[31370]: from='client.? 172.21.15.105:0/2612729345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:32 smithi174 ceph-mon[36179]: from='client.? 172.21.15.105:0/1847500760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]: dispatch 2024-05-22T23:51:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:32 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]: dispatch 2024-05-22T23:51:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:32 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c"}]': finished 2024-05-22T23:51:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:32 smithi174 ceph-mon[36179]: osdmap e10: 2 total, 1 up, 2 in 2024-05-22T23:51:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:32 smithi174 ceph-mon[36179]: from='client.? 172.21.15.105:0/2612729345' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:33.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:33 smithi105 ceph-mon[36426]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:33.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:33 smithi103 ceph-mon[31370]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:33 smithi174 ceph-mon[36179]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:35.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:35 smithi105 ceph-mon[36426]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:35.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:35 smithi103 ceph-mon[31370]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:35 smithi174 ceph-mon[36179]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:37.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:37 smithi105 ceph-mon[36426]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:37.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:37 smithi103 ceph-mon[31370]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:37.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:37 smithi174 ceph-mon[36179]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:39.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:39 smithi105 ceph-mon[36426]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:39.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-22T23:51:39.200 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:39.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:39 smithi103 ceph-mon[31370]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:39.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-22T23:51:39.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:39.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:39 smithi174 ceph-mon[36179]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:39.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-22T23:51:39.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:40.178 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:40 smithi105 ceph-mon[36426]: Deploying daemon osd.1 on smithi105 2024-05-22T23:51:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:40 smithi174 ceph-mon[36179]: Deploying daemon osd.1 on smithi105 2024-05-22T23:51:40.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:40 smithi103 ceph-mon[31370]: Deploying daemon osd.1 on smithi105 2024-05-22T23:51:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:41 smithi105 ceph-mon[36426]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:41 smithi174 ceph-mon[36179]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:41.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:41 smithi103 ceph-mon[31370]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:42.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:42 smithi105 ceph-mon[36426]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:42 smithi174 ceph-mon[36179]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:42.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:42 smithi103 ceph-mon[31370]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:44.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:44.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:44.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:44.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:44 smithi105 ceph-mon[36426]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:44.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:44.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:44.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:44.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:44 smithi103 ceph-mon[31370]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:44.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:44.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:44.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:44.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:44 smithi174 ceph-mon[36179]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:47.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:47.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:47.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:46 smithi174 ceph-mon[36179]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:47.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:47.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:47.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.212 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:46 smithi105 ceph-mon[36426]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:47.266 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.261+0000 7f4c417fa700 1 -- 172.21.15.105:0/4210173398 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4c440627a0 con 0x7f4c2c0a2b90 2024-05-22T23:51:47.267 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 1 on host 'smithi105' 2024-05-22T23:51:47.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4c2c0a2b90 msgr2=0x7f4c2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:47.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4c2c0a2b90 0x7f4c2c0a5050 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4c38005950 tx=0x7f4c380058e0 comp rx=0 tx=0).stop 2024-05-22T23:51:47.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 msgr2=0x7f4c441975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:47.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 0x7f4c441975b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4c340098e0 tx=0x7f4c34009580 comp rx=0 tx=0).stop 2024-05-22T23:51:47.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 shutdown_connections 2024-05-22T23:51:47.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c441012e0 0x7f4c441975b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:47.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4c2c0a2b90 0x7f4c2c0a5050 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:47.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c44102e80 0x7f4c4419eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:47.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 --2- 172.21.15.105:0/4210173398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4c441024e0 0x7f4c44197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:47.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 >> 172.21.15.105:0/4210173398 conn(0x7f4c440fc8c0 msgr2=0x7f4c441057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:51:47.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 shutdown_connections 2024-05-22T23:51:47.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:51:47.264+0000 7f4c4a8dc700 1 -- 172.21.15.105:0/4210173398 wait complete. 2024-05-22T23:51:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:47.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:46 smithi103 ceph-mon[31370]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:47.926 DEBUG:teuthology.orchestra.run.smithi105:osd.1> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.1.service 2024-05-22T23:51:47.928 INFO:tasks.cephadm:Deploying osd.2 on smithi174 with /dev/vg_nvme/lv_4... 2024-05-22T23:51:47.929 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-22T23:51:48.169 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:51:48.558 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:48.559 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:48.559 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:48.559 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:48 smithi174 ceph-mon[36179]: from='osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-22T23:51:48.559 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:48 smithi174 ceph-mon[36179]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-22T23:51:48.559 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:48 smithi174 ceph-mon[36179]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:48 smithi105 ceph-mon[36426]: from='osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-22T23:51:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:48 smithi105 ceph-mon[36426]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-22T23:51:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:48 smithi105 ceph-mon[36426]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:48.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:51:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:48 smithi103 ceph-mon[31370]: from='osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-22T23:51:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:48 smithi103 ceph-mon[31370]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-22T23:51:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:48 smithi103 ceph-mon[31370]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:49 smithi105 ceph-mon[36426]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-22T23:51:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:49 smithi105 ceph-mon[36426]: osdmap e11: 2 total, 1 up, 2 in 2024-05-22T23:51:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:49 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:49 smithi105 ceph-mon[36426]: from='osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:51:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:49 smithi105 ceph-mon[36426]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:51:49.561 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:51:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1[40991]: 2024-05-22T23:51:49.275+0000 7f7549020700 -1 osd.1 0 waiting for initial osdmap 2024-05-22T23:51:49.561 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:51:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1[40991]: 2024-05-22T23:51:49.280+0000 7f7544e13700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:51:49.562 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:49 smithi174 ceph-mon[36179]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-22T23:51:49.562 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:49 smithi174 ceph-mon[36179]: osdmap e11: 2 total, 1 up, 2 in 2024-05-22T23:51:49.562 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:49 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:49.562 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:49 smithi174 ceph-mon[36179]: from='osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:51:49.562 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:49 smithi174 ceph-mon[36179]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:51:49.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:49 smithi103 ceph-mon[31370]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-22T23:51:49.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:49 smithi103 ceph-mon[31370]: osdmap e11: 2 total, 1 up, 2 in 2024-05-22T23:51:49.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:49.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:49 smithi103 ceph-mon[31370]: from='osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:51:49.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:49 smithi103 ceph-mon[31370]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:51:50.346 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-22T23:51:50.346 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-22T23:51:50.346 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-05-22T23:51:50.346 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-05-22T23:51:50.346 INFO:teuthology.orchestra.run.smithi174.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112654 s, 931 MB/s 2024-05-22T23:51:50.346 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-05-22T23:51:50.520 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:50 smithi105 ceph-mon[36426]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:51:50.521 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:50 smithi105 ceph-mon[36426]: osdmap e12: 2 total, 1 up, 2 in 2024-05-22T23:51:50.521 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:50.521 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:50.521 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:50 smithi105 ceph-mon[36426]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:50.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:50 smithi103 ceph-mon[31370]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:51:50.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:50 smithi103 ceph-mon[31370]: osdmap e12: 2 total, 1 up, 2 in 2024-05-22T23:51:50.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:50.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:50.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:50 smithi103 ceph-mon[31370]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:50.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:50 smithi174 ceph-mon[36179]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:51:50.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:50 smithi174 ceph-mon[36179]: osdmap e12: 2 total, 1 up, 2 in 2024-05-22T23:51:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:50.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:50 smithi174 ceph-mon[36179]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-22T23:51:50.941 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_4 2024-05-22T23:51:51.173 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] boot 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: osdmap e13: 2 total, 2 up, 2 in 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi105 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:51:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-22T23:51:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:51 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:51:51.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:51:51.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:51.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] boot 2024-05-22T23:51:51.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: osdmap e13: 2 total, 2 up, 2 in 2024-05-22T23:51:51.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:51.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi105 2024-05-22T23:51:51.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:51:51.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-22T23:51:51.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:51.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:51.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:51 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: osd.1 [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] boot 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: osdmap e13: 2 total, 2 up, 2 in 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi105 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi105 to 17088M 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:51.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:51.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:51:51.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:51 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:51:52.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:52 smithi103 ceph-mon[31370]: osdmap e14: 2 total, 2 up, 2 in 2024-05-22T23:51:52.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:52 smithi103 ceph-mon[31370]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:52.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:52 smithi174 ceph-mon[36179]: osdmap e14: 2 total, 2 up, 2 in 2024-05-22T23:51:52.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:52 smithi174 ceph-mon[36179]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:52.662 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.654+0000 7f378eab6700 1 -- 172.21.15.174:0/812498466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3788102e70 msgr2=0x7f3788109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:52.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.654+0000 7f378eab6700 1 --2- 172.21.15.174:0/812498466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3788102e70 0x7f3788109720 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f378000b0a0 tx=0x7f378000b3b0 comp rx=0 tx=0).stop 2024-05-22T23:51:52.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 -- 172.21.15.174:0/812498466 shutdown_connections 2024-05-22T23:51:52.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 --2- 172.21.15.174:0/812498466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3788102e70 0x7f3788109720 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:52.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 --2- 172.21.15.174:0/812498466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37881012d0 0x7f37881016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:52.663 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 --2- 172.21.15.174:0/812498466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37881024d0 0x7f3788102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:52.664 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 -- 172.21.15.174:0/812498466 >> 172.21.15.174:0/812498466 conn(0x7f37880fc890 msgr2=0x7f37880fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:51:52.664 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 -- 172.21.15.174:0/812498466 shutdown_connections 2024-05-22T23:51:52.664 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 -- 172.21.15.174:0/812498466 wait complete. 2024-05-22T23:51:52.664 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 Processor -- start 2024-05-22T23:51:52.664 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.655+0000 7f378eab6700 1 -- start start 2024-05-22T23:51:52.664 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378eab6700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37881012d0 0x7f3788197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:51:52.664 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378eab6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37881024d0 0x7f3788197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:51:52.664 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378eab6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3788102e70 0x7f378819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:51:52.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378eab6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f378810cf30 con 0x7f3788102e70 2024-05-22T23:51:52.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378eab6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f378810cdb0 con 0x7f37881024d0 2024-05-22T23:51:52.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378eab6700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f378810d0b0 con 0x7f37881012d0 2024-05-22T23:51:52.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37881012d0 0x7f3788197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:51:52.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37881012d0 0x7f3788197540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:60702/0 (socket says 172.21.15.174:60702) 2024-05-22T23:51:52.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 -- 172.21.15.174:0/1302289365 learned_addr learned my addr 172.21.15.174:0/1302289365 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:51:52.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f3787fff700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37881024d0 0x7f3788197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:51:52.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 -- 172.21.15.174:0/1302289365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37881024d0 msgr2=0x7f3788197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:52.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37881024d0 0x7f3788197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:52.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 -- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3788102e70 msgr2=0x7f378819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:51:52.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3788102e70 0x7f378819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:51:52.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 -- 172.21.15.174:0/1302289365 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f378819f1a0 con 0x7f37881012d0 2024-05-22T23:51:52.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378c852700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37881012d0 0x7f3788197540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f377c00b530 tx=0x7f377c00b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:51:52.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f3785ffb700 1 -- 172.21.15.174:0/1302289365 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c015020 con 0x7f37881012d0 2024-05-22T23:51:52.666 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f3785ffb700 1 -- 172.21.15.174:0/1302289365 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f377c011d10 con 0x7f37881012d0 2024-05-22T23:51:52.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f378819f490 con 0x7f37881012d0 2024-05-22T23:51:52.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.656+0000 7f3785ffb700 1 -- 172.21.15.174:0/1302289365 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c00f2e0 con 0x7f37881012d0 2024-05-22T23:51:52.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.657+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f378819f9d0 con 0x7f37881012d0 2024-05-22T23:51:52.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.657+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3788069050 con 0x7f37881012d0 2024-05-22T23:51:52.669 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.658+0000 7f3785ffb700 1 -- 172.21.15.174:0/1302289365 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f377c018020 con 0x7f37881012d0 2024-05-22T23:51:52.669 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.659+0000 7f3785ffb700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37780a2b90 0x7f37780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:51:52.669 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.659+0000 7f3785ffb700 1 -- 172.21.15.174:0/1302289365 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f377c0ccf20 con 0x7f37881012d0 2024-05-22T23:51:52.669 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.660+0000 7f3787fff700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37780a2b90 0x7f37780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:51:52.669 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.660+0000 7f3787fff700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37780a2b90 0x7f37780a5050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3774007c60 tx=0x7f37740073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:51:52.669 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.664+0000 7f3785ffb700 1 -- 172.21.15.174:0/1302289365 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f377c01f020 con 0x7f37881012d0 2024-05-22T23:51:52.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:52 smithi105 ceph-mon[36426]: osdmap e14: 2 total, 2 up, 2 in 2024-05-22T23:51:52.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:52 smithi105 ceph-mon[36426]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:52.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:51:52.864+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f37880627a0 con 0x7f37780a2b90 2024-05-22T23:51:53.577 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:53 smithi174 ceph-mon[36179]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:51:53.577 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:51:53.577 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:51:53.577 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:53.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:53 smithi103 ceph-mon[31370]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:51:53.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:51:53.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:51:53.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:53.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:53 smithi105 ceph-mon[36426]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:51:53.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:51:53.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:51:53.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:51:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:54 smithi174 ceph-mon[36179]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:54.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:54 smithi103 ceph-mon[31370]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:54.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:54 smithi105 ceph-mon[36426]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:55.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:55 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/3143935653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]: dispatch 2024-05-22T23:51:55.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:55 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]: dispatch 2024-05-22T23:51:55.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:55 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]': finished 2024-05-22T23:51:55.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:55 smithi174 ceph-mon[36179]: osdmap e15: 3 total, 2 up, 3 in 2024-05-22T23:51:55.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:55 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:51:55.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:55 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/3143935653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]: dispatch 2024-05-22T23:51:55.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:55 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]: dispatch 2024-05-22T23:51:55.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:55 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]': finished 2024-05-22T23:51:55.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:55 smithi105 ceph-mon[36426]: osdmap e15: 3 total, 2 up, 3 in 2024-05-22T23:51:55.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:55 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:51:55.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:55 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/3143935653' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]: dispatch 2024-05-22T23:51:55.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:55 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]: dispatch 2024-05-22T23:51:55.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:55 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63567395-a0a0-48f9-9f29-d65443b5c4ce"}]': finished 2024-05-22T23:51:55.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:55 smithi103 ceph-mon[31370]: osdmap e15: 3 total, 2 up, 3 in 2024-05-22T23:51:55.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:55 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:51:56.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:56 smithi105 ceph-mon[36426]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:56.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:56 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/3582724024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:56.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:56 smithi103 ceph-mon[31370]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:56.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:56 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/3582724024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:56.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:56 smithi174 ceph-mon[36179]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:56.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:56 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/3582724024' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:51:59.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:51:59 smithi105 ceph-mon[36426]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:59.324 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:51:59 smithi174 ceph-mon[36179]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:51:59.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:51:59 smithi103 ceph-mon[31370]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:01.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:01 smithi103 ceph-mon[31370]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:01.343 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:01 smithi174 ceph-mon[36179]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:01 smithi105 ceph-mon[36426]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:03.331 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:03 smithi174 ceph-mon[36179]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:03.331 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:03 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-22T23:52:03.331 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:03 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:03.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:03 smithi103 ceph-mon[31370]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:03.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:03 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-22T23:52:03.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:03 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:03 smithi105 ceph-mon[36426]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:03 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-22T23:52:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:03 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:04.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:04 smithi103 ceph-mon[31370]: Deploying daemon osd.2 on smithi174 2024-05-22T23:52:04.345 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:04 smithi174 ceph-mon[36179]: Deploying daemon osd.2 on smithi174 2024-05-22T23:52:04.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:04 smithi105 ceph-mon[36426]: Deploying daemon osd.2 on smithi174 2024-05-22T23:52:05.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:05 smithi105 ceph-mon[36426]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:05.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:05 smithi103 ceph-mon[31370]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:05.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:05 smithi174 ceph-mon[36179]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:07.323 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:07 smithi174 ceph-mon[36179]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:07.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:07 smithi105 ceph-mon[36426]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:07.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:07 smithi103 ceph-mon[31370]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:09.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:09.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:09.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:09.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:08 smithi105 ceph-mon[36426]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:09.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:09.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:08 smithi103 ceph-mon[31370]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:09.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:09.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:09.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:09.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:08 smithi174 ceph-mon[36179]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:11.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:11.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:11.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.140 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:10 smithi174 ceph-mon[36179]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:11.198 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.193+0000 7f3785ffb700 1 -- 172.21.15.174:0/1302289365 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f37880627a0 con 0x7f37780a2b90 2024-05-22T23:52:11.199 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 2 on host 'smithi174' 2024-05-22T23:52:11.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37780a2b90 msgr2=0x7f37780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:11.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37780a2b90 0x7f37780a5050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3774007c60 tx=0x7f37740073d0 comp rx=0 tx=0).stop 2024-05-22T23:52:11.200 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37881012d0 msgr2=0x7f3788197540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37881012d0 0x7f3788197540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f377c00b530 tx=0x7f377c00b840 comp rx=0 tx=0).stop 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 shutdown_connections 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37780a2b90 0x7f37780a5050 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3788102e70 0x7f378819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37881012d0 0x7f3788197540 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 --2- 172.21.15.174:0/1302289365 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37881024d0 0x7f3788197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.195+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 >> 172.21.15.174:0/1302289365 conn(0x7f37880fc890 msgr2=0x7f3788105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.196+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 shutdown_connections 2024-05-22T23:52:11.201 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:52:11.196+0000 7f378eab6700 1 -- 172.21.15.174:0/1302289365 wait complete. 2024-05-22T23:52:11.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:11.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:11.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:10 smithi105 ceph-mon[36426]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:10 smithi103 ceph-mon[31370]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:11.802 DEBUG:teuthology.orchestra.run.smithi174:osd.2> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.2.service 2024-05-22T23:52:11.804 INFO:tasks.cephadm:Deploying osd.3 on smithi103 with /dev/vg_nvme/lv_3... 2024-05-22T23:52:11.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-22T23:52:12.260 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:52:12.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:12.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:12.442 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:12.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:12 smithi103 ceph-mon[31370]: from='osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-22T23:52:12.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:12 smithi103 ceph-mon[31370]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-22T23:52:12.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:12 smithi103 ceph-mon[31370]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:12 smithi105 ceph-mon[36426]: from='osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-22T23:52:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:12 smithi105 ceph-mon[36426]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-22T23:52:12.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:12 smithi105 ceph-mon[36426]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:12.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:12.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:12.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:12.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:12 smithi174 ceph-mon[36179]: from='osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-22T23:52:12.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:12 smithi174 ceph-mon[36179]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-22T23:52:12.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:12 smithi174 ceph-mon[36179]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:13.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:13 smithi105 ceph-mon[36426]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-22T23:52:13.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:13 smithi105 ceph-mon[36426]: osdmap e16: 3 total, 2 up, 3 in 2024-05-22T23:52:13.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:13 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:13.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:13 smithi105 ceph-mon[36426]: from='osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:52:13.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:13 smithi105 ceph-mon[36426]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:52:13.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:13 smithi103 ceph-mon[31370]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-22T23:52:13.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:13 smithi103 ceph-mon[31370]: osdmap e16: 3 total, 2 up, 3 in 2024-05-22T23:52:13.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:13 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:13.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:13 smithi103 ceph-mon[31370]: from='osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:52:13.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:13 smithi103 ceph-mon[31370]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:52:13.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:13 smithi174 ceph-mon[36179]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-22T23:52:13.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:13 smithi174 ceph-mon[36179]: osdmap e16: 3 total, 2 up, 3 in 2024-05-22T23:52:13.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:13 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:13.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:13 smithi174 ceph-mon[36179]: from='osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:52:13.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:13 smithi174 ceph-mon[36179]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:52:13.622 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:13 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2[41286]: 2024-05-22T23:52:13.197+0000 7f1dbf0d1700 -1 osd.2 0 waiting for initial osdmap 2024-05-22T23:52:13.622 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:13 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2[41286]: 2024-05-22T23:52:13.203+0000 7f1dbaec4700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:52:14.348 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:14 smithi174 ceph-mon[36179]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:52:14.348 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:14 smithi174 ceph-mon[36179]: osdmap e17: 3 total, 2 up, 3 in 2024-05-22T23:52:14.348 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:14.348 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:14 smithi174 ceph-mon[36179]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:14.348 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:14 smithi105 ceph-mon[36426]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:52:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:14 smithi105 ceph-mon[36426]: osdmap e17: 3 total, 2 up, 3 in 2024-05-22T23:52:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:14 smithi105 ceph-mon[36426]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:14 smithi103 ceph-mon[31370]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:52:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:14 smithi103 ceph-mon[31370]: osdmap e17: 3 total, 2 up, 3 in 2024-05-22T23:52:14.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:14.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:14 smithi103 ceph-mon[31370]: pgmap v89: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-22T23:52:14.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:15.095 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-22T23:52:15.096 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-22T23:52:15.096 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-05-22T23:52:15.096 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-05-22T23:52:15.096 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0314973 s, 333 MB/s 2024-05-22T23:52:15.096 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] boot 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: osdmap e18: 3 total, 3 up, 3 in 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi174 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:15.469 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi174 to 17088M 2024-05-22T23:52:15.469 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.469 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:15.469 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:15.469 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:52:15.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:52:15.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] boot 2024-05-22T23:52:15.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: osdmap e18: 3 total, 3 up, 3 in 2024-05-22T23:52:15.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi174 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi174 to 17088M 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: osd.2 [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] boot 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: osdmap e18: 3 total, 3 up, 3 in 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi174 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi174 to 17088M 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:15.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:15.904 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-05-22T23:52:16.325 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:52:16.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:16 smithi103 ceph-mon[31370]: osdmap e19: 3 total, 3 up, 3 in 2024-05-22T23:52:16.729 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:16 smithi103 ceph-mon[31370]: pgmap v92: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:17.015 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:16 smithi174 ceph-mon[36179]: osdmap e19: 3 total, 3 up, 3 in 2024-05-22T23:52:17.015 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:16 smithi174 ceph-mon[36179]: pgmap v92: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:17.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:16 smithi105 ceph-mon[36426]: osdmap e19: 3 total, 3 up, 3 in 2024-05-22T23:52:17.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:16 smithi105 ceph-mon[36426]: pgmap v92: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:18.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-22T23:52:18.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-22T23:52:18.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:17 smithi105 ceph-mon[36426]: osdmap e20: 3 total, 3 up, 3 in 2024-05-22T23:52:18.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-22T23:52:18.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-22T23:52:18.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-22T23:52:18.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:17 smithi103 ceph-mon[31370]: osdmap e20: 3 total, 3 up, 3 in 2024-05-22T23:52:18.076 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-22T23:52:18.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-22T23:52:18.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-22T23:52:18.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:17 smithi174 ceph-mon[36179]: osdmap e20: 3 total, 3 up, 3 in 2024-05-22T23:52:18.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-22T23:52:18.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 -- 172.21.15.103:0/492558985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f581c108150 msgr2=0x7f581c10a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:18.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 --2- 172.21.15.103:0/492558985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f581c108150 0x7f581c10a540 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5814009f90 tx=0x7f581400b330 comp rx=0 tx=0).stop 2024-05-22T23:52:18.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 -- 172.21.15.103:0/492558985 shutdown_connections 2024-05-22T23:52:18.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 --2- 172.21.15.103:0/492558985 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f581c102ef0 0x7f581c1052e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:18.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 --2- 172.21.15.103:0/492558985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f581c108150 0x7f581c10a540 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:18.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 --2- 172.21.15.103:0/492558985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f581c105820 0x7f581c107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:18.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 -- 172.21.15.103:0/492558985 >> 172.21.15.103:0/492558985 conn(0x7f581c0fc860 msgr2=0x7f581c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:52:18.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 -- 172.21.15.103:0/492558985 shutdown_connections 2024-05-22T23:52:18.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.526+0000 7f5823400700 1 -- 172.21.15.103:0/492558985 wait complete. 2024-05-22T23:52:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.527+0000 7f5823400700 1 Processor -- start 2024-05-22T23:52:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.527+0000 7f5823400700 1 -- start start 2024-05-22T23:52:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5823400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f581c102ef0 0x7f581c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:52:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5823400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f581c105820 0x7f581c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:52:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5823400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f581c108150 0x7f581c1a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5823400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f581c10dfd0 con 0x7f581c108150 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5823400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f581c10de50 con 0x7f581c105820 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5823400700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f581c10e150 con 0x7f581c102ef0 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5820a00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f581c102ef0 0x7f581c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5820a00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f581c102ef0 0x7f581c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:53374/0 (socket says 172.21.15.103:53374) 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5820a00700 1 -- 172.21.15.103:0/1681387212 learned_addr learned my addr 172.21.15.103:0/1681387212 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5821400700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f581c108150 0x7f581c1a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f581be00700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f581c105820 0x7f581c19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.528+0000 7f5820a00700 1 -- 172.21.15.103:0/1681387212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f581c105820 msgr2=0x7f581c19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5820a00700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f581c105820 0x7f581c19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5820a00700 1 -- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f581c108150 msgr2=0x7f581c1a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5820a00700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f581c108150 0x7f581c1a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:18.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5820a00700 1 -- 172.21.15.103:0/1681387212 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f581c1a23f0 con 0x7f581c102ef0 2024-05-22T23:52:18.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5820a00700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f581c102ef0 0x7f581c19a790 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f581000e9d0 tx=0x7f581000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:52:18.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5819600700 1 -- 172.21.15.103:0/1681387212 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581000cbd0 con 0x7f581c102ef0 2024-05-22T23:52:18.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5819600700 1 -- 172.21.15.103:0/1681387212 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f581000ee70 con 0x7f581c102ef0 2024-05-22T23:52:18.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5819600700 1 -- 172.21.15.103:0/1681387212 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5810018660 con 0x7f581c102ef0 2024-05-22T23:52:18.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f581c1a26e0 con 0x7f581c102ef0 2024-05-22T23:52:18.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.529+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f581c1a2bd0 con 0x7f581c102ef0 2024-05-22T23:52:18.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.531+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f581c069050 con 0x7f581c102ef0 2024-05-22T23:52:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.532+0000 7f5819600700 1 -- 172.21.15.103:0/1681387212 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 15) ==== 137778+0+0 (secure 0 0 0) 0x7f5810010430 con 0x7f581c102ef0 2024-05-22T23:52:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.534+0000 7f5819600700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58040a2c10 0x7f58040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:52:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.534+0000 7f5819600700 1 -- 172.21.15.103:0/1681387212 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f58100cdd60 con 0x7f581c102ef0 2024-05-22T23:52:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.534+0000 7f581be00700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58040a2c10 0x7f58040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:52:18.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.534+0000 7f581be00700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58040a2c10 0x7f58040a50d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5808009750 tx=0x7f5808009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:52:18.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.538+0000 7f5819600700 1 -- 172.21.15.103:0/1681387212 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f58100d3020 con 0x7f581c102ef0 2024-05-22T23:52:18.745 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 22 23:52:18 smithi103 sudo[50651]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-22T23:52:18.745 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 22 23:52:18 smithi103 sudo[50651]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:18.745 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 22 23:52:18 smithi103 sudo[50651]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:18.746 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 22 23:52:18 smithi103 sudo[50651]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:18.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.745+0000 7f5819600700 1 -- 172.21.15.103:0/1681387212 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5810098040 con 0x7f581c102ef0 2024-05-22T23:52:18.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:18.779+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f581c0627a0 con 0x7f58040a2c10 2024-05-22T23:52:19.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-22T23:52:19.048 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:18 smithi103 ceph-mon[31370]: osdmap e21: 3 total, 3 up, 3 in 2024-05-22T23:52:19.049 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:18 smithi103 ceph-mon[31370]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:19.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-22T23:52:19.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:18 smithi105 ceph-mon[36426]: osdmap e21: 3 total, 3 up, 3 in 2024-05-22T23:52:19.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:18 smithi105 ceph-mon[36426]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:19.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-22T23:52:19.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:18 smithi174 ceph-mon[36179]: osdmap e21: 3 total, 3 up, 3 in 2024-05-22T23:52:19.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:18 smithi174 ceph-mon[36179]: pgmap v95: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:19.331 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 22 23:52:19 smithi103 sudo[50741]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-22T23:52:19.332 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 22 23:52:19 smithi103 sudo[50741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:19.332 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 22 23:52:19 smithi103 sudo[50741]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:19.332 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 22 23:52:19 smithi103 sudo[50741]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:19.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:52:19 smithi105 sudo[44842]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-22T23:52:19.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:52:19 smithi105 sudo[44842]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:19.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:52:19 smithi105 sudo[44842]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:19.586 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:52:19 smithi105 sudo[44842]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:19.933 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:19 smithi103 ceph-mon[31370]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-22T23:52:19.933 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:19 smithi103 ceph-mon[31370]: osdmap e22: 3 total, 3 up, 3 in 2024-05-22T23:52:19.933 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:19 smithi103 ceph-mon[31370]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:52:19.933 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:52:19.933 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:52:19.933 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:19.933 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:19.933 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:20.060 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:52:19 smithi105 sudo[44845]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-22T23:52:20.060 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:52:19 smithi105 sudo[44845]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:20.060 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:52:19 smithi105 sudo[44845]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:20.060 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 22 23:52:19 smithi105 sudo[44845]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:19 smithi105 ceph-mon[36426]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-22T23:52:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:19 smithi105 ceph-mon[36426]: osdmap e22: 3 total, 3 up, 3 in 2024-05-22T23:52:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:19 smithi105 ceph-mon[36426]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:52:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:52:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:52:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:20.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:20.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:20.120 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:19 smithi174 sudo[45317]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-22T23:52:20.120 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:19 smithi174 sudo[45317]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:20.120 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:19 smithi174 sudo[45317]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:20.120 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:19 smithi174 sudo[45317]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:20.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:19 smithi174 ceph-mon[36179]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-22T23:52:20.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:19 smithi174 ceph-mon[36179]: osdmap e22: 3 total, 3 up, 3 in 2024-05-22T23:52:20.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:19 smithi174 ceph-mon[36179]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:52:20.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:52:20.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:52:20.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:20.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:20.122 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:20.619 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:20 smithi174 sudo[45320]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-22T23:52:20.619 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:20 smithi174 sudo[45320]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:20.619 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:20 smithi174 sudo[45320]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:20.620 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 22 23:52:20 smithi174 sudo[45320]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:21.022 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:20 smithi103 sudo[50821]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-22T23:52:21.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:20 smithi103 sudo[50821]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:21.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:20 smithi103 sudo[50821]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:21.023 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:20 smithi103 sudo[50821]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:21.580 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 ceph-mon[36179]: pgmap v97: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:21.580 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:21.580 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:21.580 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 sudo[45323]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-22T23:52:21.580 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 sudo[45323]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:21.581 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 sudo[45323]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:21.581 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 sudo[45323]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:21.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:21 smithi103 sudo[50849]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-22T23:52:21.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:21 smithi103 sudo[50849]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:21.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:21 smithi103 sudo[50849]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:21.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:21 smithi103 sudo[50849]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:21.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:21 smithi103 ceph-mon[31370]: pgmap v97: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:21.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:21.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:21 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:21.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:21 smithi105 ceph-mon[36426]: pgmap v97: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:21.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:21.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:21 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:21.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 sudo[45326]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-22T23:52:21.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 sudo[45326]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:21.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 sudo[45326]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:21.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:21 smithi174 sudo[45326]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:22.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 sudo[44848]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-22T23:52:22.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 sudo[44848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:22.313 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 sudo[44848]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:22.313 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 sudo[44848]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:22.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:22.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:22.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:22.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:22 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:22.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 sudo[44851]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-22T23:52:22.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 sudo[44851]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-22T23:52:22.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 sudo[44851]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-22T23:52:22.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 sudo[44851]: pam_unix(sudo:session): session closed for user root 2024-05-22T23:52:22.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:22.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:22.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:22 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:22.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:22.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-22T23:52:22.833 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-22T23:52:22.834 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:22.834 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:22 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-22T23:52:23.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:23 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1028203638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]: dispatch 2024-05-22T23:52:23.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:23 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]: dispatch 2024-05-22T23:52:23.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:23 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]': finished 2024-05-22T23:52:23.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:23 smithi174 ceph-mon[36179]: osdmap e23: 4 total, 3 up, 4 in 2024-05-22T23:52:23.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:23 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:23.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:23 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1028203638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]: dispatch 2024-05-22T23:52:23.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:23 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]: dispatch 2024-05-22T23:52:23.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:23 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]': finished 2024-05-22T23:52:23.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:23 smithi105 ceph-mon[36426]: osdmap e23: 4 total, 3 up, 4 in 2024-05-22T23:52:23.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:23 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:23.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:23 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1028203638' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]: dispatch 2024-05-22T23:52:23.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:23 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]: dispatch 2024-05-22T23:52:23.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:23 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eea36383-d6a9-49c6-a3cb-9d55f4275529"}]': finished 2024-05-22T23:52:23.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:23 smithi103 ceph-mon[31370]: osdmap e23: 4 total, 3 up, 4 in 2024-05-22T23:52:23.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:23 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:24.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:24 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3348060922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:52:24.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:24 smithi174 ceph-mon[36179]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:24.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:24 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3348060922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:52:24.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:24 smithi105 ceph-mon[36426]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:24.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:24 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3348060922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:52:24.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:24 smithi103 ceph-mon[31370]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:27.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:27 smithi103 ceph-mon[31370]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:27.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:27 smithi174 ceph-mon[36179]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:27.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:27 smithi105 ceph-mon[36426]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:29.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:29 smithi103 ceph-mon[31370]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-05-22T23:52:29.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:29 smithi174 ceph-mon[36179]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-05-22T23:52:29.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:29 smithi105 ceph-mon[36426]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-05-22T23:52:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:31 smithi103 ceph-mon[31370]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-05-22T23:52:31.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:31 smithi174 ceph-mon[36179]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-05-22T23:52:31.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:31 smithi105 ceph-mon[36426]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-05-22T23:52:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-22T23:52:32.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:32.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-22T23:52:32.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:32.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-22T23:52:32.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:33.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:33 smithi103 ceph-mon[31370]: Deploying daemon osd.3 on smithi103 2024-05-22T23:52:33.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:33 smithi103 ceph-mon[31370]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-05-22T23:52:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:33 smithi174 ceph-mon[36179]: Deploying daemon osd.3 on smithi103 2024-05-22T23:52:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:33 smithi174 ceph-mon[36179]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-05-22T23:52:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:33 smithi105 ceph-mon[36426]: Deploying daemon osd.3 on smithi103 2024-05-22T23:52:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:33 smithi105 ceph-mon[36426]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-05-22T23:52:34.411 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:34 smithi103 ceph-mon[31370]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 888 B/s rd, 23 KiB/s wr, 2 op/s 2024-05-22T23:52:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:34 smithi105 ceph-mon[36426]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 888 B/s rd, 23 KiB/s wr, 2 op/s 2024-05-22T23:52:34.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:34 smithi174 ceph-mon[36179]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 888 B/s rd, 23 KiB/s wr, 2 op/s 2024-05-22T23:52:37.275 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:37 smithi103 ceph-mon[31370]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-22T23:52:37.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:37 smithi174 ceph-mon[36179]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-22T23:52:37.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:37 smithi105 ceph-mon[36426]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-22T23:52:39.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:39 smithi103 ceph-mon[31370]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-22T23:52:39.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:39 smithi174 ceph-mon[36179]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-22T23:52:39.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:39 smithi105 ceph-mon[36426]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-05-22T23:52:41.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:41 smithi103 ceph-mon[31370]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:41.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:41.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:41.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:41 smithi174 ceph-mon[36179]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:41.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:41.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:41 smithi105 ceph-mon[36426]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:41.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:41 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:41.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:41 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:41.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:41 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:43.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:43 smithi103 ceph-mon[31370]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:43.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:43 smithi174 ceph-mon[36179]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:43.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:43 smithi105 ceph-mon[36426]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:44.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:44.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:44.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:44.986 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:44.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:44.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:44 smithi103 ceph-mon[31370]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:44.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:44 smithi103 ceph-mon[31370]: from='osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-22T23:52:44.987 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:44 smithi103 ceph-mon[31370]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-22T23:52:45.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:45.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:45.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:45.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:45.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:45.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:44 smithi105 ceph-mon[36426]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:45.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:44 smithi105 ceph-mon[36426]: from='osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-22T23:52:45.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:44 smithi105 ceph-mon[36426]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-22T23:52:45.090 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.084+0000 7f5819600700 1 -- 172.21.15.103:0/1681387212 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f581c0627a0 con 0x7f58040a2c10 2024-05-22T23:52:45.090 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 3 on host 'smithi103' 2024-05-22T23:52:45.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58040a2c10 msgr2=0x7f58040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:45.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58040a2c10 0x7f58040a50d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5808009750 tx=0x7f5808009310 comp rx=0 tx=0).stop 2024-05-22T23:52:45.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f581c102ef0 msgr2=0x7f581c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:45.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f581c102ef0 0x7f581c19a790 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f581000e9d0 tx=0x7f581000ece0 comp rx=0 tx=0).stop 2024-05-22T23:52:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 shutdown_connections 2024-05-22T23:52:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f58040a2c10 0x7f58040a50d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f581c108150 0x7f581c1a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f581c102ef0 0x7f581c19a790 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 --2- 172.21.15.103:0/1681387212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f581c105820 0x7f581c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.087+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 >> 172.21.15.103:0/1681387212 conn(0x7f581c0fc860 msgr2=0x7f581c102550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:52:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.088+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 shutdown_connections 2024-05-22T23:52:45.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:52:45.088+0000 7f5823400700 1 -- 172.21.15.103:0/1681387212 wait complete. 2024-05-22T23:52:45.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:45.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:45.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:45.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:45.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:45.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:44 smithi174 ceph-mon[36179]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:45.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:44 smithi174 ceph-mon[36179]: from='osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-22T23:52:45.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:44 smithi174 ceph-mon[36179]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-22T23:52:45.866 DEBUG:teuthology.orchestra.run.smithi103:osd.3> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.3.service 2024-05-22T23:52:45.871 INFO:tasks.cephadm:Deploying osd.4 on smithi105 with /dev/vg_nvme/lv_3... 2024-05-22T23:52:45.871 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-22T23:52:46.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:45 smithi105 ceph-mon[36426]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-22T23:52:46.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:45 smithi105 ceph-mon[36426]: from='osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:52:46.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:45 smithi105 ceph-mon[36426]: osdmap e24: 4 total, 3 up, 4 in 2024-05-22T23:52:46.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:45 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:46.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:45 smithi105 ceph-mon[36426]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:52:46.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:45 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:46.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:45 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:46.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:45 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:46.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:45 smithi103 ceph-mon[31370]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-22T23:52:46.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:45 smithi103 ceph-mon[31370]: from='osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:52:46.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:45 smithi103 ceph-mon[31370]: osdmap e24: 4 total, 3 up, 4 in 2024-05-22T23:52:46.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:45 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:46.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:45 smithi103 ceph-mon[31370]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:52:46.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:45 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:46.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:45 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:46.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:45 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:46.114 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-22T23:52:46.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:45 smithi174 ceph-mon[36179]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-22T23:52:46.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:45 smithi174 ceph-mon[36179]: from='osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:52:46.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:45 smithi174 ceph-mon[36179]: osdmap e24: 4 total, 3 up, 4 in 2024-05-22T23:52:46.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:45 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:46.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:45 smithi174 ceph-mon[36179]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:52:46.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:45 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:46.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:45 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:46.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:45 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:52:46.582 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 22 23:52:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-22T23:52:46.333+0000 7fd1cf000700 -1 osd.3 0 waiting for initial osdmap 2024-05-22T23:52:46.582 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 22 23:52:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-22T23:52:46.350+0000 7fd1c4400700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:52:46.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:52:46.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: osdmap e25: 4 total, 3 up, 4 in 2024-05-22T23:52:46.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:46.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:46.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:46.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: from='osd.3 ' entity='osd.3' 2024-05-22T23:52:46.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] boot 2024-05-22T23:52:46.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: osdmap e26: 4 total, 4 up, 4 in 2024-05-22T23:52:46.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:46.988 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:52:46.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: osdmap e25: 4 total, 3 up, 4 in 2024-05-22T23:52:46.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:46.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:46.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:46.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: from='osd.3 ' entity='osd.3' 2024-05-22T23:52:46.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] boot 2024-05-22T23:52:46.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: osdmap e26: 4 total, 4 up, 4 in 2024-05-22T23:52:46.989 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:47.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:52:47.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: osdmap e25: 4 total, 3 up, 4 in 2024-05-22T23:52:47.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:47.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:47.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-22T23:52:47.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: from='osd.3 ' entity='osd.3' 2024-05-22T23:52:47.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: osd.3 [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] boot 2024-05-22T23:52:47.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: osdmap e26: 4 total, 4 up, 4 in 2024-05-22T23:52:47.121 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-22T23:52:47.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:47 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:52:47.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:47 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:52:47.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:47 smithi103 ceph-mon[31370]: osdmap e27: 4 total, 4 up, 4 in 2024-05-22T23:52:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-22T23:52:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-22T23:52:47.956 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-22T23:52:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-22T23:52:47.956 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113123 s, 927 MB/s 2024-05-22T23:52:47.956 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-22T23:52:47.979 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:47 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:52:47.979 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:47 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:52:47.979 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:47 smithi105 ceph-mon[36426]: osdmap e27: 4 total, 4 up, 4 in 2024-05-22T23:52:48.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:47 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:52:48.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:47 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:52:48.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:47 smithi174 ceph-mon[36179]: osdmap e27: 4 total, 4 up, 4 in 2024-05-22T23:52:48.535 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2024-05-22T23:52:48.773 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-22T23:52:48.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:48 smithi105 ceph-mon[36426]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 357 GiB / 358 GiB avail 2024-05-22T23:52:48.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:48 smithi105 ceph-mon[36426]: osdmap e28: 4 total, 4 up, 4 in 2024-05-22T23:52:49.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:48 smithi103 ceph-mon[31370]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 357 GiB / 358 GiB avail 2024-05-22T23:52:49.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:48 smithi103 ceph-mon[31370]: osdmap e28: 4 total, 4 up, 4 in 2024-05-22T23:52:49.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:48 smithi174 ceph-mon[36179]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 357 GiB / 358 GiB avail 2024-05-22T23:52:49.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:48 smithi174 ceph-mon[36179]: osdmap e28: 4 total, 4 up, 4 in 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 -- 172.21.15.105:0/4674568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 msgr2=0x7fa7401016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 --2- 172.21.15.105:0/4674568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 0x7fa7401016e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa7300099c0 tx=0x7fa730009cd0 comp rx=0 tx=0).stop 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 -- 172.21.15.105:0/4674568 shutdown_connections 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 --2- 172.21.15.105:0/4674568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 0x7fa7401016e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 --2- 172.21.15.105:0/4674568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa740102e60 0x7fa740109710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 --2- 172.21.15.105:0/4674568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7401024c0 0x7fa740102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 -- 172.21.15.105:0/4674568 >> 172.21.15.105:0/4674568 conn(0x7fa7400fc860 msgr2=0x7fa7400feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 -- 172.21.15.105:0/4674568 shutdown_connections 2024-05-22T23:52:50.415 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 -- 172.21.15.105:0/4674568 wait complete. 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.408+0000 7fa745195700 1 Processor -- start 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa745195700 1 -- start start 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa745195700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 0x7fa740198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa745195700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7401024c0 0x7fa740198c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa745195700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa740102e60 0x7fa74019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa745195700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010cf80 con 0x7fa7401024c0 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa745195700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010ce00 con 0x7fa7401012c0 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa745195700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa74010d100 con 0x7fa740102e60 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 0x7fa740198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 0x7fa740198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51794/0 (socket says 172.21.15.105:51794) 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 -- 172.21.15.105:0/4136334104 learned_addr learned my addr 172.21.15.105:0/4136334104 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-22T23:52:50.416 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73f59e700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa740102e60 0x7fa74019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 -- 172.21.15.105:0/4136334104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa740102e60 msgr2=0x7fa74019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa740102e60 0x7fa74019fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 -- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7401024c0 msgr2=0x7fa740198c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7401024c0 0x7fa740198c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 -- 172.21.15.105:0/4136334104 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7401a03b0 con 0x7fa7401012c0 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73f59e700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa740102e60 0x7fa74019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73ed9d700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 0x7fa740198750 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa73000c010 tx=0x7fa730003680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.409+0000 7fa73cd99700 1 -- 172.21.15.105:0/4136334104 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa730027070 con 0x7fa7401012c0 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.410+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7401a0640 con 0x7fa7401012c0 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.410+0000 7fa73cd99700 1 -- 172.21.15.105:0/4136334104 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa730022470 con 0x7fa7401012c0 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.410+0000 7fa73cd99700 1 -- 172.21.15.105:0/4136334104 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73000f6a0 con 0x7fa7401012c0 2024-05-22T23:52:50.417 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.410+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7401a0ba0 con 0x7fa7401012c0 2024-05-22T23:52:50.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.410+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa740069050 con 0x7fa7401012c0 2024-05-22T23:52:50.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.412+0000 7fa73cd99700 1 -- 172.21.15.105:0/4136334104 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa73002a070 con 0x7fa7401012c0 2024-05-22T23:52:50.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.412+0000 7fa73cd99700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa72c0a2c20 0x7fa72c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:52:50.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.413+0000 7fa73cd99700 1 -- 172.21.15.105:0/4136334104 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7fa7300cf150 con 0x7fa7401012c0 2024-05-22T23:52:50.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.413+0000 7fa73659c700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa72c0a2c20 0x7fa72c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:52:50.419 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.414+0000 7fa73659c700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa72c0a2c20 0x7fa72c0a50e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa7280097b0 tx=0x7fa728006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:52:50.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.416+0000 7fa73cd99700 1 -- 172.21.15.105:0/4136334104 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa730013070 con 0x7fa7401012c0 2024-05-22T23:52:50.625 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:52:50.620+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa7400feaa0 con 0x7fa72c0a2c20 2024-05-22T23:52:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:50 smithi105 ceph-mon[36426]: osdmap e29: 4 total, 4 up, 4 in 2024-05-22T23:52:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:50 smithi105 ceph-mon[36426]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 357 GiB / 358 GiB avail 2024-05-22T23:52:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:52:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:52:51.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:51.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:50 smithi103 ceph-mon[31370]: osdmap e29: 4 total, 4 up, 4 in 2024-05-22T23:52:51.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:50 smithi103 ceph-mon[31370]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 357 GiB / 358 GiB avail 2024-05-22T23:52:51.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:52:51.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:52:51.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:51.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:50 smithi174 ceph-mon[36179]: osdmap e29: 4 total, 4 up, 4 in 2024-05-22T23:52:51.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:50 smithi174 ceph-mon[36179]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 357 GiB / 358 GiB avail 2024-05-22T23:52:51.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:52:51.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:52:51.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:51.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:51 smithi103 ceph-mon[31370]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:52:51.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:51 smithi103 ceph-mon[31370]: osdmap e30: 4 total, 4 up, 4 in 2024-05-22T23:52:52.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:51 smithi105 ceph-mon[36426]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:52:52.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:51 smithi105 ceph-mon[36426]: osdmap e30: 4 total, 4 up, 4 in 2024-05-22T23:52:52.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:51 smithi174 ceph-mon[36179]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:52:52.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:51 smithi174 ceph-mon[36179]: osdmap e30: 4 total, 4 up, 4 in 2024-05-22T23:52:53.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-05-22T23:52:53.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi103 2024-05-22T23:52:53.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:53.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:53.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:53.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:53.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-05-22T23:52:53.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi103 2024-05-22T23:52:53.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:53.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:53.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:53.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:53.371 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-05-22T23:52:53.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi103 2024-05-22T23:52:53.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:53.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:52:53.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:53.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:52:53.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:52:53.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:52:54.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:54 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi103 to 8544M 2024-05-22T23:52:54.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:54 smithi103 ceph-mon[31370]: from='client.? 172.21.15.105:0/660558316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]: dispatch 2024-05-22T23:52:54.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:54 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]: dispatch 2024-05-22T23:52:54.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:54 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]': finished 2024-05-22T23:52:54.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:54 smithi103 ceph-mon[31370]: osdmap e31: 5 total, 4 up, 5 in 2024-05-22T23:52:54.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:52:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:54 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi103 to 8544M 2024-05-22T23:52:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:54 smithi174 ceph-mon[36179]: from='client.? 172.21.15.105:0/660558316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]: dispatch 2024-05-22T23:52:54.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:54 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]: dispatch 2024-05-22T23:52:54.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:54 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]': finished 2024-05-22T23:52:54.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:54 smithi174 ceph-mon[36179]: osdmap e31: 5 total, 4 up, 5 in 2024-05-22T23:52:54.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:52:54.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:54 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi103 to 8544M 2024-05-22T23:52:54.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:54 smithi105 ceph-mon[36426]: from='client.? 172.21.15.105:0/660558316' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]: dispatch 2024-05-22T23:52:54.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:54 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]: dispatch 2024-05-22T23:52:54.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:54 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71550539-dfa3-4158-b4b9-4357ae474da2"}]': finished 2024-05-22T23:52:54.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:54 smithi105 ceph-mon[36426]: osdmap e31: 5 total, 4 up, 5 in 2024-05-22T23:52:54.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:52:55.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:55 smithi103 ceph-mon[31370]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-22T23:52:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:55 smithi103 ceph-mon[31370]: from='client.? 172.21.15.105:0/2453021817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:52:55.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:55 smithi174 ceph-mon[36179]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-22T23:52:55.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:55 smithi174 ceph-mon[36179]: from='client.? 172.21.15.105:0/2453021817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:52:55.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:55 smithi105 ceph-mon[36426]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-05-22T23:52:55.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:55 smithi105 ceph-mon[36426]: from='client.? 172.21.15.105:0/2453021817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:52:57.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:57 smithi174 ceph-mon[36179]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-22T23:52:57.482 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:57 smithi105 ceph-mon[36426]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-22T23:52:57.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:57 smithi103 ceph-mon[31370]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-05-22T23:52:59.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:52:59 smithi174 ceph-mon[36179]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-22T23:52:59.398 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:52:59 smithi105 ceph-mon[36426]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-22T23:52:59.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:52:59 smithi103 ceph-mon[31370]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-22T23:53:00.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:00 smithi105 ceph-mon[36426]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-22T23:53:00.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:00 smithi103 ceph-mon[31370]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-22T23:53:00.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:00 smithi174 ceph-mon[36179]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-05-22T23:53:01.555 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:01 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-22T23:53:01.555 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:01 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:01.555 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:01 smithi105 ceph-mon[36426]: Deploying daemon osd.4 on smithi105 2024-05-22T23:53:01.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:01 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-22T23:53:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:01 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:01.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:01 smithi103 ceph-mon[31370]: Deploying daemon osd.4 on smithi105 2024-05-22T23:53:01.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:01 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-22T23:53:01.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:01 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:01.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:01 smithi174 ceph-mon[36179]: Deploying daemon osd.4 on smithi105 2024-05-22T23:53:02.521 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:02 smithi105 ceph-mon[36426]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:02 smithi103 ceph-mon[31370]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:02.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:02 smithi174 ceph-mon[36179]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:05.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:05 smithi103 ceph-mon[31370]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:05.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:05 smithi174 ceph-mon[36179]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:05.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:05 smithi105 ceph-mon[36426]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:07.144 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:07 smithi105 ceph-mon[36426]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:07.144 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:07.144 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:07.145 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:07 smithi103 ceph-mon[31370]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:07.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:07.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:07 smithi174 ceph-mon[36179]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:07.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:07.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:07.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:09.362 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:09 smithi105 ceph-mon[36426]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:09.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:09 smithi174 ceph-mon[36179]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:09.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:09 smithi103 ceph-mon[31370]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:10.541 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.541 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.541 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:10.542 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:10.542 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.542 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:10 smithi105 ceph-mon[36426]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:10 smithi103 ceph-mon[31370]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:10.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.607+0000 7fa73cd99700 1 -- 172.21.15.105:0/4136334104 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa7400feaa0 con 0x7fa72c0a2c20 2024-05-22T23:53:10.613 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 4 on host 'smithi105' 2024-05-22T23:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa72c0a2c20 msgr2=0x7fa72c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa72c0a2c20 0x7fa72c0a50e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa7280097b0 tx=0x7fa728006d20 comp rx=0 tx=0).stop 2024-05-22T23:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 msgr2=0x7fa740198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 0x7fa740198750 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa73000c010 tx=0x7fa730003680 comp rx=0 tx=0).stop 2024-05-22T23:53:10.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 shutdown_connections 2024-05-22T23:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7401012c0 0x7fa740198750 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa73000c010 tx=0x7fa730003680 comp rx=0 tx=0).stop 2024-05-22T23:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa72c0a2c20 0x7fa72c0a50e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa740102e60 0x7fa74019fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 --2- 172.21.15.105:0/4136334104 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7401024c0 0x7fa740198c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 >> 172.21.15.105:0/4136334104 conn(0x7fa7400fc860 msgr2=0x7fa7400fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 shutdown_connections 2024-05-22T23:53:10.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:53:10.610+0000 7fa745195700 1 -- 172.21.15.105:0/4136334104 wait complete. 2024-05-22T23:53:10.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:10.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:10.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:10.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:10 smithi174 ceph-mon[36179]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:11.331 DEBUG:teuthology.orchestra.run.smithi105:osd.4> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.4.service 2024-05-22T23:53:11.333 INFO:tasks.cephadm:Deploying osd.5 on smithi174 with /dev/vg_nvme/lv_3... 2024-05-22T23:53:11.334 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-22T23:53:11.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:11 smithi105 ceph-mon[36426]: from='osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-22T23:53:11.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:11 smithi105 ceph-mon[36426]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-22T23:53:11.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:11 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:11.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:11 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:11.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:11 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:11.576 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:53:11.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:11 smithi103 ceph-mon[31370]: from='osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-22T23:53:11.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:11 smithi103 ceph-mon[31370]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-22T23:53:11.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:11.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:11.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:11.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:11 smithi174 ceph-mon[36179]: from='osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-22T23:53:11.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:11 smithi174 ceph-mon[36179]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-22T23:53:11.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:11 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:11.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:11 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:11.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:11 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:12 smithi105 ceph-mon[36426]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-22T23:53:12.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:12 smithi105 ceph-mon[36426]: from='osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:53:12.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:12 smithi105 ceph-mon[36426]: osdmap e32: 5 total, 4 up, 5 in 2024-05-22T23:53:12.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:12.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:12 smithi105 ceph-mon[36426]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:53:12.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:12 smithi105 ceph-mon[36426]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:12.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:12 smithi105 ceph-mon[36426]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:53:12.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:12 smithi105 ceph-mon[36426]: osdmap e33: 5 total, 4 up, 5 in 2024-05-22T23:53:12.561 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 22 23:53:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-22T23:53:12.258+0000 7f714ce7f700 -1 osd.4 0 waiting for initial osdmap 2024-05-22T23:53:12.561 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 22 23:53:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-22T23:53:12.265+0000 7f7148c72700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:53:12.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:12 smithi103 ceph-mon[31370]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-22T23:53:12.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:12 smithi103 ceph-mon[31370]: from='osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:53:12.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:12 smithi103 ceph-mon[31370]: osdmap e32: 5 total, 4 up, 5 in 2024-05-22T23:53:12.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:12.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:12 smithi103 ceph-mon[31370]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:53:12.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:12 smithi103 ceph-mon[31370]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:12.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:12 smithi103 ceph-mon[31370]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:53:12.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:12 smithi103 ceph-mon[31370]: osdmap e33: 5 total, 4 up, 5 in 2024-05-22T23:53:12.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:12 smithi174 ceph-mon[36179]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-22T23:53:12.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:12 smithi174 ceph-mon[36179]: from='osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:53:12.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:12 smithi174 ceph-mon[36179]: osdmap e32: 5 total, 4 up, 5 in 2024-05-22T23:53:12.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:12.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:12 smithi174 ceph-mon[36179]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:53:12.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:12 smithi174 ceph-mon[36179]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:12.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:12 smithi174 ceph-mon[36179]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:53:12.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:12 smithi174 ceph-mon[36179]: osdmap e33: 5 total, 4 up, 5 in 2024-05-22T23:53:13.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:13 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:13.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:13 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:13.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:13 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:13.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:13 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:13.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:13 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:13.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:13 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:13.723 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-22T23:53:13.723 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-22T23:53:13.723 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-05-22T23:53:13.723 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-05-22T23:53:13.723 INFO:teuthology.orchestra.run.smithi174.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.015158 s, 692 MB/s 2024-05-22T23:53:13.723 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-05-22T23:53:14.394 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:14 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:53:14.395 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:14 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:53:14.395 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:14 smithi105 ceph-mon[36426]: osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] boot 2024-05-22T23:53:14.395 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:14 smithi105 ceph-mon[36426]: osdmap e34: 5 total, 5 up, 5 in 2024-05-22T23:53:14.395 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:14.395 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:14 smithi105 ceph-mon[36426]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:14.410 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_3 2024-05-22T23:53:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:14 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:53:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:14 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:53:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:14 smithi103 ceph-mon[31370]: osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] boot 2024-05-22T23:53:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:14 smithi103 ceph-mon[31370]: osdmap e34: 5 total, 5 up, 5 in 2024-05-22T23:53:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:14.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:14 smithi103 ceph-mon[31370]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:14.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:14 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:53:14.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:14 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:53:14.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:14 smithi174 ceph-mon[36179]: osd.4 [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] boot 2024-05-22T23:53:14.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:14 smithi174 ceph-mon[36179]: osdmap e34: 5 total, 5 up, 5 in 2024-05-22T23:53:14.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-22T23:53:14.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:14 smithi174 ceph-mon[36179]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-22T23:53:14.653 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:53:15.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: osdmap e35: 5 total, 5 up, 5 in 2024-05-22T23:53:15.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi105 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: osdmap e35: 5 total, 5 up, 5 in 2024-05-22T23:53:15.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi105 2024-05-22T23:53:15.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:15.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:15.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-22T23:53:15.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:15.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:15.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: osdmap e35: 5 total, 5 up, 5 in 2024-05-22T23:53:15.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi105 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi105 to 8544M 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:15.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:16.324 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.316+0000 7f992f55c700 1 -- 172.21.15.174:0/1845500075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928105780 msgr2=0x7f9928107b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:16.325 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.316+0000 7f992f55c700 1 --2- 172.21.15.174:0/1845500075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928105780 0x7f9928107b70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9918009990 tx=0x7f9918009ca0 comp rx=0 tx=0).stop 2024-05-22T23:53:16.325 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 -- 172.21.15.174:0/1845500075 shutdown_connections 2024-05-22T23:53:16.325 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 --2- 172.21.15.174:0/1845500075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f99281080b0 0x7f992810a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:16.325 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 --2- 172.21.15.174:0/1845500075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9928102e50 0x7f9928105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:16.325 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 --2- 172.21.15.174:0/1845500075 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928105780 0x7f9928107b70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:16.326 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 -- 172.21.15.174:0/1845500075 >> 172.21.15.174:0/1845500075 conn(0x7f99280fc880 msgr2=0x7f99280feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:53:16.326 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 -- 172.21.15.174:0/1845500075 shutdown_connections 2024-05-22T23:53:16.326 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 -- 172.21.15.174:0/1845500075 wait complete. 2024-05-22T23:53:16.326 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 Processor -- start 2024-05-22T23:53:16.326 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 -- start start 2024-05-22T23:53:16.326 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928102e50 0x7f9928198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:53:16.327 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9928105780 0x7f9928198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:53:16.327 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99281080b0 0x7f992819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:53:16.327 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f992810e090 con 0x7f99281080b0 2024-05-22T23:53:16.327 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f992810df10 con 0x7f9928105780 2024-05-22T23:53:16.327 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992f55c700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f992810e210 con 0x7f9928102e50 2024-05-22T23:53:16.327 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992d2f8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928102e50 0x7f9928198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:53:16.328 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992d2f8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928102e50 0x7f9928198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:49510/0 (socket says 172.21.15.174:49510) 2024-05-22T23:53:16.328 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.317+0000 7f992d2f8700 1 -- 172.21.15.174:0/531933965 learned_addr learned my addr 172.21.15.174:0/531933965 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:53:16.328 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992caf7700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9928105780 0x7f9928198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:53:16.328 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992d2f8700 1 -- 172.21.15.174:0/531933965 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9928105780 msgr2=0x7f9928198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:16.329 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992d2f8700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9928105780 0x7f9928198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:16.329 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992d2f8700 1 -- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99281080b0 msgr2=0x7f992819fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:16.329 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992d2f8700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99281080b0 0x7f992819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:16.329 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992d2f8700 1 -- 172.21.15.174:0/531933965 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99281a03a0 con 0x7f9928102e50 2024-05-22T23:53:16.329 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992caf7700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9928105780 0x7f9928198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-22T23:53:16.330 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992d2f8700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928102e50 0x7f9928198740 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f992400b560 tx=0x7f992400b920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:53:16.330 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f991e7fc700 1 -- 172.21.15.174:0/531933965 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9924015020 con 0x7f9928102e50 2024-05-22T23:53:16.330 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99281a0690 con 0x7f9928102e50 2024-05-22T23:53:16.330 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f991e7fc700 1 -- 172.21.15.174:0/531933965 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9924011d10 con 0x7f9928102e50 2024-05-22T23:53:16.330 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f991e7fc700 1 -- 172.21.15.174:0/531933965 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992400f2e0 con 0x7f9928102e50 2024-05-22T23:53:16.330 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.318+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99281a0ca0 con 0x7f9928102e50 2024-05-22T23:53:16.330 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.319+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9928069050 con 0x7f9928102e50 2024-05-22T23:53:16.332 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.320+0000 7f991e7fc700 1 -- 172.21.15.174:0/531933965 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9924018020 con 0x7f9928102e50 2024-05-22T23:53:16.332 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.321+0000 7f991e7fc700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f99140a2c20 0x7f99140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:53:16.332 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.321+0000 7f991e7fc700 1 -- 172.21.15.174:0/531933965 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 4101+0+0 (secure 0 0 0) 0x7f99240cdd50 con 0x7f9928102e50 2024-05-22T23:53:16.332 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.321+0000 7f992caf7700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f99140a2c20 0x7f99140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:53:16.332 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.322+0000 7f992caf7700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f99140a2c20 0x7f99140a50e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f99180039c0 tx=0x7f9918005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:53:16.332 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.325+0000 7f991e7fc700 1 -- 172.21.15.174:0/531933965 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f992401f020 con 0x7f9928102e50 2024-05-22T23:53:16.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:16 smithi174 ceph-mon[36179]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:16.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:16.530+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f99280627a0 con 0x7f99140a2c20 2024-05-22T23:53:16.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:16 smithi105 ceph-mon[36426]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:16.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:16 smithi103 ceph-mon[31370]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:17.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:17 smithi174 ceph-mon[36179]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:53:17.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:53:17.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:53:17.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:17.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:17 smithi105 ceph-mon[36426]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:53:17.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:53:17.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:53:17.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:17.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:17 smithi103 ceph-mon[31370]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:53:17.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:53:17.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:53:17.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:18.433 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:18 smithi174 ceph-mon[36179]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:18.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:18 smithi103 ceph-mon[31370]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:18.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:18 smithi105 ceph-mon[36426]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:19.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:19 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/1415524221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]: dispatch 2024-05-22T23:53:19.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:19 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]: dispatch 2024-05-22T23:53:19.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:19 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]': finished 2024-05-22T23:53:19.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:19 smithi103 ceph-mon[31370]: osdmap e36: 6 total, 5 up, 6 in 2024-05-22T23:53:19.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:19 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:19.609 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:19 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/1415524221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]: dispatch 2024-05-22T23:53:19.610 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:19 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]: dispatch 2024-05-22T23:53:19.610 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:19 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]': finished 2024-05-22T23:53:19.610 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:19 smithi174 ceph-mon[36179]: osdmap e36: 6 total, 5 up, 6 in 2024-05-22T23:53:19.610 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:19 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:19.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:19 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/1415524221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]: dispatch 2024-05-22T23:53:19.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:19 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]: dispatch 2024-05-22T23:53:19.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:19 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d76fc01a-a243-4277-bbdc-1c31965446db"}]': finished 2024-05-22T23:53:19.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:19 smithi105 ceph-mon[36426]: osdmap e36: 6 total, 5 up, 6 in 2024-05-22T23:53:19.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:19 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:20.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:20 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/12960555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:53:20.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:20 smithi103 ceph-mon[31370]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:20.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:20 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/12960555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:53:20.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:20 smithi174 ceph-mon[36179]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:20 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/12960555' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:53:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:20 smithi105 ceph-mon[36426]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:23.231 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:23 smithi174 ceph-mon[36179]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:23.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:23 smithi103 ceph-mon[31370]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:23.513 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:23 smithi105 ceph-mon[36426]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:25.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:25 smithi103 ceph-mon[31370]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:25.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:25 smithi174 ceph-mon[36179]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:25.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:25 smithi105 ceph-mon[36426]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:27.346 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:27 smithi174 ceph-mon[36179]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:27.347 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:27 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-22T23:53:27.347 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:27 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:27.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:27 smithi105 ceph-mon[36426]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:27.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:27 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-22T23:53:27.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:27 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:27.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:27 smithi103 ceph-mon[31370]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:27.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:27 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-22T23:53:27.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:27 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:28.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:28 smithi174 ceph-mon[36179]: Deploying daemon osd.5 on smithi174 2024-05-22T23:53:28.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:28 smithi105 ceph-mon[36426]: Deploying daemon osd.5 on smithi174 2024-05-22T23:53:28.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:28 smithi103 ceph-mon[31370]: Deploying daemon osd.5 on smithi174 2024-05-22T23:53:29.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:29 smithi174 ceph-mon[36179]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:29.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:29 smithi105 ceph-mon[36426]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:29.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:29 smithi103 ceph-mon[31370]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:30.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:30 smithi174 ceph-mon[36179]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:30.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:30 smithi105 ceph-mon[36426]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:30.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:30 smithi103 ceph-mon[31370]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:33.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:33.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:33.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:33.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:32 smithi105 ceph-mon[36426]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:33.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:33.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:33.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:33.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:32 smithi103 ceph-mon[31370]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:32 smithi174 ceph-mon[36179]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:35.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:35 smithi103 ceph-mon[31370]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:35.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:35.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:35.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:35 smithi174 ceph-mon[36179]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:35.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:35.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:35 smithi105 ceph-mon[36426]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:35.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:35.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:35.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:35.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.864+0000 7f991e7fc700 1 -- 172.21.15.174:0/531933965 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f99280627a0 con 0x7f99140a2c20 2024-05-22T23:53:35.870 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 5 on host 'smithi174' 2024-05-22T23:53:35.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f99140a2c20 msgr2=0x7f99140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:35.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f99140a2c20 0x7f99140a50e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f99180039c0 tx=0x7f9918005fb0 comp rx=0 tx=0).stop 2024-05-22T23:53:35.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928102e50 msgr2=0x7f9928198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:35.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928102e50 0x7f9928198740 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f992400b560 tx=0x7f992400b920 comp rx=0 tx=0).stop 2024-05-22T23:53:35.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 shutdown_connections 2024-05-22T23:53:35.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f99140a2c20 0x7f99140a50e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:35.872 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f99281080b0 0x7f992819fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:35.873 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9928102e50 0x7f9928198740 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:35.873 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 --2- 172.21.15.174:0/531933965 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9928105780 0x7f9928198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:35.873 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 >> 172.21.15.174:0/531933965 conn(0x7f99280fc880 msgr2=0x7f9928103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:53:35.873 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 shutdown_connections 2024-05-22T23:53:35.873 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:53:35.867+0000 7f992f55c700 1 -- 172.21.15.174:0/531933965 wait complete. 2024-05-22T23:53:36.566 DEBUG:teuthology.orchestra.run.smithi174:osd.5> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.5.service 2024-05-22T23:53:36.568 INFO:tasks.cephadm:Deploying osd.6 on smithi103 with /dev/vg_nvme/lv_2... 2024-05-22T23:53:36.568 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-22T23:53:37.012 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:53:37.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:37.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:37.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:37.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:36 smithi103 ceph-mon[31370]: from='osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-22T23:53:37.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:36 smithi103 ceph-mon[31370]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-22T23:53:37.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:36 smithi103 ceph-mon[31370]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:37.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:37.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:37.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:37.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:36 smithi174 ceph-mon[36179]: from='osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-22T23:53:37.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:36 smithi174 ceph-mon[36179]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-22T23:53:37.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:36 smithi174 ceph-mon[36179]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:37.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:37.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:37.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:53:37.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:36 smithi105 ceph-mon[36426]: from='osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-22T23:53:37.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:36 smithi105 ceph-mon[36426]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-22T23:53:37.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:36 smithi105 ceph-mon[36426]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:38.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:37 smithi174 ceph-mon[36179]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-22T23:53:38.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:37 smithi174 ceph-mon[36179]: from='osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:53:38.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:37 smithi174 ceph-mon[36179]: osdmap e37: 6 total, 5 up, 6 in 2024-05-22T23:53:38.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:37 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:38.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:37 smithi174 ceph-mon[36179]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:53:38.210 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:37 smithi103 ceph-mon[31370]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-22T23:53:38.211 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:37 smithi103 ceph-mon[31370]: from='osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:53:38.211 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:37 smithi103 ceph-mon[31370]: osdmap e37: 6 total, 5 up, 6 in 2024-05-22T23:53:38.211 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:37 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:38.211 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:37 smithi103 ceph-mon[31370]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:53:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:37 smithi105 ceph-mon[36426]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-22T23:53:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:37 smithi105 ceph-mon[36426]: from='osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:53:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:37 smithi105 ceph-mon[36426]: osdmap e37: 6 total, 5 up, 6 in 2024-05-22T23:53:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:37 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:37 smithi105 ceph-mon[36426]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:53:38.620 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 22 23:53:38 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-22T23:53:38.311+0000 7ff2c7840700 -1 osd.5 0 waiting for initial osdmap 2024-05-22T23:53:38.620 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 22 23:53:38 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-22T23:53:38.320+0000 7ff2bf62b700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:53:39.013 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:38 smithi103 ceph-mon[31370]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:53:39.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:38 smithi103 ceph-mon[31370]: osdmap e38: 6 total, 5 up, 6 in 2024-05-22T23:53:39.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:38 smithi103 ceph-mon[31370]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:39.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:38 smithi103 ceph-mon[31370]: osdmap e39: 6 total, 5 up, 6 in 2024-05-22T23:53:39.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:38 smithi103 ceph-mon[31370]: from='osd.5 ' entity='osd.5' 2024-05-22T23:53:39.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:38 smithi174 ceph-mon[36179]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:53:39.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:38 smithi174 ceph-mon[36179]: osdmap e38: 6 total, 5 up, 6 in 2024-05-22T23:53:39.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:38 smithi174 ceph-mon[36179]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:39.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:38 smithi174 ceph-mon[36179]: osdmap e39: 6 total, 5 up, 6 in 2024-05-22T23:53:39.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:38 smithi174 ceph-mon[36179]: from='osd.5 ' entity='osd.5' 2024-05-22T23:53:39.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:38 smithi105 ceph-mon[36426]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:53:39.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:38 smithi105 ceph-mon[36426]: osdmap e38: 6 total, 5 up, 6 in 2024-05-22T23:53:39.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:38 smithi105 ceph-mon[36426]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-05-22T23:53:39.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:38 smithi105 ceph-mon[36426]: osdmap e39: 6 total, 5 up, 6 in 2024-05-22T23:53:39.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:39.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:38 smithi105 ceph-mon[36426]: from='osd.5 ' entity='osd.5' 2024-05-22T23:53:40.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] boot 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: osdmap e40: 6 total, 6 up, 6 in 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:40.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] boot 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: osdmap e40: 6 total, 6 up, 6 in 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:40.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: osd.5 [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] boot 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: osdmap e40: 6 total, 6 up, 6 in 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.323 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:40.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:53:40.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:40.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:53:40.324 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:53:40.324 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-22T23:53:40.324 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-22T23:53:40.324 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-05-22T23:53:40.324 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-05-22T23:53:40.324 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0317994 s, 330 MB/s 2024-05-22T23:53:40.325 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-05-22T23:53:41.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:40 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi174 2024-05-22T23:53:41.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:40 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi174 to 8544M 2024-05-22T23:53:41.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:40 smithi103 ceph-mon[31370]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:53:41.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:40 smithi103 ceph-mon[31370]: osdmap e41: 6 total, 6 up, 6 in 2024-05-22T23:53:41.253 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-05-22T23:53:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:40 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi174 2024-05-22T23:53:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:40 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi174 to 8544M 2024-05-22T23:53:41.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:40 smithi105 ceph-mon[36426]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:53:41.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:40 smithi105 ceph-mon[36426]: osdmap e41: 6 total, 6 up, 6 in 2024-05-22T23:53:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:40 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi174 2024-05-22T23:53:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:40 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi174 to 8544M 2024-05-22T23:53:41.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:40 smithi174 ceph-mon[36179]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:53:41.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:40 smithi174 ceph-mon[36179]: osdmap e41: 6 total, 6 up, 6 in 2024-05-22T23:53:41.680 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:53:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:42 smithi105 ceph-mon[36426]: osdmap e42: 6 total, 6 up, 6 in 2024-05-22T23:53:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:42 smithi105 ceph-mon[36426]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:42.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:42 smithi103 ceph-mon[31370]: osdmap e42: 6 total, 6 up, 6 in 2024-05-22T23:53:42.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:42 smithi103 ceph-mon[31370]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:42.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:42 smithi174 ceph-mon[36179]: osdmap e42: 6 total, 6 up, 6 in 2024-05-22T23:53:42.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:42 smithi174 ceph-mon[36179]: pgmap v158: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:43.466 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:43 smithi103 ceph-mon[31370]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-22T23:53:43.467 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:43 smithi103 ceph-mon[31370]: osdmap e43: 6 total, 6 up, 6 in 2024-05-22T23:53:43.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:43 smithi105 ceph-mon[36426]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-22T23:53:43.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:43 smithi105 ceph-mon[36426]: osdmap e43: 6 total, 6 up, 6 in 2024-05-22T23:53:43.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:43 smithi174 ceph-mon[36179]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-22T23:53:43.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:43 smithi174 ceph-mon[36179]: osdmap e43: 6 total, 6 up, 6 in 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.957+0000 7f9fb5000700 1 -- 172.21.15.103:0/273451937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb0105700 msgr2=0x7f9fb0105b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.957+0000 7f9fb5000700 1 --2- 172.21.15.103:0/273451937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb0105700 0x7f9fb0105b60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0099c0 tx=0x7f9f9c009cd0 comp rx=0 tx=0).stop 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.958+0000 7f9fb5000700 1 -- 172.21.15.103:0/273451937 shutdown_connections 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.958+0000 7f9fb5000700 1 --2- 172.21.15.103:0/273451937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fb01060a0 0x7f9fb010c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.958+0000 7f9fb5000700 1 --2- 172.21.15.103:0/273451937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb0104500 0x7f9fb0104920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.958+0000 7f9fb5000700 1 --2- 172.21.15.103:0/273451937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb0105700 0x7f9fb0105b60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.958+0000 7f9fb5000700 1 -- 172.21.15.103:0/273451937 >> 172.21.15.103:0/273451937 conn(0x7f9fb00ffaa0 msgr2=0x7f9fb0101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.958+0000 7f9fb5000700 1 -- 172.21.15.103:0/273451937 shutdown_connections 2024-05-22T23:53:43.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.958+0000 7f9fb5000700 1 -- 172.21.15.103:0/273451937 wait complete. 2024-05-22T23:53:43.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.959+0000 7f9fb5000700 1 Processor -- start 2024-05-22T23:53:43.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.959+0000 7f9fb5000700 1 -- start start 2024-05-22T23:53:43.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.959+0000 7f9fb5000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb0104500 0x7f9fb019a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:53:43.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.959+0000 7f9fb5000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fb0105700 0x7f9fb019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:53:43.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.959+0000 7f9fb5000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb01060a0 0x7f9fb01a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:53:43.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.959+0000 7f9fb5000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb01100d0 con 0x7f9fb01060a0 2024-05-22T23:53:43.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.959+0000 7f9fb5000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb010ff50 con 0x7f9fb0105700 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.959+0000 7f9fb5000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fb0110250 con 0x7f9fb0104500 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9fafe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb01060a0 0x7f9fb01a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9faf400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb0104500 0x7f9fb019a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9fafe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb01060a0 0x7f9fb01a1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:50552/0 (socket says 172.21.15.103:50552) 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9fafe00700 1 -- 172.21.15.103:0/346374150 learned_addr learned my addr 172.21.15.103:0/346374150 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9faea00700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fb0105700 0x7f9fb019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9faf400700 1 -- 172.21.15.103:0/346374150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fb0105700 msgr2=0x7f9fb019abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9faf400700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fb0105700 0x7f9fb019abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9faf400700 1 -- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb01060a0 msgr2=0x7f9fb01a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9faf400700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb01060a0 0x7f9fb01a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9faf400700 1 -- 172.21.15.103:0/346374150 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fb01a22f0 con 0x7f9fb0104500 2024-05-22T23:53:43.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.960+0000 7f9fafe00700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb01060a0 0x7f9fb01a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:53:43.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.961+0000 7f9faf400700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb0104500 0x7f9fb019a690 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000e9f0 tx=0x7f9fa000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:53:43.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.961+0000 7f9fa7e00700 1 -- 172.21.15.103:0/346374150 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0004220 con 0x7f9fb0104500 2024-05-22T23:53:43.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.961+0000 7f9fa7e00700 1 -- 172.21.15.103:0/346374150 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fa00043c0 con 0x7f9fb0104500 2024-05-22T23:53:43.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.961+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fb01a25e0 con 0x7f9fb0104500 2024-05-22T23:53:43.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.961+0000 7f9fa7e00700 1 -- 172.21.15.103:0/346374150 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa0003cb0 con 0x7f9fb0104500 2024-05-22T23:53:43.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.961+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fb01a2b70 con 0x7f9fb0104500 2024-05-22T23:53:43.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.962+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fb0069050 con 0x7f9fb0104500 2024-05-22T23:53:43.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.964+0000 7f9fa7e00700 1 -- 172.21.15.103:0/346374150 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9fa000c7c0 con 0x7f9fb0104500 2024-05-22T23:53:43.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.966+0000 7f9fa7e00700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9f940a2ca0 0x7f9f940a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:53:43.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.966+0000 7f9fa7e00700 1 -- 172.21.15.103:0/346374150 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4533+0+0 (secure 0 0 0) 0x7f9fa00ce470 con 0x7f9fb0104500 2024-05-22T23:53:43.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.966+0000 7f9faea00700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9f940a2ca0 0x7f9f940a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:53:43.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.967+0000 7f9faea00700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9f940a2ca0 0x7f9f940a5160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c000c00 tx=0x7f9f9c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:53:43.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:43.968+0000 7f9fa7e00700 1 -- 172.21.15.103:0/346374150 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9fa009c280 con 0x7f9fb0104500 2024-05-22T23:53:44.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:53:44.199+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f9fb00627a0 con 0x7f9f940a2ca0 2024-05-22T23:53:44.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:44 smithi105 ceph-mon[36426]: osdmap e44: 6 total, 6 up, 6 in 2024-05-22T23:53:44.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:44 smithi105 ceph-mon[36426]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:44.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:53:44.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:53:44.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:44.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:44 smithi103 ceph-mon[31370]: osdmap e44: 6 total, 6 up, 6 in 2024-05-22T23:53:44.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:44 smithi103 ceph-mon[31370]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:44.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:53:44.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:53:44.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:44.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:44 smithi174 ceph-mon[36179]: osdmap e44: 6 total, 6 up, 6 in 2024-05-22T23:53:44.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:44 smithi174 ceph-mon[36179]: pgmap v161: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:44.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:53:44.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:53:44.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:45.487 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:45 smithi103 ceph-mon[31370]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:53:45.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:45 smithi105 ceph-mon[36426]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:53:45.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:45 smithi174 ceph-mon[36179]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:53:46.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:46 smithi105 ceph-mon[36426]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:46.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:46 smithi103 ceph-mon[31370]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:46 smithi174 ceph-mon[36179]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-05-22T23:53:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:48 smithi105 ceph-mon[36426]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-22T23:53:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:48 smithi105 ceph-mon[36426]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-22T23:53:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:48 smithi105 ceph-mon[36426]: Cluster is now healthy 2024-05-22T23:53:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:48 smithi103 ceph-mon[31370]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-22T23:53:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:48 smithi103 ceph-mon[31370]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-22T23:53:48.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:48 smithi103 ceph-mon[31370]: Cluster is now healthy 2024-05-22T23:53:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:48 smithi174 ceph-mon[36179]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-05-22T23:53:48.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:48 smithi174 ceph-mon[36179]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-22T23:53:48.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:48 smithi174 ceph-mon[36179]: Cluster is now healthy 2024-05-22T23:53:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:49 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/128484154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9494aa47-b660-41dd-af98-e7c55899525c"}]: dispatch 2024-05-22T23:53:49.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:49 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/128484154' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9494aa47-b660-41dd-af98-e7c55899525c"}]': finished 2024-05-22T23:53:49.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:49 smithi103 ceph-mon[31370]: osdmap e45: 7 total, 6 up, 7 in 2024-05-22T23:53:49.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:53:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:49 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/128484154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9494aa47-b660-41dd-af98-e7c55899525c"}]: dispatch 2024-05-22T23:53:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:49 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/128484154' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9494aa47-b660-41dd-af98-e7c55899525c"}]': finished 2024-05-22T23:53:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:49 smithi105 ceph-mon[36426]: osdmap e45: 7 total, 6 up, 7 in 2024-05-22T23:53:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:49 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:53:49.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:49 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/128484154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9494aa47-b660-41dd-af98-e7c55899525c"}]: dispatch 2024-05-22T23:53:49.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:49 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/128484154' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9494aa47-b660-41dd-af98-e7c55899525c"}]': finished 2024-05-22T23:53:49.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:49 smithi174 ceph-mon[36179]: osdmap e45: 7 total, 6 up, 7 in 2024-05-22T23:53:49.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:49 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:53:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:50 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/25052728' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:53:50.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:50 smithi105 ceph-mon[36426]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-22T23:53:50.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:50 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/25052728' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:53:50.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:50 smithi103 ceph-mon[31370]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-22T23:53:50.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:50 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/25052728' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:53:50.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:50 smithi174 ceph-mon[36179]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-22T23:53:53.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:53 smithi174 ceph-mon[36179]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-22T23:53:53.534 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:53 smithi105 ceph-mon[36426]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-22T23:53:53.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:53 smithi103 ceph-mon[31370]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-22T23:53:55.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:55 smithi103 ceph-mon[31370]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-22T23:53:55.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:55 smithi174 ceph-mon[36179]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-22T23:53:55.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:55 smithi105 ceph-mon[36426]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-22T23:53:57.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:57 smithi103 ceph-mon[31370]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-22T23:53:57.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:57 smithi174 ceph-mon[36179]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-22T23:53:57.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:57 smithi105 ceph-mon[36426]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-22T23:53:58.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:58 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-22T23:53:58.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:58 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:58.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:58 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-22T23:53:58.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:58 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:58.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:58 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-22T23:53:58.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:58 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:53:59.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:59 smithi103 ceph-mon[31370]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:53:59.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:53:59 smithi103 ceph-mon[31370]: Deploying daemon osd.6 on smithi103 2024-05-22T23:53:59.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:59 smithi174 ceph-mon[36179]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:53:59.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:53:59 smithi174 ceph-mon[36179]: Deploying daemon osd.6 on smithi103 2024-05-22T23:53:59.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:59 smithi105 ceph-mon[36426]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:53:59.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:53:59 smithi105 ceph-mon[36426]: Deploying daemon osd.6 on smithi103 2024-05-22T23:54:00.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:00 smithi103 ceph-mon[31370]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:00.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:00 smithi105 ceph-mon[36426]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:00.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:00 smithi174 ceph-mon[36179]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:03.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:03 smithi103 ceph-mon[31370]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:03.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:03 smithi174 ceph-mon[36179]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:03 smithi105 ceph-mon[36426]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:05.255 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:05 smithi103 ceph-mon[31370]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:05.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:05 smithi174 ceph-mon[36179]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:05.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:05 smithi105 ceph-mon[36426]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:07.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:07 smithi174 ceph-mon[36179]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:07.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:07.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:07.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:07.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:07 smithi105 ceph-mon[36426]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:07.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:07.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:07.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:07.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:07 smithi103 ceph-mon[31370]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:07.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:07.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:07.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:08.351 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:08 smithi103 ceph-mon[31370]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:08.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:08 smithi174 ceph-mon[36179]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:08.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:08 smithi105 ceph-mon[36426]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:11 smithi103 ceph-mon[31370]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:11 smithi103 ceph-mon[31370]: from='osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-22T23:54:11.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:11 smithi103 ceph-mon[31370]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-22T23:54:11.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:11 smithi174 ceph-mon[36179]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:11.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:11 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:11.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:11 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:11.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:11 smithi174 ceph-mon[36179]: from='osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-22T23:54:11.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:11 smithi174 ceph-mon[36179]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-22T23:54:11.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:11 smithi105 ceph-mon[36426]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:11.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:11 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:11.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:11 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:11.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:11 smithi105 ceph-mon[36426]: from='osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-22T23:54:11.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:11 smithi105 ceph-mon[36426]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-22T23:54:11.956 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 6 on host 'smithi103' 2024-05-22T23:54:11.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.949+0000 7f9fa7e00700 1 -- 172.21.15.103:0/346374150 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9fb00627a0 con 0x7f9f940a2ca0 2024-05-22T23:54:11.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.952+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9f940a2ca0 msgr2=0x7f9f940a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:11.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.952+0000 7f9fb5000700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9f940a2ca0 0x7f9f940a5160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c000c00 tx=0x7f9f9c005fb0 comp rx=0 tx=0).stop 2024-05-22T23:54:11.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb0104500 msgr2=0x7f9fb019a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:11.958 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb0104500 0x7f9fb019a690 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f9fa000e9f0 tx=0x7f9fa000ed00 comp rx=0 tx=0).stop 2024-05-22T23:54:11.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 shutdown_connections 2024-05-22T23:54:11.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9f940a2ca0 0x7f9f940a5160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:11.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fb01060a0 0x7f9fb01a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:11.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fb0104500 0x7f9fb019a690 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:11.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 --2- 172.21.15.103:0/346374150 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fb0105700 0x7f9fb019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:11.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 >> 172.21.15.103:0/346374150 conn(0x7f9fb00ffaa0 msgr2=0x7f9fb0108a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:54:11.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 shutdown_connections 2024-05-22T23:54:11.959 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:54:11.953+0000 7f9fb5000700 1 -- 172.21.15.103:0/346374150 wait complete. 2024-05-22T23:54:12.577 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: osdmap e46: 7 total, 6 up, 7 in 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:12.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-22T23:54:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: osdmap e46: 7 total, 6 up, 7 in 2024-05-22T23:54:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:54:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:54:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:12.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-22T23:54:12.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: osdmap e46: 7 total, 6 up, 7 in 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-22T23:54:12.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:12.883 DEBUG:teuthology.orchestra.run.smithi103:osd.6> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.6.service 2024-05-22T23:54:12.888 INFO:tasks.cephadm:Deploying osd.7 on smithi105 with /dev/vg_nvme/lv_2... 2024-05-22T23:54:12.888 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-22T23:54:13.135 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-22T23:54:13.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:13 smithi105 ceph-mon[36426]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:54:13.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:13 smithi105 ceph-mon[36426]: osdmap e47: 7 total, 6 up, 7 in 2024-05-22T23:54:13.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:13 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:13.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:13 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:13.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:13 smithi103 ceph-mon[31370]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:54:13.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:13 smithi103 ceph-mon[31370]: osdmap e47: 7 total, 6 up, 7 in 2024-05-22T23:54:13.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:13 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:13.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:13 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:13.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:13 smithi174 ceph-mon[36179]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:54:13.871 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:13 smithi174 ceph-mon[36179]: osdmap e47: 7 total, 6 up, 7 in 2024-05-22T23:54:13.871 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:13 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:13.871 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:13 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:14.725 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:14 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:54:14.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:14 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:54:14.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:14 smithi103 ceph-mon[31370]: osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] boot 2024-05-22T23:54:14.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:14 smithi103 ceph-mon[31370]: osdmap e48: 7 total, 7 up, 7 in 2024-05-22T23:54:14.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:14.726 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:14 smithi103 ceph-mon[31370]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:14.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:14 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:54:14.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:14 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:54:14.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:14 smithi105 ceph-mon[36426]: osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] boot 2024-05-22T23:54:14.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:14 smithi105 ceph-mon[36426]: osdmap e48: 7 total, 7 up, 7 in 2024-05-22T23:54:14.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:14.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:14 smithi105 ceph-mon[36426]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:14.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:14 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:54:14.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:14 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:54:14.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:14 smithi174 ceph-mon[36179]: osd.6 [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] boot 2024-05-22T23:54:14.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:14 smithi174 ceph-mon[36179]: osdmap e48: 7 total, 7 up, 7 in 2024-05-22T23:54:14.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-22T23:54:14.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:14 smithi174 ceph-mon[36179]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:15.184 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-22T23:54:15.185 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-22T23:54:15.185 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-22T23:54:15.185 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-22T23:54:15.185 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113095 s, 927 MB/s 2024-05-22T23:54:15.185 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-22T23:54:15.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:15 smithi105 ceph-mon[36426]: osdmap e49: 7 total, 7 up, 7 in 2024-05-22T23:54:15.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:15 smithi103 ceph-mon[31370]: osdmap e49: 7 total, 7 up, 7 in 2024-05-22T23:54:15.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:15 smithi174 ceph-mon[36179]: osdmap e49: 7 total, 7 up, 7 in 2024-05-22T23:54:15.984 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2024-05-22T23:54:16.228 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-22T23:54:16.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:16 smithi105 ceph-mon[36426]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:16.824 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:16 smithi103 ceph-mon[31370]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:16.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:16 smithi174 ceph-mon[36179]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:17.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 -- 172.21.15.105:0/95986103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca88102e60 msgr2=0x7fca88109710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:17.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 --2- 172.21.15.105:0/95986103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca88102e60 0x7fca88109710 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fca8000b0d0 tx=0x7fca8000b3e0 comp rx=0 tx=0).stop 2024-05-22T23:54:17.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 -- 172.21.15.105:0/95986103 shutdown_connections 2024-05-22T23:54:17.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 --2- 172.21.15.105:0/95986103 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca88102e60 0x7fca88109710 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:17.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 --2- 172.21.15.105:0/95986103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca881012c0 0x7fca881016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:17.908 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 --2- 172.21.15.105:0/95986103 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca881024c0 0x7fca88102920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 -- 172.21.15.105:0/95986103 >> 172.21.15.105:0/95986103 conn(0x7fca880fc860 msgr2=0x7fca880feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 -- 172.21.15.105:0/95986103 shutdown_connections 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.900+0000 7fca8d485700 1 -- 172.21.15.105:0/95986103 wait complete. 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca8d485700 1 Processor -- start 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca8d485700 1 -- start start 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca8d485700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca881012c0 0x7fca8806aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca8d485700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca881024c0 0x7fca8819d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca8d485700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca88102e60 0x7fca8819f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca8d485700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fca8810cea0 con 0x7fca881012c0 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca8d485700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fca8810cd20 con 0x7fca881024c0 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca8d485700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca8810d020 con 0x7fca88102e60 2024-05-22T23:54:17.909 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca867fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca881024c0 0x7fca8819d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca867fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca881024c0 0x7fca8819d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50352/0 (socket says 172.21.15.105:50352) 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca867fc700 1 -- 172.21.15.105:0/3757943305 learned_addr learned my addr 172.21.15.105:0/3757943305 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.901+0000 7fca877fe700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca88102e60 0x7fca8819f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca86ffd700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca881012c0 0x7fca8806aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca867fc700 1 -- 172.21.15.105:0/3757943305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca88102e60 msgr2=0x7fca8819f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca867fc700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca88102e60 0x7fca8819f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca867fc700 1 -- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca881012c0 msgr2=0x7fca8806aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca867fc700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca881012c0 0x7fca8806aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca867fc700 1 -- 172.21.15.105:0/3757943305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca881a0220 con 0x7fca881024c0 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca86ffd700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca881012c0 0x7fca8806aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca867fc700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca881024c0 0x7fca8819d310 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c920 tx=0x7fca7c00cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:54:17.910 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca6ffff700 1 -- 172.21.15.105:0/3757943305 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c007b00 con 0x7fca881024c0 2024-05-22T23:54:17.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca6ffff700 1 -- 172.21.15.105:0/3757943305 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca7c004d10 con 0x7fca881024c0 2024-05-22T23:54:17.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca881a0510 con 0x7fca881024c0 2024-05-22T23:54:17.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca6ffff700 1 -- 172.21.15.105:0/3757943305 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c005680 con 0x7fca881024c0 2024-05-22T23:54:17.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.902+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca881a0a50 con 0x7fca881024c0 2024-05-22T23:54:17.911 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.903+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca88069050 con 0x7fca881024c0 2024-05-22T23:54:17.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.904+0000 7fca6ffff700 1 -- 172.21.15.105:0/3757943305 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fca7c004750 con 0x7fca881024c0 2024-05-22T23:54:17.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.905+0000 7fca6ffff700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fca700a2c20 0x7fca700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:54:17.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.905+0000 7fca6ffff700 1 -- 172.21.15.105:0/3757943305 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 4965+0+0 (secure 0 0 0) 0x7fca7c0cdec0 con 0x7fca881024c0 2024-05-22T23:54:17.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.905+0000 7fca86ffd700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fca700a2c20 0x7fca700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:54:17.912 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.905+0000 7fca86ffd700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fca700a2c20 0x7fca700a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca78005fd0 tx=0x7fca78005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:54:17.914 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:17.909+0000 7fca6ffff700 1 -- 172.21.15.105:0/3757943305 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fca7c014030 con 0x7fca881024c0 2024-05-22T23:54:18.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:18.117+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fca880627a0 con 0x7fca700a2c20 2024-05-22T23:54:19.009 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi103 2024-05-22T23:54:19.009 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.009 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.009 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.009 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.009 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.009 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi103 to 5696M 2024-05-22T23:54:19.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:19.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:19.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:19.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:54:19.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:54:19.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:18 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:19.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi103 2024-05-22T23:54:19.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi103 to 5696M 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:54:19.083 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:18 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:19.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi103 2024-05-22T23:54:19.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi103 to 5696M 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:54:19.120 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:18 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:20.010 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:19 smithi105 ceph-mon[36426]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:54:20.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:19 smithi103 ceph-mon[31370]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:54:20.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:19 smithi174 ceph-mon[36179]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:54:21.200 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:20 smithi103 ceph-mon[31370]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:21.263 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:20 smithi105 ceph-mon[36426]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:21.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:20 smithi174 ceph-mon[36179]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:22.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:21 smithi105 ceph-mon[36426]: from='client.? 172.21.15.105:0/240685549' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]: dispatch 2024-05-22T23:54:22.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:21 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]: dispatch 2024-05-22T23:54:22.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:21 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]': finished 2024-05-22T23:54:22.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:21 smithi105 ceph-mon[36426]: osdmap e50: 8 total, 7 up, 8 in 2024-05-22T23:54:22.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:22.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:21 smithi105 ceph-mon[36426]: from='client.? 172.21.15.105:0/1963650207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:54:22.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:21 smithi103 ceph-mon[31370]: from='client.? 172.21.15.105:0/240685549' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]: dispatch 2024-05-22T23:54:22.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:21 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]: dispatch 2024-05-22T23:54:22.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:21 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]': finished 2024-05-22T23:54:22.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:21 smithi103 ceph-mon[31370]: osdmap e50: 8 total, 7 up, 8 in 2024-05-22T23:54:22.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:22.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:21 smithi103 ceph-mon[31370]: from='client.? 172.21.15.105:0/1963650207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:54:22.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:21 smithi174 ceph-mon[36179]: from='client.? 172.21.15.105:0/240685549' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]: dispatch 2024-05-22T23:54:22.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:21 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]: dispatch 2024-05-22T23:54:22.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:21 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05"}]': finished 2024-05-22T23:54:22.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:21 smithi174 ceph-mon[36179]: osdmap e50: 8 total, 7 up, 8 in 2024-05-22T23:54:22.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:22.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:21 smithi174 ceph-mon[36179]: from='client.? 172.21.15.105:0/1963650207' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:54:23.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:22 smithi105 ceph-mon[36426]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:23.234 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:22 smithi103 ceph-mon[31370]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:23.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:22 smithi174 ceph-mon[36179]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:25.263 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:25 smithi105 ceph-mon[36426]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:25.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:25 smithi103 ceph-mon[31370]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:25.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:25 smithi174 ceph-mon[36179]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:27.238 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:27 smithi105 ceph-mon[36426]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:27.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:27 smithi174 ceph-mon[36179]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:27.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:27 smithi103 ceph-mon[31370]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:28.419 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:28 smithi105 ceph-mon[36426]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:28.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:28 smithi103 ceph-mon[31370]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:28.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:28 smithi174 ceph-mon[36179]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:29.406 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-22T23:54:29.407 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:29.407 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:29 smithi105 ceph-mon[36426]: Deploying daemon osd.7 on smithi105 2024-05-22T23:54:29.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-22T23:54:29.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:29.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:29 smithi103 ceph-mon[31370]: Deploying daemon osd.7 on smithi105 2024-05-22T23:54:29.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-22T23:54:29.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:29.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:29 smithi174 ceph-mon[36179]: Deploying daemon osd.7 on smithi105 2024-05-22T23:54:30.425 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:30 smithi105 ceph-mon[36426]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:30.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:30 smithi103 ceph-mon[31370]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:30.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:30 smithi174 ceph-mon[36179]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:33.252 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:33 smithi105 ceph-mon[36426]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:33.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:33 smithi174 ceph-mon[36179]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:33 smithi103 ceph-mon[31370]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:35.148 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:35 smithi105 ceph-mon[36426]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:35.148 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:35.148 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:35.148 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:35 smithi174 ceph-mon[36179]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:35.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:35 smithi103 ceph-mon[31370]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:35.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:35.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:35.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:36.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:36 smithi103 ceph-mon[31370]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:36.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:36 smithi105 ceph-mon[36426]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:36.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:36 smithi174 ceph-mon[36179]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:38.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.056+0000 7fca6ffff700 1 -- 172.21.15.105:0/3757943305 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fca880627a0 con 0x7fca700a2c20 2024-05-22T23:54:38.061 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 7 on host 'smithi105' 2024-05-22T23:54:38.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fca700a2c20 msgr2=0x7fca700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:38.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fca700a2c20 0x7fca700a50e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fca78005fd0 tx=0x7fca78005f00 comp rx=0 tx=0).stop 2024-05-22T23:54:38.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca881024c0 msgr2=0x7fca8819d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:38.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca881024c0 0x7fca8819d310 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00c920 tx=0x7fca7c00cc30 comp rx=0 tx=0).stop 2024-05-22T23:54:38.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 shutdown_connections 2024-05-22T23:54:38.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fca700a2c20 0x7fca700a50e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:38.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca88102e60 0x7fca8819f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:38.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fca881012c0 0x7fca8806aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:38.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 --2- 172.21.15.105:0/3757943305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fca881024c0 0x7fca8819d310 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:38.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 >> 172.21.15.105:0/3757943305 conn(0x7fca880fc860 msgr2=0x7fca881057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:54:38.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 shutdown_connections 2024-05-22T23:54:38.064 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:54:38.058+0000 7fca8d485700 1 -- 172.21.15.105:0/3757943305 wait complete. 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-22T23:54:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:38 smithi105 ceph-mon[36426]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:38.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-22T23:54:38.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:38 smithi103 ceph-mon[31370]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-22T23:54:38.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:38.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:38.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:54:38.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:38.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:38.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-22T23:54:38.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:38 smithi174 ceph-mon[36179]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-22T23:54:38.637 DEBUG:teuthology.orchestra.run.smithi105:osd.7> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.7.service 2024-05-22T23:54:38.639 INFO:tasks.cephadm:Deploying osd.8 on smithi174 with /dev/vg_nvme/lv_2... 2024-05-22T23:54:38.640 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-22T23:54:38.882 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:54:40.156 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:40 smithi105 ceph-mon[36426]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-22T23:54:40.156 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:40 smithi105 ceph-mon[36426]: from='osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:54:40.156 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:40 smithi105 ceph-mon[36426]: osdmap e51: 8 total, 7 up, 8 in 2024-05-22T23:54:40.156 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:40 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:40.156 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:40 smithi105 ceph-mon[36426]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:54:40.264 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:40 smithi174 ceph-mon[36179]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-22T23:54:40.265 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:40 smithi174 ceph-mon[36179]: from='osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:54:40.265 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:40 smithi174 ceph-mon[36179]: osdmap e51: 8 total, 7 up, 8 in 2024-05-22T23:54:40.265 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:40.265 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:40 smithi174 ceph-mon[36179]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:54:40.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:40 smithi103 ceph-mon[31370]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-22T23:54:40.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:40 smithi103 ceph-mon[31370]: from='osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:54:40.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:40 smithi103 ceph-mon[31370]: osdmap e51: 8 total, 7 up, 8 in 2024-05-22T23:54:40.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:40.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:40 smithi103 ceph-mon[31370]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:54:40.802 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-22T23:54:40.802 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-22T23:54:40.802 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-05-22T23:54:40.803 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-05-22T23:54:40.803 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113459 s, 924 MB/s 2024-05-22T23:54:40.803 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-05-22T23:54:40.810 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 22 23:54:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-22T23:54:40.507+0000 7f28906dd700 -1 osd.7 0 waiting for initial osdmap 2024-05-22T23:54:40.810 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 22 23:54:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-22T23:54:40.520+0000 7f2887cc7700 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:54:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:41 smithi105 ceph-mon[36426]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:54:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:41 smithi105 ceph-mon[36426]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:41 smithi105 ceph-mon[36426]: osdmap e52: 8 total, 7 up, 8 in 2024-05-22T23:54:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:41 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:41 smithi105 ceph-mon[36426]: from='osd.7 ' entity='osd.7' 2024-05-22T23:54:41.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:41 smithi103 ceph-mon[31370]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:54:41.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:41 smithi103 ceph-mon[31370]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:41.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:41 smithi103 ceph-mon[31370]: osdmap e52: 8 total, 7 up, 8 in 2024-05-22T23:54:41.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:41.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:41 smithi103 ceph-mon[31370]: from='osd.7 ' entity='osd.7' 2024-05-22T23:54:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:41 smithi174 ceph-mon[36179]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:54:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:41 smithi174 ceph-mon[36179]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:41 smithi174 ceph-mon[36179]: osdmap e52: 8 total, 7 up, 8 in 2024-05-22T23:54:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:41 smithi174 ceph-mon[36179]: from='osd.7 ' entity='osd.7' 2024-05-22T23:54:41.496 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_2 2024-05-22T23:54:41.734 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] boot 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: osdmap e53: 8 total, 8 up, 8 in 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:42.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:42.333 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:42 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:54:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:54:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] boot 2024-05-22T23:54:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: osdmap e53: 8 total, 8 up, 8 in 2024-05-22T23:54:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:42.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:42.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:42 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:54:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:54:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: osd.7 [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] boot 2024-05-22T23:54:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: osdmap e53: 8 total, 8 up, 8 in 2024-05-22T23:54:42.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:54:42.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:42 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:54:43.247 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:43 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi105 2024-05-22T23:54:43.247 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:43 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-22T23:54:43.247 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:43 smithi174 ceph-mon[36179]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:43.247 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:43 smithi174 ceph-mon[36179]: osdmap e54: 8 total, 8 up, 8 in 2024-05-22T23:54:43.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:43 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi105 2024-05-22T23:54:43.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:43 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-22T23:54:43.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:43 smithi103 ceph-mon[31370]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:43.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:43 smithi103 ceph-mon[31370]: osdmap e54: 8 total, 8 up, 8 in 2024-05-22T23:54:43.470 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 -- 172.21.15.174:0/3980124483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 msgr2=0x7f5af4102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:43.470 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 --2- 172.21.15.174:0/3980124483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 0x7f5af4102940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009990 tx=0x7f5ae0009ca0 comp rx=0 tx=0).stop 2024-05-22T23:54:43.470 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 -- 172.21.15.174:0/3980124483 shutdown_connections 2024-05-22T23:54:43.470 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 --2- 172.21.15.174:0/3980124483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5af4102e80 0x7f5af4109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:43.470 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 --2- 172.21.15.174:0/3980124483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af41012e0 0x7f5af4101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:43.470 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 --2- 172.21.15.174:0/3980124483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 0x7f5af4102940 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 -- 172.21.15.174:0/3980124483 >> 172.21.15.174:0/3980124483 conn(0x7f5af40fc8c0 msgr2=0x7f5af40fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 -- 172.21.15.174:0/3980124483 shutdown_connections 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.462+0000 7f5afabe8700 1 -- 172.21.15.174:0/3980124483 wait complete. 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5afabe8700 1 Processor -- start 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5afabe8700 1 -- start start 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5afabe8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af41012e0 0x7f5af4197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5afabe8700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 0x7f5af4197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5afabe8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5af4102e80 0x7f5af419eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5afabe8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5af410cf60 con 0x7f5af41012e0 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5afabe8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5af410cde0 con 0x7f5af4102e80 2024-05-22T23:54:43.471 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5afabe8700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5af410d0e0 con 0x7f5af41024e0 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5af3fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 0x7f5af4197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5af3fff700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 0x7f5af4197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:49400/0 (socket says 172.21.15.174:49400) 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.463+0000 7f5af3fff700 1 -- 172.21.15.174:0/566656294 learned_addr learned my addr 172.21.15.174:0/566656294 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af9185700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5af4102e80 0x7f5af419eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af3fff700 1 -- 172.21.15.174:0/566656294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5af4102e80 msgr2=0x7f5af419eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af3fff700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5af4102e80 0x7f5af419eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af3fff700 1 -- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af41012e0 msgr2=0x7f5af4197550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af3fff700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af41012e0 0x7f5af4197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af3fff700 1 -- 172.21.15.174:0/566656294 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af419f1b0 con 0x7f5af41024e0 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af3fff700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 0x7f5af4197a90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0004790 tx=0x7f5ae0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af1ffb700 1 -- 172.21.15.174:0/566656294 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae0027070 con 0x7f5af41024e0 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af1ffb700 1 -- 172.21.15.174:0/566656294 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ae0004210 con 0x7f5af41024e0 2024-05-22T23:54:43.472 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af419f440 con 0x7f5af41024e0 2024-05-22T23:54:43.473 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5af1ffb700 1 -- 172.21.15.174:0/566656294 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae000f460 con 0x7f5af41024e0 2024-05-22T23:54:43.473 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.464+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af419f910 con 0x7f5af41024e0 2024-05-22T23:54:43.473 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.465+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af4069050 con 0x7f5af41024e0 2024-05-22T23:54:43.473 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.466+0000 7f5af1ffb700 1 -- 172.21.15.174:0/566656294 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5ae002a070 con 0x7f5af41024e0 2024-05-22T23:54:43.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.467+0000 7f5af1ffb700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5ae40a2c20 0x7f5ae40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:54:43.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.467+0000 7f5af1ffb700 1 -- 172.21.15.174:0/566656294 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5397+0+0 (secure 0 0 0) 0x7f5ae00cf7c0 con 0x7f5af41024e0 2024-05-22T23:54:43.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.468+0000 7f5af8984700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5ae40a2c20 0x7f5ae40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:54:43.474 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.468+0000 7f5af8984700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5ae40a2c20 0x7f5ae40a50e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5af4102340 tx=0x7f5ae8006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:54:43.476 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.471+0000 7f5af1ffb700 1 -- 172.21.15.174:0/566656294 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5ae0013070 con 0x7f5af41024e0 2024-05-22T23:54:43.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:43 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi105 2024-05-22T23:54:43.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:43 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi105 to 5696M 2024-05-22T23:54:43.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:43 smithi105 ceph-mon[36426]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-22T23:54:43.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:43 smithi105 ceph-mon[36426]: osdmap e54: 8 total, 8 up, 8 in 2024-05-22T23:54:43.679 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:54:43.673+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f5af40627a0 con 0x7f5ae40a2c20 2024-05-22T23:54:44.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:44 smithi103 ceph-mon[31370]: osdmap e55: 8 total, 8 up, 8 in 2024-05-22T23:54:44.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:44 smithi103 ceph-mon[31370]: osdmap e56: 8 total, 8 up, 8 in 2024-05-22T23:54:44.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:54:44.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:54:44.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:44 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:44.333 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:44 smithi174 ceph-mon[36179]: osdmap e55: 8 total, 8 up, 8 in 2024-05-22T23:54:44.333 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:44 smithi174 ceph-mon[36179]: osdmap e56: 8 total, 8 up, 8 in 2024-05-22T23:54:44.333 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:54:44.333 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:54:44.333 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:44 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:44.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:44 smithi105 ceph-mon[36426]: osdmap e55: 8 total, 8 up, 8 in 2024-05-22T23:54:44.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:44 smithi105 ceph-mon[36426]: osdmap e56: 8 total, 8 up, 8 in 2024-05-22T23:54:44.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:54:44.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:54:44.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:44 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:45.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:45 smithi103 ceph-mon[31370]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:54:45.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:45 smithi103 ceph-mon[31370]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:45.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:45 smithi103 ceph-mon[31370]: osdmap e57: 8 total, 8 up, 8 in 2024-05-22T23:54:45.335 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:45 smithi174 ceph-mon[36179]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:54:45.336 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:45 smithi174 ceph-mon[36179]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:45.336 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:45 smithi174 ceph-mon[36179]: osdmap e57: 8 total, 8 up, 8 in 2024-05-22T23:54:45.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:45 smithi105 ceph-mon[36426]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:54:45.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:45 smithi105 ceph-mon[36426]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:45.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:45 smithi105 ceph-mon[36426]: osdmap e57: 8 total, 8 up, 8 in 2024-05-22T23:54:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:47 smithi174 ceph-mon[36179]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:47 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/608951367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]: dispatch 2024-05-22T23:54:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:47 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]: dispatch 2024-05-22T23:54:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:47 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]': finished 2024-05-22T23:54:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:47 smithi174 ceph-mon[36179]: osdmap e58: 9 total, 8 up, 9 in 2024-05-22T23:54:47.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:47 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:54:47.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:47 smithi105 ceph-mon[36426]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:47.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:47 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/608951367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]: dispatch 2024-05-22T23:54:47.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:47 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]: dispatch 2024-05-22T23:54:47.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:47 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]': finished 2024-05-22T23:54:47.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:47 smithi105 ceph-mon[36426]: osdmap e58: 9 total, 8 up, 9 in 2024-05-22T23:54:47.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:47 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:54:47.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:47 smithi103 ceph-mon[31370]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:47.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:47 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/608951367' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]: dispatch 2024-05-22T23:54:47.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:47 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]: dispatch 2024-05-22T23:54:47.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:47 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad862c3c-61c3-46da-adc1-008a4d5f5fce"}]': finished 2024-05-22T23:54:47.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:47 smithi103 ceph-mon[31370]: osdmap e58: 9 total, 8 up, 9 in 2024-05-22T23:54:47.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:47 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:54:48.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:48 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/3270717970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:54:48.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:48 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/3270717970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:54:48.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:48 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/3270717970' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:54:49.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:49 smithi174 ceph-mon[36179]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:49 smithi105 ceph-mon[36426]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:49.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:49 smithi103 ceph-mon[31370]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:51.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:51 smithi174 ceph-mon[36179]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:51.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:51 smithi105 ceph-mon[36426]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:51.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:51 smithi103 ceph-mon[31370]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:53.362 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:53 smithi105 ceph-mon[36426]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:53.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:53 smithi174 ceph-mon[36179]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:53.384 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:53 smithi103 ceph-mon[31370]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-22T23:54:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:54.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-22T23:54:54.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:54.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-22T23:54:54.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:54:55.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:55 smithi103 ceph-mon[31370]: Deploying daemon osd.8 on smithi174 2024-05-22T23:54:55.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:55 smithi103 ceph-mon[31370]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:55.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:55 smithi174 ceph-mon[36179]: Deploying daemon osd.8 on smithi174 2024-05-22T23:54:55.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:55 smithi174 ceph-mon[36179]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:55.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:55 smithi105 ceph-mon[36426]: Deploying daemon osd.8 on smithi174 2024-05-22T23:54:55.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:55 smithi105 ceph-mon[36426]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:56.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:56 smithi105 ceph-mon[36426]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:56.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:56 smithi103 ceph-mon[31370]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:56.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:56 smithi174 ceph-mon[36179]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:59.332 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:54:59 smithi174 ceph-mon[36179]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:54:59 smithi105 ceph-mon[36426]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:54:59.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:54:59 smithi103 ceph-mon[31370]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:00.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:00 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:00.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:00 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:00.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:00 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:00.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:00 smithi174 ceph-mon[36179]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:01.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:00 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:01.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:00 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:01.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:00 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:01.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:00 smithi105 ceph-mon[36426]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:01.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:00 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:01.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:00 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:01.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:00 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:01.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:00 smithi103 ceph-mon[31370]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:03.215 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:03 smithi174 ceph-mon[36179]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:03.313 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 8 on host 'smithi174' 2024-05-22T23:55:03.313 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.308+0000 7f5af1ffb700 1 -- 172.21.15.174:0/566656294 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5af40627a0 con 0x7f5ae40a2c20 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5ae40a2c20 msgr2=0x7f5ae40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5ae40a2c20 0x7f5ae40a50e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5af4102340 tx=0x7f5ae8006d20 comp rx=0 tx=0).stop 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 msgr2=0x7f5af4197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 0x7f5af4197a90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0004790 tx=0x7f5ae0009580 comp rx=0 tx=0).stop 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 shutdown_connections 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5af4102e80 0x7f5af419eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5ae40a2c20 0x7f5ae40a50e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5af41012e0 0x7f5af4197550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:03.315 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 --2- 172.21.15.174:0/566656294 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5af41024e0 0x7f5af4197a90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:03.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 >> 172.21.15.174:0/566656294 conn(0x7f5af40fc8c0 msgr2=0x7f5af41057e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:55:03.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 shutdown_connections 2024-05-22T23:55:03.316 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:55:03.310+0000 7f5afabe8700 1 -- 172.21.15.174:0/566656294 wait complete. 2024-05-22T23:55:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:03 smithi105 ceph-mon[36426]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:03.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:03 smithi103 ceph-mon[31370]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:04.021 DEBUG:teuthology.orchestra.run.smithi174:osd.8> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.8.service 2024-05-22T23:55:04.027 INFO:tasks.cephadm:Deploying osd.9 on smithi103 with /dev/vg_nvme/lv_1... 2024-05-22T23:55:04.027 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-22T23:55:04.450 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:04 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.450 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:04 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.450 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:04 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:04 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:04 smithi103 ceph-mon[31370]: from='osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-22T23:55:04.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:04 smithi103 ceph-mon[31370]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-22T23:55:04.451 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:04 smithi103 ceph-mon[31370]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:04.474 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:55:04.556 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:04 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.556 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:04 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.556 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:04 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.556 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:04 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.556 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:04 smithi174 ceph-mon[36179]: from='osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-22T23:55:04.557 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:04 smithi174 ceph-mon[36179]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-22T23:55:04.557 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:04 smithi174 ceph-mon[36179]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:04 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:04 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:04 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:04 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:04 smithi105 ceph-mon[36426]: from='osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-22T23:55:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:04 smithi105 ceph-mon[36426]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-22T23:55:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:04 smithi105 ceph-mon[36426]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:05.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:05 smithi174 ceph-mon[36179]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-22T23:55:05.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:05 smithi174 ceph-mon[36179]: from='osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:55:05.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:05 smithi174 ceph-mon[36179]: osdmap e59: 9 total, 8 up, 9 in 2024-05-22T23:55:05.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:05.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:05 smithi174 ceph-mon[36179]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:55:05.498 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 22 23:55:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-22T23:55:05.346+0000 7f81cbecf700 -1 osd.8 0 waiting for initial osdmap 2024-05-22T23:55:05.498 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 22 23:55:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-22T23:55:05.357+0000 7f81c7cc2700 -1 osd.8 60 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:55:05.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:05 smithi105 ceph-mon[36426]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-22T23:55:05.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:05 smithi105 ceph-mon[36426]: from='osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:55:05.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:05 smithi105 ceph-mon[36426]: osdmap e59: 9 total, 8 up, 9 in 2024-05-22T23:55:05.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:05.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:05 smithi105 ceph-mon[36426]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:55:05.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:05 smithi103 ceph-mon[31370]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-22T23:55:05.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:05 smithi103 ceph-mon[31370]: from='osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:55:05.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:05 smithi103 ceph-mon[31370]: osdmap e59: 9 total, 8 up, 9 in 2024-05-22T23:55:05.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:05.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:05 smithi103 ceph-mon[31370]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:55:06.594 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:06 smithi174 ceph-mon[36179]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:55:06.594 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:06 smithi174 ceph-mon[36179]: osdmap e60: 9 total, 8 up, 9 in 2024-05-22T23:55:06.595 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:06 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:06.595 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:06 smithi174 ceph-mon[36179]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:06.634 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:06 smithi103 ceph-mon[31370]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:55:06.634 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:06 smithi103 ceph-mon[31370]: osdmap e60: 9 total, 8 up, 9 in 2024-05-22T23:55:06.634 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:06 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:06.634 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:06 smithi103 ceph-mon[31370]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:06.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:06 smithi105 ceph-mon[36426]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:55:06.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:06 smithi105 ceph-mon[36426]: osdmap e60: 9 total, 8 up, 9 in 2024-05-22T23:55:06.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:06 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:06.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:06 smithi105 ceph-mon[36426]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-22T23:55:07.676 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-22T23:55:07.677 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-22T23:55:07.677 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-05-22T23:55:07.677 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-05-22T23:55:07.677 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0333727 s, 314 MB/s 2024-05-22T23:55:07.677 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-05-22T23:55:07.677 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] boot 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: osdmap e61: 9 total, 9 up, 9 in 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi174 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.678 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:07.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:07.679 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] boot 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: osdmap e61: 9 total, 9 up, 9 in 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi174 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:55:07.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:55:07.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:07.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: osd.8 [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] boot 2024-05-22T23:55:07.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: osdmap e61: 9 total, 9 up, 9 in 2024-05-22T23:55:07.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-22T23:55:07.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi174 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:07.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:08.559 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_1 2024-05-22T23:55:08.631 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:08 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi174 to 5696M 2024-05-22T23:55:08.631 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:08 smithi103 ceph-mon[31370]: osdmap e62: 9 total, 9 up, 9 in 2024-05-22T23:55:08.631 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:08.631 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:08.631 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:08.631 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:08.632 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:08 smithi103 ceph-mon[31370]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:08.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:08 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi174 to 5696M 2024-05-22T23:55:08.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:08 smithi105 ceph-mon[36426]: osdmap e62: 9 total, 9 up, 9 in 2024-05-22T23:55:08.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:08.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:08.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:08.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:08.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:08 smithi105 ceph-mon[36426]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:08.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:08 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi174 to 5696M 2024-05-22T23:55:08.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:08 smithi174 ceph-mon[36179]: osdmap e62: 9 total, 9 up, 9 in 2024-05-22T23:55:08.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:08.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:08.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:08.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:08.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:08 smithi174 ceph-mon[36179]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:08.995 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:55:09.514 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:09 smithi103 ceph-mon[31370]: osdmap e63: 9 total, 9 up, 9 in 2024-05-22T23:55:09.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:09 smithi105 ceph-mon[36426]: osdmap e63: 9 total, 9 up, 9 in 2024-05-22T23:55:09.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:09 smithi174 ceph-mon[36179]: osdmap e63: 9 total, 9 up, 9 in 2024-05-22T23:55:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:10 smithi105 ceph-mon[36426]: osdmap e64: 9 total, 9 up, 9 in 2024-05-22T23:55:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:10 smithi105 ceph-mon[36426]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:10.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:10 smithi103 ceph-mon[31370]: osdmap e64: 9 total, 9 up, 9 in 2024-05-22T23:55:10.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:10 smithi103 ceph-mon[31370]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:10.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:10 smithi174 ceph-mon[36179]: osdmap e64: 9 total, 9 up, 9 in 2024-05-22T23:55:10.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:10 smithi174 ceph-mon[36179]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:11.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 -- 172.21.15.103:0/3423920362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c102e50 msgr2=0x7f3d5c105240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:11.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3423920362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c102e50 0x7f3d5c105240 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009990 tx=0x7f3d4c009ca0 comp rx=0 tx=0).stop 2024-05-22T23:55:11.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 -- 172.21.15.103:0/3423920362 shutdown_connections 2024-05-22T23:55:11.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3423920362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d5c1080b0 0x7f3d5c10a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:11.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3423920362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c102e50 0x7f3d5c105240 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:11.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3423920362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d5c105780 0x7f3d5c107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:11.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 -- 172.21.15.103:0/3423920362 >> 172.21.15.103:0/3423920362 conn(0x7f3d5c0fc860 msgr2=0x7f3d5c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:55:11.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 -- 172.21.15.103:0/3423920362 shutdown_connections 2024-05-22T23:55:11.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.451+0000 7f3d61a00700 1 -- 172.21.15.103:0/3423920362 wait complete. 2024-05-22T23:55:11.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.452+0000 7f3d61a00700 1 Processor -- start 2024-05-22T23:55:11.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.452+0000 7f3d61a00700 1 -- start start 2024-05-22T23:55:11.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.452+0000 7f3d61a00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d5c102e50 0x7f3d5c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:55:11.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.452+0000 7f3d61a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c105780 0x7f3d5c19ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:55:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.452+0000 7f3d61a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d5c1080b0 0x7f3d5c1a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:55:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.452+0000 7f3d61a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c10dfd0 con 0x7f3d5c105780 2024-05-22T23:55:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.452+0000 7f3d61a00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c10de50 con 0x7f3d5c1080b0 2024-05-22T23:55:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.452+0000 7f3d61a00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c10e150 con 0x7f3d5c102e50 2024-05-22T23:55:11.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d52000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c105780 0x7f3d5c19ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:55:11.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d5aa00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d5c102e50 0x7f3d5c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:55:11.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d52000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c105780 0x7f3d5c19ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47960/0 (socket says 172.21.15.103:47960) 2024-05-22T23:55:11.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d52000700 1 -- 172.21.15.103:0/3593996772 learned_addr learned my addr 172.21.15.103:0/3593996772 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:55:11.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d5b400700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d5c1080b0 0x7f3d5c1a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:55:11.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d52000700 1 -- 172.21.15.103:0/3593996772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d5c102e50 msgr2=0x7f3d5c19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:11.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d52000700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d5c102e50 0x7f3d5c19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:11.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d52000700 1 -- 172.21.15.103:0/3593996772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d5c1080b0 msgr2=0x7f3d5c1a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:11.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d52000700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d5c1080b0 0x7f3d5c1a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:11.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d52000700 1 -- 172.21.15.103:0/3593996772 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c1a2490 con 0x7f3d5c105780 2024-05-22T23:55:11.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.453+0000 7f3d5aa00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d5c102e50 0x7f3d5c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:55:11.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.454+0000 7f3d52000700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c105780 0x7f3d5c19ad70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d4400e9d0 tx=0x7f3d4400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:55:11.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.454+0000 7f3d53e00700 1 -- 172.21.15.103:0/3593996772 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4400cbd0 con 0x7f3d5c105780 2024-05-22T23:55:11.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.454+0000 7f3d53e00700 1 -- 172.21.15.103:0/3593996772 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d4400ee70 con 0x7f3d5c105780 2024-05-22T23:55:11.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.454+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5c1a2780 con 0x7f3d5c105780 2024-05-22T23:55:11.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.454+0000 7f3d53e00700 1 -- 172.21.15.103:0/3593996772 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d44018660 con 0x7f3d5c105780 2024-05-22T23:55:11.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.454+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5c1a2d10 con 0x7f3d5c105780 2024-05-22T23:55:11.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.455+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d5c069050 con 0x7f3d5c105780 2024-05-22T23:55:11.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.457+0000 7f3d53e00700 1 -- 172.21.15.103:0/3593996772 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3d44010430 con 0x7f3d5c105780 2024-05-22T23:55:11.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.458+0000 7f3d53e00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3d400a2ca0 0x7f3d400a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:55:11.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.459+0000 7f3d53e00700 1 -- 172.21.15.103:0/3593996772 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5829+0+0 (secure 0 0 0) 0x7f3d440ce7f0 con 0x7f3d5c105780 2024-05-22T23:55:11.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.459+0000 7f3d5aa00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3d400a2ca0 0x7f3d400a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:55:11.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.459+0000 7f3d5aa00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3d400a2ca0 0x7f3d400a5160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00b7b0 tx=0x7f3d4c00b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:55:11.468 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.463+0000 7f3d53e00700 1 -- 172.21.15.103:0/3593996772 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3d4409c100 con 0x7f3d5c105780 2024-05-22T23:55:11.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:11.698+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3d5c0627a0 con 0x7f3d400a2ca0 2024-05-22T23:55:11.704 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:11 smithi103 ceph-mon[31370]: osdmap e65: 9 total, 9 up, 9 in 2024-05-22T23:55:11.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:11 smithi105 ceph-mon[36426]: osdmap e65: 9 total, 9 up, 9 in 2024-05-22T23:55:11.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:11 smithi174 ceph-mon[36179]: osdmap e65: 9 total, 9 up, 9 in 2024-05-22T23:55:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:12 smithi105 ceph-mon[36426]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:55:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:55:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:55:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:12 smithi105 ceph-mon[36426]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-22T23:55:12.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:12 smithi103 ceph-mon[31370]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:55:12.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:55:12.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:55:12.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:12.980 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:12 smithi103 ceph-mon[31370]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-22T23:55:12.981 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:12 smithi174 ceph-mon[36179]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:55:12.981 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:55:12.981 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:55:12.981 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:12.981 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:12 smithi174 ceph-mon[36179]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-22T23:55:15.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:15 smithi105 ceph-mon[36426]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-22T23:55:15.571 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:15 smithi174 ceph-mon[36179]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-22T23:55:15.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:15 smithi103 ceph-mon[31370]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-22T23:55:16.457 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:16 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3436709853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]: dispatch 2024-05-22T23:55:16.457 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:16 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]: dispatch 2024-05-22T23:55:16.457 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:16 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]': finished 2024-05-22T23:55:16.457 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:16 smithi103 ceph-mon[31370]: osdmap e66: 10 total, 9 up, 10 in 2024-05-22T23:55:16.457 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:16 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:16.457 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:16 smithi103 ceph-mon[31370]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-22T23:55:16.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:16 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3436709853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]: dispatch 2024-05-22T23:55:16.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:16 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]: dispatch 2024-05-22T23:55:16.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:16 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]': finished 2024-05-22T23:55:16.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:16 smithi105 ceph-mon[36426]: osdmap e66: 10 total, 9 up, 10 in 2024-05-22T23:55:16.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:16 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:16.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:16 smithi105 ceph-mon[36426]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-22T23:55:16.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:16 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3436709853' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]: dispatch 2024-05-22T23:55:16.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:16 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]: dispatch 2024-05-22T23:55:16.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:16 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91405846-87e9-40b3-a158-ffcc2b762c2d"}]': finished 2024-05-22T23:55:16.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:16 smithi174 ceph-mon[36179]: osdmap e66: 10 total, 9 up, 10 in 2024-05-22T23:55:16.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:16 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:16.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:16 smithi174 ceph-mon[36179]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-05-22T23:55:17.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:17 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1270350525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:55:17.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:17 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1270350525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:55:17.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:17 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1270350525' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:55:18.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:18 smithi105 ceph-mon[36426]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-22T23:55:18.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:18 smithi103 ceph-mon[31370]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-22T23:55:18.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:18 smithi174 ceph-mon[36179]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-22T23:55:21.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:21 smithi103 ceph-mon[31370]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-22T23:55:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:21 smithi105 ceph-mon[36426]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-22T23:55:21.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:21 smithi174 ceph-mon[36179]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-22T23:55:22.311 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:22 smithi103 ceph-mon[31370]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:22.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:22 smithi105 ceph-mon[36426]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:22.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:22 smithi174 ceph-mon[36179]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:24.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:24 smithi103 ceph-mon[31370]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:24.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:24 smithi105 ceph-mon[36426]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:24.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:24 smithi174 ceph-mon[36179]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:25.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-22T23:55:25.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:25.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-22T23:55:25.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:25.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-22T23:55:25.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:26.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:26 smithi174 ceph-mon[36179]: Deploying daemon osd.9 on smithi103 2024-05-22T23:55:26.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:26 smithi174 ceph-mon[36179]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:26.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:26 smithi103 ceph-mon[31370]: Deploying daemon osd.9 on smithi103 2024-05-22T23:55:26.621 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:26 smithi103 ceph-mon[31370]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:26.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:26 smithi105 ceph-mon[36426]: Deploying daemon osd.9 on smithi103 2024-05-22T23:55:26.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:26 smithi105 ceph-mon[36426]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:28.468 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:28 smithi103 ceph-mon[31370]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:28.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:28 smithi105 ceph-mon[36426]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:28.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:28 smithi174 ceph-mon[36179]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:30 smithi103 ceph-mon[31370]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:30 smithi105 ceph-mon[36426]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:30.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:30 smithi174 ceph-mon[36179]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:33 smithi103 ceph-mon[31370]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:33 smithi105 ceph-mon[36426]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:33 smithi174 ceph-mon[36179]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:35.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:35 smithi103 ceph-mon[31370]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:35.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:35.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:35.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:35.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:35 smithi174 ceph-mon[36179]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:35.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:35.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:35.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:35 smithi105 ceph-mon[36426]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:35.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:35.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:35.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:37.230 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:37 smithi103 ceph-mon[31370]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:37.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:37 smithi174 ceph-mon[36179]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:37.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:37 smithi105 ceph-mon[36426]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:39.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:39.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:38 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:39.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:38 smithi103 ceph-mon[31370]: from='osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-22T23:55:39.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:38 smithi103 ceph-mon[31370]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-22T23:55:39.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:38 smithi103 ceph-mon[31370]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:39.259 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 9 on host 'smithi103' 2024-05-22T23:55:39.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.255+0000 7f3d53e00700 1 -- 172.21.15.103:0/3593996772 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3d5c0627a0 con 0x7f3d400a2ca0 2024-05-22T23:55:39.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3d400a2ca0 msgr2=0x7f3d400a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:39.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3d400a2ca0 0x7f3d400a5160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00b7b0 tx=0x7f3d4c00b880 comp rx=0 tx=0).stop 2024-05-22T23:55:39.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c105780 msgr2=0x7f3d5c19ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:39.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c105780 0x7f3d5c19ad70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3d4400e9d0 tx=0x7f3d4400ece0 comp rx=0 tx=0).stop 2024-05-22T23:55:39.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 shutdown_connections 2024-05-22T23:55:39.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3d5c1080b0 0x7f3d5c1a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:39.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3d400a2ca0 0x7f3d400a5160 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:39.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d5c102e50 0x7f3d5c19a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:39.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 --2- 172.21.15.103:0/3593996772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d5c105780 0x7f3d5c19ad70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:39.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 >> 172.21.15.103:0/3593996772 conn(0x7f3d5c0fc860 msgr2=0x7f3d5c1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:55:39.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.258+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 shutdown_connections 2024-05-22T23:55:39.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:55:39.259+0000 7f3d61a00700 1 -- 172.21.15.103:0/3593996772 wait complete. 2024-05-22T23:55:39.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:39.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:38 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:39.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:38 smithi105 ceph-mon[36426]: from='osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-22T23:55:39.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:38 smithi105 ceph-mon[36426]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-22T23:55:39.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:38 smithi105 ceph-mon[36426]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:39.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:39.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:38 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:39.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:38 smithi174 ceph-mon[36179]: from='osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-22T23:55:39.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:38 smithi174 ceph-mon[36179]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-22T23:55:39.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:38 smithi174 ceph-mon[36179]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:40.124 DEBUG:teuthology.orchestra.run.smithi103:osd.9> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.9.service 2024-05-22T23:55:40.131 INFO:tasks.cephadm:Deploying osd.10 on smithi105 with /dev/vg_nvme/lv_1... 2024-05-22T23:55:40.131 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-22T23:55:40.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-22T23:55:40.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: osdmap e67: 10 total, 9 up, 10 in 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:39 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:40.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-22T23:55:40.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: osdmap e67: 10 total, 9 up, 10 in 2024-05-22T23:55:40.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:40.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:55:40.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:55:40.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:40.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:40.334 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.335 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:39 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: osdmap e67: 10 total, 9 up, 10 in 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:40.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:39 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:55:40.375 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-22T23:55:40.831 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 22 23:55:40 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-22T23:55:40.610+0000 7f8c13600700 -1 osd.9 0 waiting for initial osdmap 2024-05-22T23:55:40.832 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 22 23:55:40 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-22T23:55:40.668+0000 7f8c09400700 -1 osd.9 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:55:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:40 smithi105 ceph-mon[36426]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:55:41.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:40 smithi105 ceph-mon[36426]: osdmap e68: 10 total, 9 up, 10 in 2024-05-22T23:55:41.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:40 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:41.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:40 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:41.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:40 smithi105 ceph-mon[36426]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:41.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:40 smithi105 ceph-mon[36426]: from='osd.9 ' entity='osd.9' 2024-05-22T23:55:41.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:40 smithi103 ceph-mon[31370]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:55:41.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:40 smithi103 ceph-mon[31370]: osdmap e68: 10 total, 9 up, 10 in 2024-05-22T23:55:41.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:41.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:40 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:41.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:40 smithi103 ceph-mon[31370]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:41.295 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:40 smithi103 ceph-mon[31370]: from='osd.9 ' entity='osd.9' 2024-05-22T23:55:41.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:40 smithi174 ceph-mon[36179]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-22T23:55:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:40 smithi174 ceph-mon[36179]: osdmap e68: 10 total, 9 up, 10 in 2024-05-22T23:55:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:40 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:40 smithi174 ceph-mon[36179]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-05-22T23:55:41.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:40 smithi174 ceph-mon[36179]: from='osd.9 ' entity='osd.9' 2024-05-22T23:55:42.170 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:41 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:55:42.170 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:41 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:55:42.170 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:42.171 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:41 smithi103 ceph-mon[31370]: osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] boot 2024-05-22T23:55:42.171 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:41 smithi103 ceph-mon[31370]: osdmap e69: 10 total, 10 up, 10 in 2024-05-22T23:55:42.171 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:41 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:42.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:41 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:55:42.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:41 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:55:42.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:41 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:42.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:41 smithi105 ceph-mon[36426]: osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] boot 2024-05-22T23:55:42.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:41 smithi105 ceph-mon[36426]: osdmap e69: 10 total, 10 up, 10 in 2024-05-22T23:55:42.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:41 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:41 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:55:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:41 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:55:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:41 smithi174 ceph-mon[36179]: osd.9 [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] boot 2024-05-22T23:55:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:41 smithi174 ceph-mon[36179]: osdmap e69: 10 total, 10 up, 10 in 2024-05-22T23:55:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:41 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-22T23:55:42.428 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-22T23:55:42.428 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-22T23:55:42.428 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-05-22T23:55:42.428 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-05-22T23:55:42.429 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114698 s, 914 MB/s 2024-05-22T23:55:42.429 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-05-22T23:55:43.215 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:43 smithi105 ceph-mon[36426]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:43.243 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_1 2024-05-22T23:55:43.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:43 smithi103 ceph-mon[31370]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:43.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:43 smithi174 ceph-mon[36179]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:43.480 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-22T23:55:44.296 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:44 smithi105 ceph-mon[36426]: osdmap e70: 10 total, 10 up, 10 in 2024-05-22T23:55:44.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:44 smithi103 ceph-mon[31370]: osdmap e70: 10 total, 10 up, 10 in 2024-05-22T23:55:44.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:44 smithi174 ceph-mon[36179]: osdmap e70: 10 total, 10 up, 10 in 2024-05-22T23:55:45.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.125+0000 7fbc87810700 1 -- 172.21.15.105:0/2663697833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc80103f60 msgr2=0x7fbc8010a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:45.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.125+0000 7fbc87810700 1 --2- 172.21.15.105:0/2663697833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc80103f60 0x7fbc8010a810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbc7800b0d0 tx=0x7fbc7800b3e0 comp rx=0 tx=0).stop 2024-05-22T23:55:45.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.125+0000 7fbc87810700 1 -- 172.21.15.105:0/2663697833 shutdown_connections 2024-05-22T23:55:45.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.125+0000 7fbc87810700 1 --2- 172.21.15.105:0/2663697833 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc80103f60 0x7fbc8010a810 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:45.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.125+0000 7fbc87810700 1 --2- 172.21.15.105:0/2663697833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc801023c0 0x7fbc801027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:45.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.125+0000 7fbc87810700 1 --2- 172.21.15.105:0/2663697833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc801035c0 0x7fbc80103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:45.130 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.125+0000 7fbc87810700 1 -- 172.21.15.105:0/2663697833 >> 172.21.15.105:0/2663697833 conn(0x7fbc800fd960 msgr2=0x7fbc800ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:55:45.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 -- 172.21.15.105:0/2663697833 shutdown_connections 2024-05-22T23:55:45.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 -- 172.21.15.105:0/2663697833 wait complete. 2024-05-22T23:55:45.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 Processor -- start 2024-05-22T23:55:45.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 -- start start 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc801023c0 0x7fbc80076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc801035c0 0x7fbc80074750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc80103f60 0x7fbc80074c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8010e040 con 0x7fbc801035c0 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8010dec0 con 0x7fbc801023c0 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.126+0000 7fbc87810700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbc8010e1c0 con 0x7fbc80103f60 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc801023c0 0x7fbc80076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc801023c0 0x7fbc80076100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37850/0 (socket says 172.21.15.105:37850) 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 -- 172.21.15.105:0/3770341986 learned_addr learned my addr 172.21.15.105:0/3770341986 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc85dad700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc80103f60 0x7fbc80074c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:55:45.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 -- 172.21.15.105:0/3770341986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc80103f60 msgr2=0x7fbc80074c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc80103f60 0x7fbc80074c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 -- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc801035c0 msgr2=0x7fbc80074750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc801035c0 0x7fbc80074750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 -- 172.21.15.105:0/3770341986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc80075540 con 0x7fbc801023c0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc85dad700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc80103f60 0x7fbc80074c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc855ac700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc801023c0 0x7fbc80076100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000b500 tx=0x7fbc7000b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc767fc700 1 -- 172.21.15.105:0/3770341986 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc70015020 con 0x7fbc801023c0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc801a27b0 con 0x7fbc801023c0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc767fc700 1 -- 172.21.15.105:0/3770341986 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc70011d10 con 0x7fbc801023c0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc767fc700 1 -- 172.21.15.105:0/3770341986 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7000f2e0 con 0x7fbc801023c0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.127+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc801a2d70 con 0x7fbc801023c0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.128+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc80069050 con 0x7fbc801023c0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.129+0000 7fbc767fc700 1 -- 172.21.15.105:0/3770341986 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fbc70018020 con 0x7fbc801023c0 2024-05-22T23:55:45.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.130+0000 7fbc767fc700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc6c0a2c20 0x7fbc6c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:55:45.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.130+0000 7fbc767fc700 1 -- 172.21.15.105:0/3770341986 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6261+0+0 (secure 0 0 0) 0x7fbc700ce050 con 0x7fbc801023c0 2024-05-22T23:55:45.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.130+0000 7fbc84dab700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc6c0a2c20 0x7fbc6c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:55:45.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.131+0000 7fbc84dab700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc6c0a2c20 0x7fbc6c0a50e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00a990 tx=0x7fbc7c005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:55:45.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.134+0000 7fbc767fc700 1 -- 172.21.15.105:0/3770341986 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbc7001f020 con 0x7fbc801023c0 2024-05-22T23:55:45.270 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:45 smithi174 ceph-mon[36179]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:45.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:45 smithi105 ceph-mon[36426]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:45.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:45 smithi103 ceph-mon[31370]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:45.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:55:45.342+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fbc800627a0 con 0x7fbc6c0a2c20 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi103 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi103 to 4272M 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:46.281 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:46.281 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.281 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:55:46.281 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:55:46.281 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:55:46.281 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:46.281 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:46 smithi105 ceph-mon[36426]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:46.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi103 2024-05-22T23:55:46.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi103 to 4272M 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:55:46.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:55:46.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:55:46.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:46.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:46 smithi103 ceph-mon[31370]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi103 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi103 to 4272M 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:46.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:55:46.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:55:46.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:55:46.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:55:46.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:55:46.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:46.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:46 smithi174 ceph-mon[36179]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:48 smithi105 ceph-mon[36426]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:48.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:48 smithi103 ceph-mon[31370]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:48 smithi174 ceph-mon[36179]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:49 smithi103 ceph-mon[31370]: from='client.? 172.21.15.105:0/3814675249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]: dispatch 2024-05-22T23:55:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:49 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]: dispatch 2024-05-22T23:55:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:49 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]': finished 2024-05-22T23:55:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:49 smithi103 ceph-mon[31370]: osdmap e71: 11 total, 10 up, 11 in 2024-05-22T23:55:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:49 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:55:49.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:49 smithi103 ceph-mon[31370]: from='client.? 172.21.15.105:0/620177168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:55:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:49 smithi105 ceph-mon[36426]: from='client.? 172.21.15.105:0/3814675249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]: dispatch 2024-05-22T23:55:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:49 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]: dispatch 2024-05-22T23:55:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:49 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]': finished 2024-05-22T23:55:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:49 smithi105 ceph-mon[36426]: osdmap e71: 11 total, 10 up, 11 in 2024-05-22T23:55:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:49 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:55:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:49 smithi105 ceph-mon[36426]: from='client.? 172.21.15.105:0/620177168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:55:49.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:49 smithi174 ceph-mon[36179]: from='client.? 172.21.15.105:0/3814675249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]: dispatch 2024-05-22T23:55:49.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:49 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]: dispatch 2024-05-22T23:55:49.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:49 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32c228c2-bec7-4f25-99a9-137e0514381d"}]': finished 2024-05-22T23:55:49.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:49 smithi174 ceph-mon[36179]: osdmap e71: 11 total, 10 up, 11 in 2024-05-22T23:55:49.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:49 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:55:49.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:49 smithi174 ceph-mon[36179]: from='client.? 172.21.15.105:0/620177168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:55:50.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:50 smithi105 ceph-mon[36426]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:50.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:50 smithi103 ceph-mon[31370]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:50.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:50 smithi174 ceph-mon[36179]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:52.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:52 smithi105 ceph-mon[36426]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:52.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:52 smithi103 ceph-mon[31370]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:52.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:52 smithi174 ceph-mon[36179]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:54.412 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:54 smithi105 ceph-mon[36426]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:54.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:54 smithi103 ceph-mon[31370]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:54.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:54 smithi174 ceph-mon[36179]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:56.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:55 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-22T23:55:56.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:55 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:56.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:55 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-22T23:55:56.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:55 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:56.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:55 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-22T23:55:56.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:55 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:55:57.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:56 smithi105 ceph-mon[36426]: Deploying daemon osd.10 on smithi105 2024-05-22T23:55:57.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:56 smithi105 ceph-mon[36426]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:57.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:56 smithi103 ceph-mon[31370]: Deploying daemon osd.10 on smithi105 2024-05-22T23:55:57.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:56 smithi103 ceph-mon[31370]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:57.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:56 smithi174 ceph-mon[36179]: Deploying daemon osd.10 on smithi105 2024-05-22T23:55:57.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:56 smithi174 ceph-mon[36179]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:58.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:55:58 smithi105 ceph-mon[36426]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:58.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:55:58 smithi103 ceph-mon[31370]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:55:58.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:55:58 smithi174 ceph-mon[36179]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:00 smithi105 ceph-mon[36426]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:00.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:00 smithi103 ceph-mon[31370]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:00.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:00 smithi174 ceph-mon[36179]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:02.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:01 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:02.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:01 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:02.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:01 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:02.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:01 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:02.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:01 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:02.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:01 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:02.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:01 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:02.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:01 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:02.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:01 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:03.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:02 smithi105 ceph-mon[36426]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:03.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:02 smithi103 ceph-mon[31370]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:03.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:02 smithi174 ceph-mon[36179]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:04.489 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:04 smithi105 ceph-mon[36426]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:04.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:04 smithi103 ceph-mon[31370]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:04 smithi174 ceph-mon[36179]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:05.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.725+0000 7fbc767fc700 1 -- 172.21.15.105:0/3770341986 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fbc800627a0 con 0x7fbc6c0a2c20 2024-05-22T23:56:05.730 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 10 on host 'smithi105' 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.727+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc6c0a2c20 msgr2=0x7fbc6c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.727+0000 7fbc87810700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc6c0a2c20 0x7fbc6c0a50e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c00a990 tx=0x7fbc7c005c80 comp rx=0 tx=0).stop 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.727+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc801023c0 msgr2=0x7fbc80076100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.727+0000 7fbc87810700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc801023c0 0x7fbc80076100 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbc7000b500 tx=0x7fbc7000b8c0 comp rx=0 tx=0).stop 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.728+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 shutdown_connections 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.728+0000 7fbc87810700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc801023c0 0x7fbc80076100 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.728+0000 7fbc87810700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc6c0a2c20 0x7fbc6c0a50e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.728+0000 7fbc87810700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc80103f60 0x7fbc80074c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.728+0000 7fbc87810700 1 --2- 172.21.15.105:0/3770341986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc801035c0 0x7fbc80074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.728+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 >> 172.21.15.105:0/3770341986 conn(0x7fbc800fd960 msgr2=0x7fbc801068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:05.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.728+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 shutdown_connections 2024-05-22T23:56:05.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-05-22T23:56:05.728+0000 7fbc87810700 1 -- 172.21.15.105:0/3770341986 wait complete. 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:06.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:06.065 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:06.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:06.435 DEBUG:teuthology.orchestra.run.smithi105:osd.10> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.10.service 2024-05-22T23:56:06.437 INFO:tasks.cephadm:Deploying osd.11 on smithi174 with /dev/vg_nvme/lv_1... 2024-05-22T23:56:06.438 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-22T23:56:06.683 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:56:06.932 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:06 smithi105 ceph-mon[36426]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-22T23:56:06.932 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:06 smithi105 ceph-mon[36426]: from='osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:56:06.932 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:06 smithi105 ceph-mon[36426]: osdmap e72: 11 total, 10 up, 11 in 2024-05-22T23:56:06.932 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:06 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:06.932 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:06 smithi105 ceph-mon[36426]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:56:06.932 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:06 smithi105 ceph-mon[36426]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:07.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:06 smithi174 ceph-mon[36179]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-22T23:56:07.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:06 smithi174 ceph-mon[36179]: from='osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:56:07.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:06 smithi174 ceph-mon[36179]: osdmap e72: 11 total, 10 up, 11 in 2024-05-22T23:56:07.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:06 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:07.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:06 smithi174 ceph-mon[36179]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:56:07.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:06 smithi174 ceph-mon[36179]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:06 smithi103 ceph-mon[31370]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-22T23:56:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:06 smithi103 ceph-mon[31370]: from='osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:56:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:06 smithi103 ceph-mon[31370]: osdmap e72: 11 total, 10 up, 11 in 2024-05-22T23:56:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:06 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:06 smithi103 ceph-mon[31370]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-05-22T23:56:07.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:06 smithi103 ceph-mon[31370]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-05-22T23:56:07.559 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 22 23:56:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-22T23:56:07.291+0000 7f337d31c700 -1 osd.10 0 waiting for initial osdmap 2024-05-22T23:56:07.560 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 22 23:56:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-22T23:56:07.302+0000 7f3375107700 -1 osd.10 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:56:08.046 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:07 smithi105 ceph-mon[36426]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:56:08.046 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:07 smithi105 ceph-mon[36426]: osdmap e73: 11 total, 10 up, 11 in 2024-05-22T23:56:08.046 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:07 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:08.046 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:07 smithi105 ceph-mon[36426]: from='osd.10 ' entity='osd.10' 2024-05-22T23:56:08.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:07 smithi174 ceph-mon[36179]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:56:08.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:07 smithi174 ceph-mon[36179]: osdmap e73: 11 total, 10 up, 11 in 2024-05-22T23:56:08.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:07 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:08.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:07 smithi174 ceph-mon[36179]: from='osd.10 ' entity='osd.10' 2024-05-22T23:56:08.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:07 smithi103 ceph-mon[31370]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-05-22T23:56:08.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:07 smithi103 ceph-mon[31370]: osdmap e73: 11 total, 10 up, 11 in 2024-05-22T23:56:08.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:07 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:08.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:07 smithi103 ceph-mon[31370]: from='osd.10 ' entity='osd.10' 2024-05-22T23:56:08.871 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-22T23:56:08.871 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-22T23:56:08.871 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-05-22T23:56:08.871 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-05-22T23:56:08.872 INFO:teuthology.orchestra.run.smithi174.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0129637 s, 809 MB/s 2024-05-22T23:56:08.872 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-05-22T23:56:09.202 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:08 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:56:09.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:08 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:56:09.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:08 smithi174 ceph-mon[36179]: osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] boot 2024-05-22T23:56:09.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:08 smithi174 ceph-mon[36179]: osdmap e74: 11 total, 11 up, 11 in 2024-05-22T23:56:09.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:09.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:08 smithi174 ceph-mon[36179]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:09.203 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:08 smithi174 ceph-mon[36179]: osdmap e75: 11 total, 11 up, 11 in 2024-05-22T23:56:09.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:08 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:56:09.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:08 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:56:09.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:08 smithi105 ceph-mon[36426]: osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] boot 2024-05-22T23:56:09.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:08 smithi105 ceph-mon[36426]: osdmap e74: 11 total, 11 up, 11 in 2024-05-22T23:56:09.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:09.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:08 smithi105 ceph-mon[36426]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:09.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:08 smithi105 ceph-mon[36426]: osdmap e75: 11 total, 11 up, 11 in 2024-05-22T23:56:09.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:08 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:56:09.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:08 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:56:09.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:08 smithi103 ceph-mon[31370]: osd.10 [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] boot 2024-05-22T23:56:09.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:08 smithi103 ceph-mon[31370]: osdmap e74: 11 total, 11 up, 11 in 2024-05-22T23:56:09.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-22T23:56:09.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:08 smithi103 ceph-mon[31370]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:09.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:08 smithi103 ceph-mon[31370]: osdmap e75: 11 total, 11 up, 11 in 2024-05-22T23:56:09.524 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_1 2024-05-22T23:56:09.768 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:56:10.324 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: osdmap e76: 11 total, 11 up, 11 in 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi105 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-22T23:56:10.325 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.326 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:10.326 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:10.326 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.326 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:10 smithi174 ceph-mon[36179]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:10.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: osdmap e76: 11 total, 11 up, 11 in 2024-05-22T23:56:10.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi105 2024-05-22T23:56:10.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:10 smithi105 ceph-mon[36426]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:10.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: osdmap e76: 11 total, 11 up, 11 in 2024-05-22T23:56:10.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi105 2024-05-22T23:56:10.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:10.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi105 to 4272M 2024-05-22T23:56:10.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:10.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:10.583 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:10.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:10 smithi103 ceph-mon[31370]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:11.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 -- 172.21.15.174:0/2630832811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d41035b0 msgr2=0x7fe7d4103a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:11.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 --2- 172.21.15.174:0/2630832811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d41035b0 0x7fe7d4103a10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe7c8009990 tx=0x7fe7c8009ca0 comp rx=0 tx=0).stop 2024-05-22T23:56:11.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 -- 172.21.15.174:0/2630832811 shutdown_connections 2024-05-22T23:56:11.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 --2- 172.21.15.174:0/2630832811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7d41023b0 0x7fe7d41027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:11.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 --2- 172.21.15.174:0/2630832811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d4103f50 0x7fe7d410a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:11.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 --2- 172.21.15.174:0/2630832811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d41035b0 0x7fe7d4103a10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:11.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 -- 172.21.15.174:0/2630832811 >> 172.21.15.174:0/2630832811 conn(0x7fe7d40fd950 msgr2=0x7fe7d40ffd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:11.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 -- 172.21.15.174:0/2630832811 shutdown_connections 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.500+0000 7fe7da301700 1 -- 172.21.15.174:0/2630832811 wait complete. 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.501+0000 7fe7da301700 1 Processor -- start 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.501+0000 7fe7da301700 1 -- start start 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.501+0000 7fe7da301700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7d41023b0 0x7fe7d419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.501+0000 7fe7da301700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d41035b0 0x7fe7d419aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.501+0000 7fe7da301700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d4103f50 0x7fe7d41a1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.501+0000 7fe7da301700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d410e090 con 0x7fe7d41035b0 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.501+0000 7fe7da301700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d410df10 con 0x7fe7d41023b0 2024-05-22T23:56:11.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.501+0000 7fe7da301700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d410e210 con 0x7fe7d4103f50 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d4103f50 0x7fe7d41a1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d4103f50 0x7fe7d41a1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:52972/0 (socket says 172.21.15.174:52972) 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 -- 172.21.15.174:0/993941595 learned_addr learned my addr 172.21.15.174:0/993941595 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d3fff700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7d41023b0 0x7fe7d419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 -- 172.21.15.174:0/993941595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7d41023b0 msgr2=0x7fe7d419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7d41023b0 0x7fe7d419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 -- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d41035b0 msgr2=0x7fe7d419aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d41035b0 0x7fe7d419aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 -- 172.21.15.174:0/993941595 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d41a25d0 con 0x7fe7d4103f50 2024-05-22T23:56:11.509 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d889e700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d4103f50 0x7fe7d41a1f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00b9a0 tx=0x7fe7cc00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:11.510 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d17fa700 1 -- 172.21.15.174:0/993941595 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc019070 con 0x7fe7d4103f50 2024-05-22T23:56:11.510 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d17fa700 1 -- 172.21.15.174:0/993941595 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7cc014440 con 0x7fe7d4103f50 2024-05-22T23:56:11.510 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d41a28c0 con 0x7fe7d4103f50 2024-05-22T23:56:11.510 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7d17fa700 1 -- 172.21.15.174:0/993941595 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0075d0 con 0x7fe7d4103f50 2024-05-22T23:56:11.510 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.502+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d41a2e30 con 0x7fe7d4103f50 2024-05-22T23:56:11.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.503+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d4069050 con 0x7fe7d4103f50 2024-05-22T23:56:11.511 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.504+0000 7fe7d17fa700 1 -- 172.21.15.174:0/993941595 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe7cc01c070 con 0x7fe7d4103f50 2024-05-22T23:56:11.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.505+0000 7fe7d17fa700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe7bc0a2c20 0x7fe7bc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:11.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.505+0000 7fe7d17fa700 1 -- 172.21.15.174:0/993941595 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 6694+0+0 (secure 0 0 0) 0x7fe7cc0cde70 con 0x7fe7d4103f50 2024-05-22T23:56:11.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.506+0000 7fe7d3fff700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe7bc0a2c20 0x7fe7bc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:11.512 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.506+0000 7fe7d3fff700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe7bc0a2c20 0x7fe7bc0a50e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006fd0 tx=0x7fe7c4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:11.513 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.510+0000 7fe7d17fa700 1 -- 172.21.15.174:0/993941595 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe7cc09b4b0 con 0x7fe7d4103f50 2024-05-22T23:56:11.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:11 smithi103 ceph-mon[31370]: osdmap e77: 11 total, 11 up, 11 in 2024-05-22T23:56:11.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:11 smithi174 ceph-mon[36179]: osdmap e77: 11 total, 11 up, 11 in 2024-05-22T23:56:11.725 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:11.721+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fe7d41a30c0 con 0x7fe7bc0a2c20 2024-05-22T23:56:11.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:11 smithi105 ceph-mon[36426]: osdmap e77: 11 total, 11 up, 11 in 2024-05-22T23:56:12.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:12 smithi103 ceph-mon[31370]: osdmap e78: 11 total, 11 up, 11 in 2024-05-22T23:56:12.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:12 smithi103 ceph-mon[31370]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:56:12.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:56:12.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:56:12.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:12 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:12.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:12 smithi103 ceph-mon[31370]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-22T23:56:12.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:12 smithi174 ceph-mon[36179]: osdmap e78: 11 total, 11 up, 11 in 2024-05-22T23:56:12.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:12 smithi174 ceph-mon[36179]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:56:12.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:56:12.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:56:12.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:12 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:12.596 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:12 smithi174 ceph-mon[36179]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-22T23:56:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:12 smithi105 ceph-mon[36426]: osdmap e78: 11 total, 11 up, 11 in 2024-05-22T23:56:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:12 smithi105 ceph-mon[36426]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:56:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-22T23:56:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-22T23:56:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:12 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:12 smithi105 ceph-mon[36426]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-22T23:56:14.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:14 smithi105 ceph-mon[36426]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-05-22T23:56:14.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:14 smithi103 ceph-mon[31370]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-05-22T23:56:14.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:14 smithi174 ceph-mon[36179]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-05-22T23:56:15.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:15 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/3573135388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]: dispatch 2024-05-22T23:56:15.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:15 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]: dispatch 2024-05-22T23:56:15.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:15 smithi105 ceph-mon[36426]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]': finished 2024-05-22T23:56:15.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:15 smithi105 ceph-mon[36426]: osdmap e79: 12 total, 11 up, 12 in 2024-05-22T23:56:15.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:15 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:15.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:15 smithi105 ceph-mon[36426]: from='client.? 172.21.15.174:0/3564519940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:56:15.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:15 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/3573135388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]: dispatch 2024-05-22T23:56:15.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:15 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]: dispatch 2024-05-22T23:56:15.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:15 smithi103 ceph-mon[31370]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]': finished 2024-05-22T23:56:15.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:15 smithi103 ceph-mon[31370]: osdmap e79: 12 total, 11 up, 12 in 2024-05-22T23:56:15.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:15 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:15.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:15 smithi103 ceph-mon[31370]: from='client.? 172.21.15.174:0/3564519940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:56:15.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:15 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/3573135388' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]: dispatch 2024-05-22T23:56:15.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:15 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]: dispatch 2024-05-22T23:56:15.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:15 smithi174 ceph-mon[36179]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c93124e-d53a-4091-bc85-bc3dd328a43c"}]': finished 2024-05-22T23:56:15.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:15 smithi174 ceph-mon[36179]: osdmap e79: 12 total, 11 up, 12 in 2024-05-22T23:56:15.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:15 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:15.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:15 smithi174 ceph-mon[36179]: from='client.? 172.21.15.174:0/3564519940' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-22T23:56:16.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:16 smithi105 ceph-mon[36426]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-22T23:56:16.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:16 smithi103 ceph-mon[31370]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-22T23:56:16.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:16 smithi174 ceph-mon[36179]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-22T23:56:18.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:18 smithi105 ceph-mon[36426]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-22T23:56:18.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:18 smithi103 ceph-mon[31370]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-22T23:56:18.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:18 smithi174 ceph-mon[36179]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-05-22T23:56:20.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:20 smithi174 ceph-mon[36179]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-22T23:56:20.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:20 smithi105 ceph-mon[36426]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-22T23:56:20.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:20 smithi103 ceph-mon[31370]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-22T23:56:22.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:22 smithi105 ceph-mon[36426]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:22.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:22 smithi103 ceph-mon[31370]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:22.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:22 smithi174 ceph-mon[36179]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:23.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:23 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-22T23:56:23.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:23 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:23.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:23 smithi105 ceph-mon[36426]: Deploying daemon osd.11 on smithi174 2024-05-22T23:56:23.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:23 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-22T23:56:23.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:23 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:23.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:23 smithi103 ceph-mon[31370]: Deploying daemon osd.11 on smithi174 2024-05-22T23:56:23.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:23 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-22T23:56:23.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:23 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:23.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:23 smithi174 ceph-mon[36179]: Deploying daemon osd.11 on smithi174 2024-05-22T23:56:24.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:24 smithi105 ceph-mon[36426]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:24.561 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:24 smithi174 ceph-mon[36179]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:24.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:24 smithi103 ceph-mon[31370]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:26.267 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:26 smithi174 ceph-mon[36179]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:26.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:26 smithi105 ceph-mon[36426]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:26.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:26 smithi103 ceph-mon[31370]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:29.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:28 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:29.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:28 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:29.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:28 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:29.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:28 smithi105 ceph-mon[36426]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:29.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:28 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:29.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:28 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:29.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:28 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:29.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:28 smithi103 ceph-mon[31370]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:29.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:28 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:29.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:28 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:29.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:28 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:29.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:28 smithi174 ceph-mon[36179]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:30 smithi174 ceph-mon[36179]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:30 smithi105 ceph-mon[36426]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:30.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:30 smithi103 ceph-mon[31370]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:31.717 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.713+0000 7fe7d17fa700 1 -- 172.21.15.174:0/993941595 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fe7d41a30c0 con 0x7fe7bc0a2c20 2024-05-22T23:56:31.717 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 11 on host 'smithi174' 2024-05-22T23:56:31.722 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe7bc0a2c20 msgr2=0x7fe7bc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe7bc0a2c20 0x7fe7bc0a50e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4006fd0 tx=0x7fe7c4008040 comp rx=0 tx=0).stop 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d4103f50 msgr2=0x7fe7d41a1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d4103f50 0x7fe7d41a1f30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00b9a0 tx=0x7fe7cc00bd60 comp rx=0 tx=0).stop 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 shutdown_connections 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe7d41023b0 0x7fe7d419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe7bc0a2c20 0x7fe7bc0a50e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe7d4103f50 0x7fe7d41a1f30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 --2- 172.21.15.174:0/993941595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7d41035b0 0x7fe7d419aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 >> 172.21.15.174:0/993941595 conn(0x7fe7d40fd950 msgr2=0x7fe7d4074590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 shutdown_connections 2024-05-22T23:56:31.723 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:31.716+0000 7fe7da301700 1 -- 172.21.15.174:0/993941595 wait complete. 2024-05-22T23:56:32.390 DEBUG:teuthology.orchestra.run.smithi174:osd.11> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.11.service 2024-05-22T23:56:32.392 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-22T23:56:32.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd stat -f json 2024-05-22T23:56:32.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:32.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:32.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:32 smithi103 ceph-mon[31370]: from='osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-22T23:56:32.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:32 smithi103 ceph-mon[31370]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-22T23:56:32.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:32 smithi103 ceph-mon[31370]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:32.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:32.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:32 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:32.850 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:56:33.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:33.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:33.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:32 smithi105 ceph-mon[36426]: from='osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-22T23:56:33.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:32 smithi105 ceph-mon[36426]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-22T23:56:33.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:32 smithi105 ceph-mon[36426]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:33.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:33.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:32 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:33.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:33.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:33.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:32 smithi174 ceph-mon[36179]: from='osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-22T23:56:33.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:32 smithi174 ceph-mon[36179]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-22T23:56:33.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:32 smithi174 ceph-mon[36179]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:33.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:33.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:32 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:34.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:33 smithi105 ceph-mon[36426]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-22T23:56:34.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:33 smithi105 ceph-mon[36426]: osdmap e80: 12 total, 11 up, 12 in 2024-05-22T23:56:34.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:33 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:34.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:33 smithi105 ceph-mon[36426]: from='osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:56:34.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:33 smithi105 ceph-mon[36426]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:56:34.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:33 smithi103 ceph-mon[31370]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-22T23:56:34.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:33 smithi103 ceph-mon[31370]: osdmap e80: 12 total, 11 up, 12 in 2024-05-22T23:56:34.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:33 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:34.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:33 smithi103 ceph-mon[31370]: from='osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:56:34.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:33 smithi103 ceph-mon[31370]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:56:34.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:33 smithi174 ceph-mon[36179]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-22T23:56:34.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:33 smithi174 ceph-mon[36179]: osdmap e80: 12 total, 11 up, 12 in 2024-05-22T23:56:34.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:33 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:34.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:33 smithi174 ceph-mon[36179]: from='osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:56:34.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:33 smithi174 ceph-mon[36179]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-05-22T23:56:34.119 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 22 23:56:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-22T23:56:33.724+0000 7fea2cf56700 -1 osd.11 0 waiting for initial osdmap 2024-05-22T23:56:34.119 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 22 23:56:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-22T23:56:33.745+0000 7fea28d49700 -1 osd.11 81 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-22T23:56:34.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:34 smithi103 ceph-mon[31370]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:56:34.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:34 smithi103 ceph-mon[31370]: osdmap e81: 12 total, 11 up, 12 in 2024-05-22T23:56:34.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:34.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:34.946 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:34 smithi103 ceph-mon[31370]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:35.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:34 smithi105 ceph-mon[36426]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:56:35.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:34 smithi105 ceph-mon[36426]: osdmap e81: 12 total, 11 up, 12 in 2024-05-22T23:56:35.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:34 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:35.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:34 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:35.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:34 smithi105 ceph-mon[36426]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:35.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:34 smithi174 ceph-mon[36179]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-05-22T23:56:35.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:34 smithi174 ceph-mon[36179]: osdmap e81: 12 total, 11 up, 12 in 2024-05-22T23:56:35.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:34 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:35.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:34 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:35.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:34 smithi174 ceph-mon[36179]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 -- 172.21.15.103:0/829002536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 msgr2=0x7f16cc10a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 --2- 172.21.15.103:0/829002536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 0x7f16cc10a820 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f16c400b0a0 tx=0x7f16c400b3b0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 -- 172.21.15.103:0/829002536 shutdown_connections 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 --2- 172.21.15.103:0/829002536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16cc1023d0 0x7f16cc1027f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 --2- 172.21.15.103:0/829002536 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 0x7f16cc10a820 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 --2- 172.21.15.103:0/829002536 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16cc1035d0 0x7f16cc103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 -- 172.21.15.103:0/829002536 >> 172.21.15.103:0/829002536 conn(0x7f16cc0fd930 msgr2=0x7f16cc0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 -- 172.21.15.103:0/829002536 shutdown_connections 2024-05-22T23:56:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.214+0000 7f16d1200700 1 -- 172.21.15.103:0/829002536 wait complete. 2024-05-22T23:56:35.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.215+0000 7f16d1200700 1 Processor -- start 2024-05-22T23:56:35.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.215+0000 7f16d1200700 1 -- start start 2024-05-22T23:56:35.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16d1200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16cc1023d0 0x7f16cc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:35.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16d1200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16cc1035d0 0x7f16cc198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:35.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16d1200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 0x7f16cc19fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:35.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16d1200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10e000 con 0x7f16cc1023d0 2024-05-22T23:56:35.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16d1200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10de80 con 0x7f16cc1035d0 2024-05-22T23:56:35.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16d1200700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc10e180 con 0x7f16cc103f70 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16ca800700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 0x7f16cc19fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16c9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16cc1023d0 0x7f16cc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16ca800700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 0x7f16cc19fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:57854/0 (socket says 172.21.15.103:57854) 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16ca800700 1 -- 172.21.15.103:0/2250484845 learned_addr learned my addr 172.21.15.103:0/2250484845 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16c9400700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16cc1035d0 0x7f16cc198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16ca800700 1 -- 172.21.15.103:0/2250484845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16cc1035d0 msgr2=0x7f16cc198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16ca800700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16cc1035d0 0x7f16cc198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16ca800700 1 -- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16cc1023d0 msgr2=0x7f16cc198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:35.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16ca800700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16cc1023d0 0x7f16cc198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.216+0000 7f16ca800700 1 -- 172.21.15.103:0/2250484845 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16cc1a01f0 con 0x7f16cc103f70 2024-05-22T23:56:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.217+0000 7f16ca800700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 0x7f16cc19fb50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16c400b070 tx=0x7f16c400cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.217+0000 7f16c2a00700 1 -- 172.21.15.103:0/2250484845 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c4021070 con 0x7f16cc103f70 2024-05-22T23:56:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.217+0000 7f16c2a00700 1 -- 172.21.15.103:0/2250484845 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16c4007b80 con 0x7f16cc103f70 2024-05-22T23:56:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.217+0000 7f16c2a00700 1 -- 172.21.15.103:0/2250484845 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16c401d450 con 0x7f16cc103f70 2024-05-22T23:56:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.217+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16cc1a0480 con 0x7f16cc103f70 2024-05-22T23:56:35.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.217+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16cc1a0980 con 0x7f16cc103f70 2024-05-22T23:56:35.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.218+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16cc069050 con 0x7f16cc103f70 2024-05-22T23:56:35.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.220+0000 7f16c2a00700 1 -- 172.21.15.103:0/2250484845 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f16c4004020 con 0x7f16cc103f70 2024-05-22T23:56:35.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.221+0000 7f16c2a00700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f16b00a2ca0 0x7f16b00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:35.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.221+0000 7f16c2a00700 1 -- 172.21.15.103:0/2250484845 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f16c400f020 con 0x7f16cc103f70 2024-05-22T23:56:35.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.222+0000 7f16c9e00700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f16b00a2ca0 0x7f16b00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:35.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.222+0000 7f16c9e00700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f16b00a2ca0 0x7f16b00a5160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f16cc103430 tx=0x7f16bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:35.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.225+0000 7f16c2a00700 1 -- 172.21.15.103:0/2250484845 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f16c402a020 con 0x7f16cc103f70 2024-05-22T23:56:35.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.449+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f16cc062f70 con 0x7f16cc103f70 2024-05-22T23:56:35.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.450+0000 7f16c2a00700 1 -- 172.21.15.103:0/2250484845 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) ==== 74+0+133 (secure 0 0 0) 0x7f16c409d4b0 con 0x7f16cc103f70 2024-05-22T23:56:35.454 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:56:35.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.453+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f16b00a2ca0 msgr2=0x7f16b00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:35.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.453+0000 7f16d1200700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f16b00a2ca0 0x7f16b00a5160 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f16cc103430 tx=0x7f16bc009260 comp rx=0 tx=0).stop 2024-05-22T23:56:35.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.453+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 msgr2=0x7f16cc19fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:35.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.453+0000 7f16d1200700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 0x7f16cc19fb50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f16c400b070 tx=0x7f16c400cf70 comp rx=0 tx=0).stop 2024-05-22T23:56:35.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.454+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 shutdown_connections 2024-05-22T23:56:35.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.454+0000 7f16d1200700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f16b00a2ca0 0x7f16b00a5160 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.454+0000 7f16d1200700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16cc103f70 0x7f16cc19fb50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.454+0000 7f16d1200700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f16cc1023d0 0x7f16cc198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.454+0000 7f16d1200700 1 --2- 172.21.15.103:0/2250484845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16cc1035d0 0x7f16cc198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:35.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.454+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 >> 172.21.15.103:0/2250484845 conn(0x7f16cc0fd930 msgr2=0x7f16cc106870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:35.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.454+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 shutdown_connections 2024-05-22T23:56:35.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:35.454+0000 7f16d1200700 1 -- 172.21.15.103:0/2250484845 wait complete. 2024-05-22T23:56:36.023 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:35 smithi174 ceph-mon[36179]: purged_snaps scrub starts 2024-05-22T23:56:36.023 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:35 smithi174 ceph-mon[36179]: purged_snaps scrub ok 2024-05-22T23:56:36.023 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:36.023 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:35 smithi174 ceph-mon[36179]: osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] boot 2024-05-22T23:56:36.023 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:35 smithi174 ceph-mon[36179]: osdmap e82: 12 total, 12 up, 12 in 2024-05-22T23:56:36.024 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:35 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:36.024 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:35 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2250484845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-22T23:56:36.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:35 smithi105 ceph-mon[36426]: purged_snaps scrub starts 2024-05-22T23:56:36.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:35 smithi105 ceph-mon[36426]: purged_snaps scrub ok 2024-05-22T23:56:36.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:36.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:35 smithi105 ceph-mon[36426]: osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] boot 2024-05-22T23:56:36.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:35 smithi105 ceph-mon[36426]: osdmap e82: 12 total, 12 up, 12 in 2024-05-22T23:56:36.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:35 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:36.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:35 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2250484845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-22T23:56:36.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:35 smithi103 ceph-mon[31370]: purged_snaps scrub starts 2024-05-22T23:56:36.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:35 smithi103 ceph-mon[31370]: purged_snaps scrub ok 2024-05-22T23:56:36.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:36.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:35 smithi103 ceph-mon[31370]: osd.11 [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] boot 2024-05-22T23:56:36.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:35 smithi103 ceph-mon[31370]: osdmap e82: 12 total, 12 up, 12 in 2024-05-22T23:56:36.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:35 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-22T23:56:36.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:35 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2250484845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-22T23:56:36.279 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1716422194,"num_in_osds":12,"osd_in_since":1716422174,"num_remapped_pgs":0} 2024-05-22T23:56:36.279 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-22T23:56:36.703 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:56:37.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: osdmap e83: 12 total, 12 up, 12 in 2024-05-22T23:56:37.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:37.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: Detected new or changed devices on smithi174 2024-05-22T23:56:37.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.044 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi174 to 4272M 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.045 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: osdmap e83: 12 total, 12 up, 12 in 2024-05-22T23:56:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: Detected new or changed devices on smithi174 2024-05-22T23:56:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi174 to 4272M 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:37.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: osdmap e83: 12 total, 12 up, 12 in 2024-05-22T23:56:37.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:37.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: Detected new or changed devices on smithi174 2024-05-22T23:56:37.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi174 to 4272M 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:37.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:38.052 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:37 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:38.052 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:37 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:38.052 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:37 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:38.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:37 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:38.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:37 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:38.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:37 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:38.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:37 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:38.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:37 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:38.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:37 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:38.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 -- 172.21.15.103:0/2487256293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb5f8108420 msgr2=0x7fb5f8108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:38.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 --2- 172.21.15.103:0/2487256293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb5f8108420 0x7fb5f8108800 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009990 tx=0x7fb5e8009ca0 comp rx=0 tx=0).stop 2024-05-22T23:56:38.893 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:38 smithi103 ceph-mon[31370]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:38.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 -- 172.21.15.103:0/2487256293 shutdown_connections 2024-05-22T23:56:38.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 --2- 172.21.15.103:0/2487256293 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb5f8108420 0x7fb5f8108800 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:38.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 --2- 172.21.15.103:0/2487256293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f8102d80 0x7fb5f810f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:38.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 --2- 172.21.15.103:0/2487256293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81023e0 0x7fb5f8102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 -- 172.21.15.103:0/2487256293 >> 172.21.15.103:0/2487256293 conn(0x7fb5f8079430 msgr2=0x7fb5f8079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 -- 172.21.15.103:0/2487256293 shutdown_connections 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.888+0000 7fb5fd200700 1 -- 172.21.15.103:0/2487256293 wait complete. 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.889+0000 7fb5fd200700 1 Processor -- start 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.889+0000 7fb5fd200700 1 -- start start 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5fd200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81023e0 0x7fb5f81a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5fd200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f8102d80 0x7fb5f81a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5fd200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb5f8108420 0x7fb5f819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5fd200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f8112680 con 0x7fb5f81023e0 2024-05-22T23:56:38.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5fd200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f8112500 con 0x7fb5f8108420 2024-05-22T23:56:38.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5fd200700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f8112800 con 0x7fb5f8102d80 2024-05-22T23:56:38.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5f5e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81023e0 0x7fb5f81a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:38.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5f5e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81023e0 0x7fb5f81a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39668/0 (socket says 172.21.15.103:39668) 2024-05-22T23:56:38.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5f5e00700 1 -- 172.21.15.103:0/3218866692 learned_addr learned my addr 172.21.15.103:0/3218866692 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:38.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5ed400700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f8102d80 0x7fb5f81a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:38.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.890+0000 7fb5f6800700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb5f8108420 0x7fb5f819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:38.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ed400700 1 -- 172.21.15.103:0/3218866692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb5f8108420 msgr2=0x7fb5f819ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:38.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ed400700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb5f8108420 0x7fb5f819ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ed400700 1 -- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81023e0 msgr2=0x7fb5f81a0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ed400700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81023e0 0x7fb5f81a0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ed400700 1 -- 172.21.15.103:0/3218866692 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f819b400 con 0x7fb5f8102d80 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5f5e00700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81023e0 0x7fb5f81a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ed400700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f8102d80 0x7fb5f81a0f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000e9f0 tx=0x7fb5e000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ef400700 1 -- 172.21.15.103:0/3218866692 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e000cb70 con 0x7fb5f8102d80 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f819b6f0 con 0x7fb5f8102d80 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ef400700 1 -- 172.21.15.103:0/3218866692 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5e000ee90 con 0x7fb5f8102d80 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5ef400700 1 -- 172.21.15.103:0/3218866692 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e0003dc0 con 0x7fb5f8102d80 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.891+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f81a79d0 con 0x7fb5f8102d80 2024-05-22T23:56:38.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.893+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f8103ad0 con 0x7fb5f8102d80 2024-05-22T23:56:38.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.894+0000 7fb5ef400700 1 -- 172.21.15.103:0/3218866692 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb5e0004000 con 0x7fb5f8102d80 2024-05-22T23:56:38.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.896+0000 7fb5ef400700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb5dc0a2c20 0x7fb5dc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:38.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.896+0000 7fb5ef400700 1 -- 172.21.15.103:0/3218866692 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fb5e00ceda0 con 0x7fb5f8102d80 2024-05-22T23:56:38.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.896+0000 7fb5f5e00700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb5dc0a2c20 0x7fb5dc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:38.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.897+0000 7fb5f5e00700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb5dc0a2c20 0x7fb5dc0a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800b2e0 tx=0x7fb5e8009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:38.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:38.900+0000 7fb5ef400700 1 -- 172.21.15.103:0/3218866692 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb5e009c100 con 0x7fb5f8102d80 2024-05-22T23:56:39.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:38 smithi105 ceph-mon[36426]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:38 smithi174 ceph-mon[36179]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:39.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.123+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5f819bfe0 con 0x7fb5f8102d80 2024-05-22T23:56:39.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.124+0000 7fb5ef400700 1 -- 172.21.15.103:0/3218866692 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18515 (secure 0 0 0) 0x7fb5e009ba00 con 0x7fb5f8102d80 2024-05-22T23:56:39.129 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:56:39.129 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":83,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:56:35.739343+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:56:39.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.127+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb5dc0a2c20 msgr2=0x7fb5dc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:39.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.127+0000 7fb5fd200700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb5dc0a2c20 0x7fb5dc0a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb5e800b2e0 tx=0x7fb5e8009590 comp rx=0 tx=0).stop 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.127+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f8102d80 msgr2=0x7fb5f81a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.127+0000 7fb5fd200700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f8102d80 0x7fb5f81a0f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb5e000e9f0 tx=0x7fb5e000ed00 comp rx=0 tx=0).stop 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.128+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 shutdown_connections 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.128+0000 7fb5fd200700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb5f8108420 0x7fb5f819ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.128+0000 7fb5fd200700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb5dc0a2c20 0x7fb5dc0a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.128+0000 7fb5fd200700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb5f81023e0 0x7fb5f81a0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.128+0000 7fb5fd200700 1 --2- 172.21.15.103:0/3218866692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb5f8102d80 0x7fb5f81a0f90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.128+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 >> 172.21.15.103:0/3218866692 conn(0x7fb5f8079430 msgr2=0x7fb5f810d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.128+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 shutdown_connections 2024-05-22T23:56:39.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:39.128+0000 7fb5fd200700 1 -- 172.21.15.103:0/3218866692 wait complete. 2024-05-22T23:56:39.972 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-22T23:56:39.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-22T23:56:40.050 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:39 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3218866692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:56:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:39 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3218866692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:56:40.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:39 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3218866692' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:56:40.411 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:56:40.815 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:40 smithi103 ceph-mon[31370]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:40 smithi105 ceph-mon[36426]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:41.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:40 smithi174 ceph-mon[36179]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:42.261 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:42 smithi103 ceph-mon[31370]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:42.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:42 smithi105 ceph-mon[36426]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:42.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:42 smithi174 ceph-mon[36179]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 -- 172.21.15.103:0/3363770126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 msgr2=0x7f06c810aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 --2- 172.21.15.103:0/3363770126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 0x7f06c810aa30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f06b80099c0 tx=0x7f06b8009cd0 comp rx=0 tx=0).stop 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 -- 172.21.15.103:0/3363770126 shutdown_connections 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 --2- 172.21.15.103:0/3363770126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c8104f60 0x7f06c8111420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 --2- 172.21.15.103:0/3363770126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 0x7f06c810aa30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 --2- 172.21.15.103:0/3363770126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06c81045a0 0x7f06c8104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 -- 172.21.15.103:0/3363770126 >> 172.21.15.103:0/3363770126 conn(0x7f06c8100070 msgr2=0x7f06c8102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 -- 172.21.15.103:0/3363770126 shutdown_connections 2024-05-22T23:56:42.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.803+0000 7f06cf000700 1 -- 172.21.15.103:0/3363770126 wait complete. 2024-05-22T23:56:42.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.804+0000 7f06cf000700 1 Processor -- start 2024-05-22T23:56:42.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.804+0000 7f06cf000700 1 -- start start 2024-05-22T23:56:42.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cf000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06c81045a0 0x7f06c81a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:42.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cf000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c8104f60 0x7f06c81a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:42.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cf000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 0x7f06c819aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:42.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cf000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06c81148c0 con 0x7f06c810a650 2024-05-22T23:56:42.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cf000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f06c8114740 con 0x7f06c81045a0 2024-05-22T23:56:42.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cf000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06c8114a40 con 0x7f06c8104f60 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cd000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 0x7f06c819aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cd000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 0x7f06c819aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39676/0 (socket says 172.21.15.103:39676) 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cd000700 1 -- 172.21.15.103:0/3391824904 learned_addr learned my addr 172.21.15.103:0/3391824904 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06c7400700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c8104f60 0x7f06c81a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06c7e00700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06c81045a0 0x7f06c81a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cd000700 1 -- 172.21.15.103:0/3391824904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c8104f60 msgr2=0x7f06c81a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cd000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c8104f60 0x7f06c81a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cd000700 1 -- 172.21.15.103:0/3391824904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06c81045a0 msgr2=0x7f06c81a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cd000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06c81045a0 0x7f06c81a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.805+0000 7f06cd000700 1 -- 172.21.15.103:0/3391824904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06c819b200 con 0x7f06c810a650 2024-05-22T23:56:42.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.806+0000 7f06cd000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 0x7f06c819aaa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f06c000ef30 tx=0x7f06c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.806+0000 7f06c4c00700 1 -- 172.21.15.103:0/3391824904 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c00092e0 con 0x7f06c810a650 2024-05-22T23:56:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.806+0000 7f06c4c00700 1 -- 172.21.15.103:0/3391824904 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06c0004bb0 con 0x7f06c810a650 2024-05-22T23:56:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.806+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06c819b490 con 0x7f06c810a650 2024-05-22T23:56:42.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.806+0000 7f06c4c00700 1 -- 172.21.15.103:0/3391824904 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c0004280 con 0x7f06c810a650 2024-05-22T23:56:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.806+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06c81a77b0 con 0x7f06c810a650 2024-05-22T23:56:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.807+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06c810cae0 con 0x7f06c810a650 2024-05-22T23:56:42.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.809+0000 7f06c4c00700 1 -- 172.21.15.103:0/3391824904 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f06c001b030 con 0x7f06c810a650 2024-05-22T23:56:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.811+0000 7f06c4c00700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06ac0a2ca0 0x7f06ac0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.811+0000 7f06c4c00700 1 -- 172.21.15.103:0/3391824904 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f06c00ce680 con 0x7f06c810a650 2024-05-22T23:56:42.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.811+0000 7f06c7e00700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06ac0a2ca0 0x7f06ac0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:42.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.812+0000 7f06c7e00700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06ac0a2ca0 0x7f06ac0a5160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f06b8000c00 tx=0x7f06b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:42.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:42.815+0000 7f06c4c00700 1 -- 172.21.15.103:0/3391824904 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f06c0022020 con 0x7f06c810a650 2024-05-22T23:56:43.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.038+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f06c810c4c0 con 0x7f06c810a650 2024-05-22T23:56:43.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.039+0000 7f06c4c00700 1 -- 172.21.15.103:0/3391824904 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f06c0010040 con 0x7f06c810a650 2024-05-22T23:56:43.043 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-22T23:56:43.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06ac0a2ca0 msgr2=0x7f06ac0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:43.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06ac0a2ca0 0x7f06ac0a5160 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f06b8000c00 tx=0x7f06b8005fb0 comp rx=0 tx=0).stop 2024-05-22T23:56:43.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 msgr2=0x7f06c819aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:43.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 0x7f06c819aaa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f06c000ef30 tx=0x7f06c000c3a0 comp rx=0 tx=0).stop 2024-05-22T23:56:43.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 shutdown_connections 2024-05-22T23:56:43.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06c81045a0 0x7f06c81a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:43.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06ac0a2ca0 0x7f06ac0a5160 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:43.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06c810a650 0x7f06c819aaa0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:43.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 --2- 172.21.15.103:0/3391824904 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06c8104f60 0x7f06c81a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:43.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.042+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 >> 172.21.15.103:0/3391824904 conn(0x7f06c8100070 msgr2=0x7f06c81015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:43.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.043+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 shutdown_connections 2024-05-22T23:56:43.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:43.043+0000 7f06cf000700 1 -- 172.21.15.103:0/3391824904 wait complete. 2024-05-22T23:56:43.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:43 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3391824904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:56:43.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:43 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3391824904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:56:43.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:43 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3391824904' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:56:43.978 INFO:tasks.cephadm:Adding mds.a on smithi103 2024-05-22T23:56:43.978 INFO:tasks.cephadm:Adding mds.d on smithi103 2024-05-22T23:56:43.978 INFO:tasks.cephadm:Adding mds.g on smithi103 2024-05-22T23:56:43.978 INFO:tasks.cephadm:Adding mds.j on smithi103 2024-05-22T23:56:43.979 INFO:tasks.cephadm:Adding mds.b on smithi105 2024-05-22T23:56:43.979 INFO:tasks.cephadm:Adding mds.e on smithi105 2024-05-22T23:56:43.979 INFO:tasks.cephadm:Adding mds.h on smithi105 2024-05-22T23:56:43.979 INFO:tasks.cephadm:Adding mds.k on smithi105 2024-05-22T23:56:43.979 INFO:tasks.cephadm:Adding mds.c on smithi174 2024-05-22T23:56:43.979 INFO:tasks.cephadm:Adding mds.f on smithi174 2024-05-22T23:56:43.979 INFO:tasks.cephadm:Adding mds.i on smithi174 2024-05-22T23:56:43.979 INFO:tasks.cephadm:Adding mds.l on smithi174 2024-05-22T23:56:43.979 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph orch apply mds all '12;smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l' 2024-05-22T23:56:44.226 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-22T23:56:44.347 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:44 smithi174 ceph-mon[36179]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:44.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:44 smithi105 ceph-mon[36426]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:44.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:44 smithi103 ceph-mon[31370]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:45.863 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 -- 172.21.15.174:0/2437243055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 msgr2=0x7fda4c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:45.863 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 --2- 172.21.15.174:0/2437243055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 0x7fda4c101ad0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fda340099c0 tx=0x7fda34009cd0 comp rx=0 tx=0).stop 2024-05-22T23:56:45.863 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 -- 172.21.15.174:0/2437243055 shutdown_connections 2024-05-22T23:56:45.863 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 --2- 172.21.15.174:0/2437243055 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda4c102010 0x7fda4c1098d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:45.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 --2- 172.21.15.174:0/2437243055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda4c102930 0x7fda4c102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:45.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 --2- 172.21.15.174:0/2437243055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 0x7fda4c101ad0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:45.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 -- 172.21.15.174:0/2437243055 >> 172.21.15.174:0/2437243055 conn(0x7fda4c0fc0c0 msgr2=0x7fda4c0fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:45.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 -- 172.21.15.174:0/2437243055 shutdown_connections 2024-05-22T23:56:45.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 -- 172.21.15.174:0/2437243055 wait complete. 2024-05-22T23:56:45.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.859+0000 7fda51bfb700 1 Processor -- start 2024-05-22T23:56:45.864 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda51bfb700 1 -- start start 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda51bfb700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 0x7fda4c198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda51bfb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda4c102010 0x7fda4c198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda51bfb700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda4c102930 0x7fda4c19fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda51bfb700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c10d000 con 0x7fda4c102930 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda51bfb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c10ce80 con 0x7fda4c102010 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda51bfb700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c10d180 con 0x7fda4c101670 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 0x7fda4c198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 0x7fda4c198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:55818/0 (socket says 172.21.15.174:55818) 2024-05-22T23:56:45.866 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 -- 172.21.15.174:0/563515992 learned_addr learned my addr 172.21.15.174:0/563515992 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4affd700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda4c102010 0x7fda4c198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 -- 172.21.15.174:0/563515992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda4c102010 msgr2=0x7fda4c198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda4c102010 0x7fda4c198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 -- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda4c102930 msgr2=0x7fda4c19fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda4c102930 0x7fda4c19fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 -- 172.21.15.174:0/563515992 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c1a02e0 con 0x7fda4c101670 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda4b7fe700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 0x7fda4c198680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda4400b9f0 tx=0x7fda4400bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda48ff9700 1 -- 172.21.15.174:0/563515992 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda44019070 con 0x7fda4c101670 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4c1a05d0 con 0x7fda4c101670 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.860+0000 7fda48ff9700 1 -- 172.21.15.174:0/563515992 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda4400ce30 con 0x7fda4c101670 2024-05-22T23:56:45.867 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.861+0000 7fda48ff9700 1 -- 172.21.15.174:0/563515992 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda440125a0 con 0x7fda4c101670 2024-05-22T23:56:45.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.861+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4c1a0b10 con 0x7fda4c101670 2024-05-22T23:56:45.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.862+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda4c069050 con 0x7fda4c101670 2024-05-22T23:56:45.868 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.862+0000 7fda48ff9700 1 -- 172.21.15.174:0/563515992 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fda4401c070 con 0x7fda4c101670 2024-05-22T23:56:45.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.863+0000 7fda48ff9700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fda380a2c20 0x7fda380a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:45.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.863+0000 7fda48ff9700 1 -- 172.21.15.174:0/563515992 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fda440ce220 con 0x7fda4c101670 2024-05-22T23:56:45.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.864+0000 7fda4affd700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fda380a2c20 0x7fda380a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:45.869 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.864+0000 7fda4affd700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fda380a2c20 0x7fda380a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fda34000c00 tx=0x7fda34005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:45.871 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:45.868+0000 7fda48ff9700 1 -- 172.21.15.174:0/563515992 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fda4409b620 con 0x7fda4c101670 2024-05-22T23:56:46.077 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.073+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}) -- 0x7fda4c0fe1d0 con 0x7fda380a2c20 2024-05-22T23:56:46.094 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.090+0000 7fda48ff9700 1 -- 172.21.15.174:0/563515992 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fda4c0fe1d0 con 0x7fda380a2c20 2024-05-22T23:56:46.094 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled mds.all update... 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fda380a2c20 msgr2=0x7fda380a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fda380a2c20 0x7fda380a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fda34000c00 tx=0x7fda34005fb0 comp rx=0 tx=0).stop 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 msgr2=0x7fda4c198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 0x7fda4c198680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda4400b9f0 tx=0x7fda4400bd00 comp rx=0 tx=0).stop 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 shutdown_connections 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fda380a2c20 0x7fda380a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fda4c102930 0x7fda4c19fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fda4c101670 0x7fda4c198680 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:46.097 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 --2- 172.21.15.174:0/563515992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda4c102010 0x7fda4c198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:46.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 >> 172.21.15.174:0/563515992 conn(0x7fda4c0fc0c0 msgr2=0x7fda4c0fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:46.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 shutdown_connections 2024-05-22T23:56:46.098 INFO:teuthology.orchestra.run.smithi174.stderr:2024-05-22T23:56:46.093+0000 7fda51bfb700 1 -- 172.21.15.174:0/563515992 wait complete. 2024-05-22T23:56:46.841 DEBUG:teuthology.orchestra.run.smithi103:mds.a> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.a.service 2024-05-22T23:56:46.849 DEBUG:teuthology.orchestra.run.smithi103:mds.d> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.d.service 2024-05-22T23:56:46.897 DEBUG:teuthology.orchestra.run.smithi103:mds.g> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.g.service 2024-05-22T23:56:46.943 DEBUG:teuthology.orchestra.run.smithi103:mds.j> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.j.service 2024-05-22T23:56:46.990 DEBUG:teuthology.orchestra.run.smithi105:mds.b> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.b.service 2024-05-22T23:56:46.993 DEBUG:teuthology.orchestra.run.smithi105:mds.e> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.e.service 2024-05-22T23:56:47.037 DEBUG:teuthology.orchestra.run.smithi105:mds.h> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.h.service 2024-05-22T23:56:47.080 DEBUG:teuthology.orchestra.run.smithi105:mds.k> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.k.service 2024-05-22T23:56:47.122 DEBUG:teuthology.orchestra.run.smithi174:mds.c> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.c.service 2024-05-22T23:56:47.125 DEBUG:teuthology.orchestra.run.smithi174:mds.f> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.f.service 2024-05-22T23:56:47.167 DEBUG:teuthology.orchestra.run.smithi174:mds.i> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.i.service 2024-05-22T23:56:47.210 DEBUG:teuthology.orchestra.run.smithi174:mds.l> sudo journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.l.service 2024-05-22T23:56:47.253 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-22T23:56:47.254 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:56:47.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:47 smithi105 ceph-mon[36426]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:56:47.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:47 smithi105 ceph-mon[36426]: Saving service mds.all spec with placement smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-05-22T23:56:47.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:47 smithi105 ceph-mon[36426]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:47.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:47 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:47.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:47 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:47.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:47 smithi103 ceph-mon[31370]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:56:47.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:47 smithi103 ceph-mon[31370]: Saving service mds.all spec with placement smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-05-22T23:56:47.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:47 smithi103 ceph-mon[31370]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:47.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:47 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:47.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:47 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:47.354 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:47 smithi174 ceph-mon[36179]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:56:47.354 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:47 smithi174 ceph-mon[36179]: Saving service mds.all spec with placement smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-05-22T23:56:47.354 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:47 smithi174 ceph-mon[36179]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:47.355 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:47 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:47.355 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:47 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:56:47.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 -- 172.21.15.103:0/985030728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281038f0 msgr2=0x7f2f28103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:47.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/985030728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281038f0 0x7f2f28103d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009900 tx=0x7f2f1c02a410 comp rx=0 tx=0).stop 2024-05-22T23:56:47.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 -- 172.21.15.103:0/985030728 shutdown_connections 2024-05-22T23:56:47.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/985030728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f281054a0 0x7f2f2810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/985030728 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f28104ae0 0x7f2f28104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/985030728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281038f0 0x7f2f28103d10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 -- 172.21.15.103:0/985030728 >> 172.21.15.103:0/985030728 conn(0x7f2f280fee80 msgr2=0x7f2f281012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:47.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 -- 172.21.15.103:0/985030728 shutdown_connections 2024-05-22T23:56:47.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.490+0000 7f2f2ee00640 1 -- 172.21.15.103:0/985030728 wait complete. 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.491+0000 7f2f2ee00640 1 Processor -- start 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.491+0000 7f2f2ee00640 1 -- start start 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2ee00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f281038f0 0x7f2f28199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2ee00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f28104ae0 0x7f2f2819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2ee00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281054a0 0x7f2f281a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2ee00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2810f560 con 0x7f2f28104ae0 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2ee00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2810f3e0 con 0x7f2f281038f0 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2ee00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f2810f6e0 con 0x7f2f281054a0 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2e400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281054a0 0x7f2f281a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f28104ae0 0x7f2f2819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:47.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2e400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281054a0 0x7f2f281a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34936/0 (socket says 172.21.15.103:34936) 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2e400640 1 -- 172.21.15.103:0/1851599654 learned_addr learned my addr 172.21.15.103:0/1851599654 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2da00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f281038f0 0x7f2f28199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2e400640 1 -- 172.21.15.103:0/1851599654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f281038f0 msgr2=0x7f2f28199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2e400640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f281038f0 0x7f2f28199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2e400640 1 -- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f28104ae0 msgr2=0x7f2f2819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2e400640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f28104ae0 0x7f2f2819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2e400640 1 -- 172.21.15.103:0/1851599654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f281a1890 con 0x7f2f281054a0 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.492+0000 7f2f2d000640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f28104ae0 0x7f2f2819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.493+0000 7f2f2e400640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281054a0 0x7f2f281a11f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000efd0 tx=0x7f2f2000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.493+0000 7f2f26000640 1 -- 172.21.15.103:0/1851599654 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f20003b70 con 0x7f2f281054a0 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.493+0000 7f2f26000640 1 -- 172.21.15.103:0/1851599654 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f20003d10 con 0x7f2f281054a0 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.493+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f281a1b80 con 0x7f2f281054a0 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.493+0000 7f2f26000640 1 -- 172.21.15.103:0/1851599654 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f200109f0 con 0x7f2f281054a0 2024-05-22T23:56:47.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.493+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f281a2050 con 0x7f2f281054a0 2024-05-22T23:56:47.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.494+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f2810ad30 con 0x7f2f281054a0 2024-05-22T23:56:47.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.496+0000 7f2f26000640 1 -- 172.21.15.103:0/1851599654 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2f20010b90 con 0x7f2f281054a0 2024-05-22T23:56:47.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.497+0000 7f2f26000640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f100a2b90 0x7f2f100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:47.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.498+0000 7f2f26000640 1 -- 172.21.15.103:0/1851599654 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f2f200cf280 con 0x7f2f281054a0 2024-05-22T23:56:47.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.498+0000 7f2f2da00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f100a2b90 0x7f2f100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:47.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.499+0000 7f2f2da00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f100a2b90 0x7f2f100a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c005ab0 tx=0x7f2f1c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.504+0000 7f2f26000640 1 -- 172.21.15.103:0/1851599654 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2f2009c5e0 con 0x7f2f281054a0 2024-05-22T23:56:47.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.704+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2f2802d450 con 0x7f2f281054a0 2024-05-22T23:56:47.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.706+0000 7f2f26000640 1 -- 172.21.15.103:0/1851599654 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18515 (secure 0 0 0) 0x7f2f2009bee0 con 0x7f2f281054a0 2024-05-22T23:56:47.711 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:56:47.711 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":83,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:56:35.739343+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:56:47.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f100a2b90 msgr2=0x7f2f100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:47.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f100a2b90 0x7f2f100a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c005ab0 tx=0x7f2f1c000bc0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281054a0 msgr2=0x7f2f281a11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:47.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281054a0 0x7f2f281a11f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000efd0 tx=0x7f2f2000c6a0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 shutdown_connections 2024-05-22T23:56:47.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f100a2b90 0x7f2f100a5050 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f281054a0 0x7f2f281a11f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f28104ae0 0x7f2f2819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 --2- 172.21.15.103:0/1851599654 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f281038f0 0x7f2f28199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 >> 172.21.15.103:0/1851599654 conn(0x7f2f280fee80 msgr2=0x7f2f28107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:47.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 shutdown_connections 2024-05-22T23:56:47.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.709+0000 7f2f2ee00640 1 -- 172.21.15.103:0/1851599654 wait complete. 2024-05-22T23:56:47.734 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-22T23:56:47.734 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-22T23:56:47.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 -- 172.21.15.103:0/176639700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 msgr2=0x7fafd8103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:47.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 --2- 172.21.15.103:0/176639700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 0x7fafd8103ef0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009900 tx=0x7fafc802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 -- 172.21.15.103:0/176639700 shutdown_connections 2024-05-22T23:56:47.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 --2- 172.21.15.103:0/176639700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd8105680 0x7fafd810bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 --2- 172.21.15.103:0/176639700 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd8104cc0 0x7fafd8105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 --2- 172.21.15.103:0/176639700 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 0x7fafd8103ef0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 -- 172.21.15.103:0/176639700 >> 172.21.15.103:0/176639700 conn(0x7fafd80ff060 msgr2=0x7fafd8101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:47.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 -- 172.21.15.103:0/176639700 shutdown_connections 2024-05-22T23:56:47.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.933+0000 7fafdde00640 1 -- 172.21.15.103:0/176639700 wait complete. 2024-05-22T23:56:47.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.934+0000 7fafdde00640 1 Processor -- start 2024-05-22T23:56:47.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.934+0000 7fafdde00640 1 -- start start 2024-05-22T23:56:47.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafdde00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 0x7fafd8199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafdde00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd8104cc0 0x7fafd819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafdde00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd8105680 0x7fafd81a13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafdde00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810f790 con 0x7fafd8105680 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafdde00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810f610 con 0x7fafd8104cc0 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafdde00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fafd810f910 con 0x7fafd8103ad0 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd8105680 0x7fafd81a13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd8105680 0x7fafd81a13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33702/0 (socket says 172.21.15.103:33702) 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd7400640 1 -- 172.21.15.103:0/1830642839 learned_addr learned my addr 172.21.15.103:0/1830642839 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd6a00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 0x7fafd8199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafce000640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd8104cc0 0x7fafd819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd6a00640 1 -- 172.21.15.103:0/1830642839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd8104cc0 msgr2=0x7fafd819a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:47.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd6a00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd8104cc0 0x7fafd819a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd6a00640 1 -- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd8105680 msgr2=0x7fafd81a13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd6a00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd8105680 0x7fafd81a13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd6a00640 1 -- 172.21.15.103:0/1830642839 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd81a1a50 con 0x7fafd8103ad0 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.935+0000 7fafd7400640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd8105680 0x7fafd81a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.936+0000 7fafd6a00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 0x7fafd8199e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009230 tx=0x7fafc8032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.936+0000 7fafdca00640 1 -- 172.21.15.103:0/1830642839 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc803f030 con 0x7fafd8103ad0 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.936+0000 7fafdca00640 1 -- 172.21.15.103:0/1830642839 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc8003ce0 con 0x7fafd8103ad0 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.936+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd81a1ce0 con 0x7fafd8103ad0 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.936+0000 7fafdca00640 1 -- 172.21.15.103:0/1830642839 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8035070 con 0x7fafd8103ad0 2024-05-22T23:56:47.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.936+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd81a21a0 con 0x7fafd8103ad0 2024-05-22T23:56:47.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.937+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd810af10 con 0x7fafd8103ad0 2024-05-22T23:56:47.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.939+0000 7fafdca00640 1 -- 172.21.15.103:0/1830642839 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fafc8042030 con 0x7fafd8103ad0 2024-05-22T23:56:47.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.940+0000 7fafdca00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fafbc0a2c10 0x7fafbc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:47.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.941+0000 7fafdca00640 1 -- 172.21.15.103:0/1830642839 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fafc80ef060 con 0x7fafd8103ad0 2024-05-22T23:56:47.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.941+0000 7fafce000640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fafbc0a2c10 0x7fafbc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:47.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.941+0000 7fafce000640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fafbc0a2c10 0x7fafbc0a50d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fafd819b340 tx=0x7fafc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:47.948 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:47.944+0000 7fafdca00640 1 -- 172.21.15.103:0/1830642839 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fafc80bbd70 con 0x7fafd8103ad0 2024-05-22T23:56:48.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.146+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fafd8062470 con 0x7fafd8103ad0 2024-05-22T23:56:48.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.146+0000 7fafdca00640 1 -- 172.21.15.103:0/1830642839 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7fafc80c0070 con 0x7fafd8103ad0 2024-05-22T23:56:48.151 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-22T23:56:48.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fafbc0a2c10 msgr2=0x7fafbc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:48.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fafbc0a2c10 0x7fafbc0a50d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fafd819b340 tx=0x7fafc0005f90 comp rx=0 tx=0).stop 2024-05-22T23:56:48.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 msgr2=0x7fafd8199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 0x7fafd8199e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009230 tx=0x7fafc8032ce0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 shutdown_connections 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fafbc0a2c10 0x7fafbc0a50d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafd8105680 0x7fafd81a13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafd8104cc0 0x7fafd819a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 --2- 172.21.15.103:0/1830642839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fafd8103ad0 0x7fafd8199e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.149+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 >> 172.21.15.103:0/1830642839 conn(0x7fafd80ff060 msgr2=0x7fafd81080e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.150+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 shutdown_connections 2024-05-22T23:56:48.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.150+0000 7fafdde00640 1 -- 172.21.15.103:0/1830642839 wait complete. 2024-05-22T23:56:48.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:56:48.186 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:48.186 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:48.186 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:48.186 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:48.187 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:48.187 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:48 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:48.187 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:48 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1851599654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:56:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.374+0000 7f98f2a00640 1 -- 172.21.15.103:0/3894319465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98ec103af0 msgr2=0x7f98ec103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:48.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.374+0000 7f98f2a00640 1 --2- 172.21.15.103:0/3894319465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98ec103af0 0x7f98ec103f10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009be0 tx=0x7f98dc02a6d0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.375+0000 7f98f2a00640 1 -- 172.21.15.103:0/3894319465 shutdown_connections 2024-05-22T23:56:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.375+0000 7f98f2a00640 1 --2- 172.21.15.103:0/3894319465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98ec1056a0 0x7f98ec10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.375+0000 7f98f2a00640 1 --2- 172.21.15.103:0/3894319465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98ec104ce0 0x7f98ec105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.375+0000 7f98f2a00640 1 --2- 172.21.15.103:0/3894319465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98ec103af0 0x7f98ec103f10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.375+0000 7f98f2a00640 1 -- 172.21.15.103:0/3894319465 >> 172.21.15.103:0/3894319465 conn(0x7f98ec0ff060 msgr2=0x7f98ec1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.375+0000 7f98f2a00640 1 -- 172.21.15.103:0/3894319465 shutdown_connections 2024-05-22T23:56:48.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.375+0000 7f98f2a00640 1 -- 172.21.15.103:0/3894319465 wait complete. 2024-05-22T23:56:48.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.376+0000 7f98f2a00640 1 Processor -- start 2024-05-22T23:56:48.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.376+0000 7f98f2a00640 1 -- start start 2024-05-22T23:56:48.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.376+0000 7f98f2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98ec103af0 0x7f98ec199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.376+0000 7f98f2a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98ec104ce0 0x7f98ec19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.376+0000 7f98f2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98ec1056a0 0x7f98ec1a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.376+0000 7f98f2a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98ec10f770 con 0x7f98ec103af0 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.376+0000 7f98f2a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f98ec10f5f0 con 0x7f98ec1056a0 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.376+0000 7f98f2a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98ec10f8f0 con 0x7f98ec104ce0 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98ebe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98ec103af0 0x7f98ec199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98ec104ce0 0x7f98ec19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98ec104ce0 0x7f98ec19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34984/0 (socket says 172.21.15.103:34984) 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 -- 172.21.15.103:0/1653682414 learned_addr learned my addr 172.21.15.103:0/1653682414 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98ebe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98ec103af0 0x7f98ec199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33720/0 (socket says 172.21.15.103:33720) 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98f0a00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98ec1056a0 0x7f98ec1a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:48.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 -- 172.21.15.103:0/1653682414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98ec1056a0 msgr2=0x7f98ec1a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98ec1056a0 0x7f98ec1a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 -- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98ec103af0 msgr2=0x7f98ec199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98ec103af0 0x7f98ec199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 -- 172.21.15.103:0/1653682414 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98ec1a1a30 con 0x7f98ec104ce0 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.377+0000 7f98e3400640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98ec104ce0 0x7f98ec19a350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98d400e9c0 tx=0x7f98d400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.378+0000 7f98e9600640 1 -- 172.21.15.103:0/1653682414 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d400cde0 con 0x7f98ec104ce0 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.378+0000 7f98e9600640 1 -- 172.21.15.103:0/1653682414 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98d40035f0 con 0x7f98ec104ce0 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.378+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98ec1a1d20 con 0x7f98ec104ce0 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.378+0000 7f98e9600640 1 -- 172.21.15.103:0/1653682414 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d4010690 con 0x7f98ec104ce0 2024-05-22T23:56:48.384 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.378+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98ec1a2260 con 0x7f98ec104ce0 2024-05-22T23:56:48.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.379+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98ec10ae30 con 0x7f98ec104ce0 2024-05-22T23:56:48.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.380+0000 7f98e9600640 1 -- 172.21.15.103:0/1653682414 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f98d40037a0 con 0x7f98ec104ce0 2024-05-22T23:56:48.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.382+0000 7f98e9600640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f98d00a2c10 0x7f98d00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:48.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.382+0000 7f98e9600640 1 -- 172.21.15.103:0/1653682414 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f98d40ceec0 con 0x7f98ec104ce0 2024-05-22T23:56:48.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.382+0000 7f98ebe00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f98d00a2c10 0x7f98d00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:48.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.383+0000 7f98ebe00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f98d00a2c10 0x7f98d00a50d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f98dc02abf0 tx=0x7f98dc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:48.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.386+0000 7f98e9600640 1 -- 172.21.15.103:0/1653682414 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f98d409c220 con 0x7f98ec104ce0 2024-05-22T23:56:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:48 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:48 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1851599654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:56:48.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:48.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:56:48.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:48.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:48.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:48.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:48 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:48.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:48 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1851599654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:56:48.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.627+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98ec062470 con 0x7f98ec104ce0 2024-05-22T23:56:48.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.628+0000 7f98e9600640 1 -- 172.21.15.103:0/1653682414 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f98d409bb20 con 0x7f98ec104ce0 2024-05-22T23:56:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:56:48.633 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-05-22T23:56:48.633 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 1 2024-05-22T23:56:48.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.630+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f98d00a2c10 msgr2=0x7f98d00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:48.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.630+0000 7f98f2a00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f98d00a2c10 0x7f98d00a50d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f98dc02abf0 tx=0x7f98dc005fd0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.630+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98ec104ce0 msgr2=0x7f98ec19a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:48.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.630+0000 7f98f2a00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98ec104ce0 0x7f98ec19a350 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98d400e9c0 tx=0x7f98d400ee80 comp rx=0 tx=0).stop 2024-05-22T23:56:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.631+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 shutdown_connections 2024-05-22T23:56:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.631+0000 7f98f2a00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f98d00a2c10 0x7f98d00a50d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.631+0000 7f98f2a00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98ec1056a0 0x7f98ec1a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.631+0000 7f98f2a00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98ec104ce0 0x7f98ec19a350 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.631+0000 7f98f2a00640 1 --2- 172.21.15.103:0/1653682414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98ec103af0 0x7f98ec199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.631+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 >> 172.21.15.103:0/1653682414 conn(0x7f98ec0ff060 msgr2=0x7f98ec108000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.631+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 shutdown_connections 2024-05-22T23:56:48.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:48.631+0000 7f98f2a00640 1 -- 172.21.15.103:0/1653682414 wait complete. 2024-05-22T23:56:49.259 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:49 smithi174 ceph-mon[36179]: Deploying daemon mds.l on smithi174 2024-05-22T23:56:49.259 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:49 smithi174 ceph-mon[36179]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:49.259 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:49 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1830642839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:56:49.259 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:49 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1653682414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:49 smithi105 ceph-mon[36426]: Deploying daemon mds.l on smithi174 2024-05-22T23:56:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:49 smithi105 ceph-mon[36426]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:49 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1830642839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:56:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:49 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1653682414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:49.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:49 smithi103 ceph-mon[31370]: Deploying daemon mds.l on smithi174 2024-05-22T23:56:49.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:49 smithi103 ceph-mon[31370]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:49.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:49 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1830642839' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:56:49.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:49 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1653682414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:49.618 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:49 smithi174 systemd[1]: Starting Ceph mds.l for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:56:49.935 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:49 smithi174 podman[66482]: 2024-05-22 23:56:49.557330569 +0000 UTC m=+0.017051297 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:56:49.935 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:49 smithi174 podman[66482]: 2024-05-22 23:56:49.662391482 +0000 UTC m=+0.122112204 container create 3c1f3b8310c09aeae4813e8efa351b294a4f4954fd6a835a9d2695ad67319396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-22T23:56:49.935 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:49 smithi174 podman[66482]: 2024-05-22 23:56:49.931763597 +0000 UTC m=+0.391484320 container init 3c1f3b8310c09aeae4813e8efa351b294a4f4954fd6a835a9d2695ad67319396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l, RELEASE=squid-2e1f762, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-22T23:56:50.368 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:49 smithi174 podman[66482]: 2024-05-22 23:56:49.934446353 +0000 UTC m=+0.394167077 container start 3c1f3b8310c09aeae4813e8efa351b294a4f4954fd6a835a9d2695ad67319396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-22T23:56:50.368 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:49 smithi174 ceph-mds[66501]: -- 172.21.15.174:0/1238242863 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5594fe95fa40 con 0x5594fe933000 2024-05-22T23:56:50.368 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:49 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l[66497]: starting mds.l at 2024-05-22T23:56:50.368 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:50 smithi174 bash[66482]: 3c1f3b8310c09aeae4813e8efa351b294a4f4954fd6a835a9d2695ad67319396 2024-05-22T23:56:50.369 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 22 23:56:50 smithi174 systemd[1]: Started Ceph mds.l for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:56:50.655 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:56:50.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 -- 172.21.15.103:0/2080862124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0108ee0 msgr2=0x7f89a0109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:50.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 --2- 172.21.15.103:0/2080862124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0108ee0 0x7f89a0109360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8988009900 tx=0x7f898802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:56:50.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 -- 172.21.15.103:0/2080862124 shutdown_connections 2024-05-22T23:56:50.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 --2- 172.21.15.103:0/2080862124 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01098a0 0x7f89a0110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:50.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 --2- 172.21.15.103:0/2080862124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0108ee0 0x7f89a0109360 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:50.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 --2- 172.21.15.103:0/2080862124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107cf0 0x7f89a0108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:50.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 -- 172.21.15.103:0/2080862124 >> 172.21.15.103:0/2080862124 conn(0x7f89a0076cb0 msgr2=0x7f89a00790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:50.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 -- 172.21.15.103:0/2080862124 shutdown_connections 2024-05-22T23:56:50.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.866+0000 7f899f200640 1 -- 172.21.15.103:0/2080862124 wait complete. 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.867+0000 7f899f200640 1 Processor -- start 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.867+0000 7f899f200640 1 -- start start 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899f200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107cf0 0x7f89a019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899f200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0108ee0 0x7f89a019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01098a0 0x7f89a01a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899f200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89a01139c0 con 0x7f89a01098a0 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899f200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89a0113840 con 0x7f89a0107cf0 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899f200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89a0113b40 con 0x7f89a0108ee0 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01098a0 0x7f89a01a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:50.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899de00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107cf0 0x7f89a019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899de00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107cf0 0x7f89a019dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:33186/0 (socket says 172.21.15.103:33186) 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899de00640 1 -- 172.21.15.103:0/2870761811 learned_addr learned my addr 172.21.15.103:0/2870761811 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899d400640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0108ee0 0x7f89a019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899de00640 1 -- 172.21.15.103:0/2870761811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0108ee0 msgr2=0x7f89a019e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899de00640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0108ee0 0x7f89a019e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899de00640 1 -- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01098a0 msgr2=0x7f89a01a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899de00640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01098a0 0x7f89a01a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.868+0000 7f899de00640 1 -- 172.21.15.103:0/2870761811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a01a5c00 con 0x7f89a0107cf0 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.869+0000 7f899de00640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107cf0 0x7f89a019dfe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f899000eb70 tx=0x7f899000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.869+0000 7f8996a00640 1 -- 172.21.15.103:0/2870761811 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8990003b70 con 0x7f89a0107cf0 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.869+0000 7f8996a00640 1 -- 172.21.15.103:0/2870761811 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f899000ee70 con 0x7f89a0107cf0 2024-05-22T23:56:50.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.869+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a01a5ef0 con 0x7f89a0107cf0 2024-05-22T23:56:50.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.869+0000 7f8996a00640 1 -- 172.21.15.103:0/2870761811 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8990010830 con 0x7f89a0107cf0 2024-05-22T23:56:50.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.869+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a01a6300 con 0x7f89a0107cf0 2024-05-22T23:56:50.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.870+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a010f010 con 0x7f89a0107cf0 2024-05-22T23:56:50.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.872+0000 7f8996a00640 1 -- 172.21.15.103:0/2870761811 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8990004020 con 0x7f89a0107cf0 2024-05-22T23:56:50.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.873+0000 7f8996a00640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89840a2b90 0x7f89840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:50.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.873+0000 7f8996a00640 1 -- 172.21.15.103:0/2870761811 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f89900cf290 con 0x7f89a0107cf0 2024-05-22T23:56:50.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.874+0000 7f899d400640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89840a2b90 0x7f89840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:50.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.874+0000 7f899d400640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89840a2b90 0x7f89840a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8988031440 tx=0x7f89880315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:50.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:50.877+0000 7f8996a00640 1 -- 172.21.15.103:0/2870761811 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f899009c5f0 con 0x7f89a0107cf0 2024-05-22T23:56:51.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6832/1973931443,v1:172.21.15.174:6833/1973931443] up:boot 2024-05-22T23:56:51.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: fsmap 1 up:standby 2024-05-22T23:56:51.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-22T23:56:51.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:51.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:50 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6832/1973931443,v1:172.21.15.174:6833/1973931443] up:boot 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: fsmap 1 up:standby 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:51.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:50 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:51.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.121+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89a002d200 con 0x7f89a0107cf0 2024-05-22T23:56:51.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.122+0000 7f8996a00640 1 -- 172.21.15.103:0/2870761811 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f899009bef0 con 0x7f89a0107cf0 2024-05-22T23:56:51.126 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:56:51.126 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-22T23:56:51.127 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 2 2024-05-22T23:56:51.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.124+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89840a2b90 msgr2=0x7f89840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:51.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.124+0000 7f899f200640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89840a2b90 0x7f89840a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8988031440 tx=0x7f89880315b0 comp rx=0 tx=0).stop 2024-05-22T23:56:51.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.124+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107cf0 msgr2=0x7f89a019dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:51.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.124+0000 7f899f200640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107cf0 0x7f89a019dfe0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f899000eb70 tx=0x7f899000c6a0 comp rx=0 tx=0).stop 2024-05-22T23:56:51.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.125+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 shutdown_connections 2024-05-22T23:56:51.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.125+0000 7f899f200640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89840a2b90 0x7f89840a5050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:51.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.125+0000 7f899f200640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01098a0 0x7f89a01a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:51.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.125+0000 7f899f200640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0108ee0 0x7f89a019e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:51.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.125+0000 7f899f200640 1 --2- 172.21.15.103:0/2870761811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107cf0 0x7f89a019dfe0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:51.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.125+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 >> 172.21.15.103:0/2870761811 conn(0x7f89a0076cb0 msgr2=0x7f89a010c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:51.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.125+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 shutdown_connections 2024-05-22T23:56:51.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:51.125+0000 7f899f200640 1 -- 172.21.15.103:0/2870761811 wait complete. 2024-05-22T23:56:51.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6832/1973931443,v1:172.21.15.174:6833/1973931443] up:boot 2024-05-22T23:56:51.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: fsmap 1 up:standby 2024-05-22T23:56:51.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-22T23:56:51.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:51.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:51.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:51.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:51.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:50 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:52.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:52 smithi174 ceph-mon[36179]: Deploying daemon mds.k on smithi105 2024-05-22T23:56:52.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:52 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2870761811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:52.428 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:52 smithi105 ceph-mon[36426]: Deploying daemon mds.k on smithi105 2024-05-22T23:56:52.428 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:52 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2870761811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:52.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:52 smithi103 ceph-mon[31370]: Deploying daemon mds.k on smithi105 2024-05-22T23:56:52.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:52 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2870761811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:53.059 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 22 23:56:53 smithi105 systemd[1]: Started Ceph mds.k for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:56:53.150 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:56:53.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 -- 172.21.15.103:0/269009072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c103490 msgr2=0x7f627c109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:53.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 --2- 172.21.15.103:0/269009072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c103490 0x7f627c109d20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6274009be0 tx=0x7f627402a6c0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 -- 172.21.15.103:0/269009072 shutdown_connections 2024-05-22T23:56:53.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 --2- 172.21.15.103:0/269009072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c103490 0x7f627c109d20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 --2- 172.21.15.103:0/269009072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c102ad0 0x7f627c102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 --2- 172.21.15.103:0/269009072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c1018e0 0x7f627c101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 -- 172.21.15.103:0/269009072 >> 172.21.15.103:0/269009072 conn(0x7f627c0fce50 msgr2=0x7f627c0ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:53.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 -- 172.21.15.103:0/269009072 shutdown_connections 2024-05-22T23:56:53.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.334+0000 7f6280e00640 1 -- 172.21.15.103:0/269009072 wait complete. 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.335+0000 7f6280e00640 1 Processor -- start 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.335+0000 7f6280e00640 1 -- start start 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6280e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c1018e0 0x7f627c199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6280e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c102ad0 0x7f627c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6280e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c103490 0x7f627c1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6280e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f627c10d560 con 0x7f627c103490 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6280e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f627c10d3e0 con 0x7f627c102ad0 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6280e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f627c10d6e0 con 0x7f627c1018e0 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f627a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c103490 0x7f627c1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:53.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f627a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c103490 0x7f627c1a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33764/0 (socket says 172.21.15.103:33764) 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6271400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c102ad0 0x7f627c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f627a800640 1 -- 172.21.15.103:0/4173575160 learned_addr learned my addr 172.21.15.103:0/4173575160 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6279e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c1018e0 0x7f627c199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6271400640 1 -- 172.21.15.103:0/4173575160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c1018e0 msgr2=0x7f627c199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6271400640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c1018e0 0x7f627c199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6271400640 1 -- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c103490 msgr2=0x7f627c1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6271400640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c103490 0x7f627c1a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f6271400640 1 -- 172.21.15.103:0/4173575160 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627c1a19b0 con 0x7f627c102ad0 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.336+0000 7f627a800640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c103490 0x7f627c1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.337+0000 7f6271400640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c102ad0 0x7f627c19a2e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f626400e790 tx=0x7f626400ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.337+0000 7f6273400640 1 -- 172.21.15.103:0/4173575160 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626400eea0 con 0x7f627c102ad0 2024-05-22T23:56:53.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.337+0000 7f6273400640 1 -- 172.21.15.103:0/4173575160 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f626400ccc0 con 0x7f627c102ad0 2024-05-22T23:56:53.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.337+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f627c1a1ca0 con 0x7f627c102ad0 2024-05-22T23:56:53.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.337+0000 7f6273400640 1 -- 172.21.15.103:0/4173575160 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6264018780 con 0x7f627c102ad0 2024-05-22T23:56:53.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.337+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f627c1a2190 con 0x7f627c102ad0 2024-05-22T23:56:53.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.338+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f627c1a23c0 con 0x7f627c102ad0 2024-05-22T23:56:53.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.340+0000 7f6273400640 1 -- 172.21.15.103:0/4173575160 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6264010430 con 0x7f627c102ad0 2024-05-22T23:56:53.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.341+0000 7f6273400640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62600a2c10 0x7f62600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:53.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.341+0000 7f6273400640 1 -- 172.21.15.103:0/4173575160 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f62640cee10 con 0x7f627c102ad0 2024-05-22T23:56:53.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.341+0000 7f6279e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62600a2c10 0x7f62600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:53.346 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.342+0000 7f6279e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62600a2c10 0x7f62600a50d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f627c19b270 tx=0x7f626c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:53.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.345+0000 7f6273400640 1 -- 172.21.15.103:0/4173575160 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f62640d2030 con 0x7f627c102ad0 2024-05-22T23:56:53.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:53 smithi174 ceph-mon[36179]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:53.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:53 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:53.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:53 smithi105 ceph-mon[36426]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:53.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:53 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:53.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:53 smithi103 ceph-mon[31370]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:53.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:53 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:53.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.589+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f627c1a2590 con 0x7f627c102ad0 2024-05-22T23:56:53.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.590+0000 7f6273400640 1 -- 172.21.15.103:0/4173575160 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f626409bb20 con 0x7f627c102ad0 2024-05-22T23:56:53.594 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:56:53.594 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-22T23:56:53.595 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 3 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62600a2c10 msgr2=0x7f62600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62600a2c10 0x7f62600a50d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f627c19b270 tx=0x7f626c005f90 comp rx=0 tx=0).stop 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c102ad0 msgr2=0x7f627c19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c102ad0 0x7f627c19a2e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f626400e790 tx=0x7f626400ec50 comp rx=0 tx=0).stop 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 shutdown_connections 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62600a2c10 0x7f62600a50d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c103490 0x7f627c1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f627c102ad0 0x7f627c19a2e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.592+0000 7f6280e00640 1 --2- 172.21.15.103:0/4173575160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c1018e0 0x7f627c199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:53.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.593+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 >> 172.21.15.103:0/4173575160 conn(0x7f627c0fce50 msgr2=0x7f627c073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:53.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.593+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 shutdown_connections 2024-05-22T23:56:53.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:53.593+0000 7f6280e00640 1 -- 172.21.15.103:0/4173575160 wait complete. 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.105:6832/1828383607,v1:172.21.15.105:6833/1828383607] up:boot 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: fsmap 2 up:standby 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:54.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:54 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/4173575160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:54.350 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:54.350 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.105:6832/1828383607,v1:172.21.15.105:6833/1828383607] up:boot 2024-05-22T23:56:54.350 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: fsmap 2 up:standby 2024-05-22T23:56:54.350 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-22T23:56:54.351 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:54.351 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:54.351 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:54.351 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:54.351 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:54 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/4173575160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:54.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:54.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.105:6832/1828383607,v1:172.21.15.105:6833/1828383607] up:boot 2024-05-22T23:56:54.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: fsmap 2 up:standby 2024-05-22T23:56:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-22T23:56:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:54.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:54 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/4173575160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:55.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:55 smithi103 ceph-mon[31370]: Deploying daemon mds.j on smithi103 2024-05-22T23:56:55.355 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:55 smithi103 ceph-mon[31370]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:55.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:55 smithi174 ceph-mon[36179]: Deploying daemon mds.j on smithi103 2024-05-22T23:56:55.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:55 smithi174 ceph-mon[36179]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:55.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:55 smithi105 ceph-mon[36426]: Deploying daemon mds.j on smithi103 2024-05-22T23:56:55.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:55 smithi105 ceph-mon[36426]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:55.616 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:56:55.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.901+0000 7fc960c00640 1 -- 172.21.15.103:0/395684573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c104cc0 msgr2=0x7fc95c105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:55.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.901+0000 7fc960c00640 1 --2- 172.21.15.103:0/395684573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c104cc0 0x7fc95c105140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc948009860 tx=0x7fc94802a340 comp rx=0 tx=0).stop 2024-05-22T23:56:55.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.902+0000 7fc960c00640 1 -- 172.21.15.103:0/395684573 shutdown_connections 2024-05-22T23:56:55.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.902+0000 7fc960c00640 1 --2- 172.21.15.103:0/395684573 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc95c105680 0x7fc95c10bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.902+0000 7fc960c00640 1 --2- 172.21.15.103:0/395684573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c104cc0 0x7fc95c105140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.902+0000 7fc960c00640 1 --2- 172.21.15.103:0/395684573 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc95c103ad0 0x7fc95c103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.902+0000 7fc960c00640 1 -- 172.21.15.103:0/395684573 >> 172.21.15.103:0/395684573 conn(0x7fc95c0ff060 msgr2=0x7fc95c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.902+0000 7fc960c00640 1 -- 172.21.15.103:0/395684573 shutdown_connections 2024-05-22T23:56:55.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.902+0000 7fc960c00640 1 -- 172.21.15.103:0/395684573 wait complete. 2024-05-22T23:56:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.903+0000 7fc960c00640 1 Processor -- start 2024-05-22T23:56:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.903+0000 7fc960c00640 1 -- start start 2024-05-22T23:56:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc960c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c103ad0 0x7fc95c071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc960c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc95c104cc0 0x7fc95c071a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc960c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc95c105680 0x7fc95c072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc960c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c06bb90 con 0x7fc95c105680 2024-05-22T23:56:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc960c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c06ba10 con 0x7fc95c104cc0 2024-05-22T23:56:55.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc960c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc95c06bd10 con 0x7fc95c103ad0 2024-05-22T23:56:55.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc959400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc95c104cc0 0x7fc95c071a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:55.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc959e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c103ad0 0x7fc95c071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:55.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc959400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc95c104cc0 0x7fc95c071a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:56190/0 (socket says 172.21.15.103:56190) 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc959400640 1 -- 172.21.15.103:0/3630953832 learned_addr learned my addr 172.21.15.103:0/3630953832 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.904+0000 7fc95a800640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc95c105680 0x7fc95c072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc959e00640 1 -- 172.21.15.103:0/3630953832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc95c104cc0 msgr2=0x7fc95c071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc959e00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc95c104cc0 0x7fc95c071a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc959e00640 1 -- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc95c105680 msgr2=0x7fc95c072a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc959e00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc95c105680 0x7fc95c072a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc959e00640 1 -- 172.21.15.103:0/3630953832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc95c0730f0 con 0x7fc95c103ad0 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc959400640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc95c104cc0 0x7fc95c071a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:56:55.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc959e00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c103ad0 0x7fc95c071500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc95c104b20 tx=0x7fc94c00dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:55.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc952a00640 1 -- 172.21.15.103:0/3630953832 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c017070 con 0x7fc95c103ad0 2024-05-22T23:56:55.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc952a00640 1 -- 172.21.15.103:0/3630953832 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc94c010460 con 0x7fc95c103ad0 2024-05-22T23:56:55.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc952a00640 1 -- 172.21.15.103:0/3630953832 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc94c00f620 con 0x7fc95c103ad0 2024-05-22T23:56:55.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.905+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc95c0733e0 con 0x7fc95c103ad0 2024-05-22T23:56:55.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.906+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc95c1aabd0 con 0x7fc95c103ad0 2024-05-22T23:56:55.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.907+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc95c068ae0 con 0x7fc95c103ad0 2024-05-22T23:56:55.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.910+0000 7fc952a00640 1 -- 172.21.15.103:0/3630953832 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc94c01a070 con 0x7fc95c103ad0 2024-05-22T23:56:55.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.911+0000 7fc952a00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc9400a2c60 0x7fc9400a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:55.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.911+0000 7fc952a00640 1 -- 172.21.15.103:0/3630953832 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc94c0ce550 con 0x7fc95c103ad0 2024-05-22T23:56:55.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.912+0000 7fc959400640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc9400a2c60 0x7fc9400a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:55.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.912+0000 7fc959400640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc9400a2c60 0x7fc9400a5120 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc95c072a10 tx=0x7fc948030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:55.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:55.916+0000 7fc952a00640 1 -- 172.21.15.103:0/3630953832 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc94c09b8b0 con 0x7fc95c103ad0 2024-05-22T23:56:56.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.159+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc95c10ae70 con 0x7fc95c103ad0 2024-05-22T23:56:56.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.160+0000 7fc952a00640 1 -- 172.21.15.103:0/3630953832 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fc94c09b1b0 con 0x7fc95c103ad0 2024-05-22T23:56:56.165 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:56:56.165 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-22T23:56:56.165 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 3 2024-05-22T23:56:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.162+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc9400a2c60 msgr2=0x7fc9400a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.162+0000 7fc960c00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc9400a2c60 0x7fc9400a5120 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc95c072a10 tx=0x7fc948030550 comp rx=0 tx=0).stop 2024-05-22T23:56:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.162+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c103ad0 msgr2=0x7fc95c071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.162+0000 7fc960c00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c103ad0 0x7fc95c071500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc95c104b20 tx=0x7fc94c00dbb0 comp rx=0 tx=0).stop 2024-05-22T23:56:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.163+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 shutdown_connections 2024-05-22T23:56:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.163+0000 7fc960c00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc9400a2c60 0x7fc9400a5120 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.163+0000 7fc960c00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc95c105680 0x7fc95c072a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:56.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.163+0000 7fc960c00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc95c104cc0 0x7fc95c071a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:56.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.163+0000 7fc960c00640 1 --2- 172.21.15.103:0/3630953832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc95c103ad0 0x7fc95c071500 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:56.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.163+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 >> 172.21.15.103:0/3630953832 conn(0x7fc95c0ff060 msgr2=0x7fc95c108040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:56.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.163+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 shutdown_connections 2024-05-22T23:56:56.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:56.163+0000 7fc960c00640 1 -- 172.21.15.103:0/3630953832 wait complete. 2024-05-22T23:56:56.831 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 22 23:56:56 smithi103 podman[72011]: 2024-05-22 23:56:56.549431007 +0000 UTC m=+0.190439893 container create 7bdcb80f9abdb26e8fa50f0f678bd1897f56e12109dc7eaeb30d27382b957a97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762) 2024-05-22T23:56:57.118 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 22 23:56:56 smithi103 podman[72011]: 2024-05-22 23:56:56.845273472 +0000 UTC m=+0.486282378 container init 7bdcb80f9abdb26e8fa50f0f678bd1897f56e12109dc7eaeb30d27382b957a97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-22T23:56:57.119 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 22 23:56:56 smithi103 podman[72011]: 2024-05-22 23:56:56.856192658 +0000 UTC m=+0.497201514 container start 7bdcb80f9abdb26e8fa50f0f678bd1897f56e12109dc7eaeb30d27382b957a97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762) 2024-05-22T23:56:57.119 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 22 23:56:56 smithi103 ceph-mds[72030]: -- 172.21.15.103:0/2902135109 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5632c3cd1a40 con 0x5632c3ca5000 2024-05-22T23:56:57.119 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 22 23:56:56 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j[72026]: starting mds.j at 2024-05-22T23:56:57.119 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 22 23:56:56 smithi103 bash[72011]: 7bdcb80f9abdb26e8fa50f0f678bd1897f56e12109dc7eaeb30d27382b957a97 2024-05-22T23:56:57.119 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 22 23:56:56 smithi103 systemd[1]: Started Ceph mds.j for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:56:57.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:57 smithi174 ceph-mon[36179]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:57.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:57 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3630953832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:57.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:57 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:57.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:57 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:57.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:57 smithi105 ceph-mon[36426]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:57.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:57 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3630953832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:57.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:57 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:57.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:57 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:57.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:57 smithi103 ceph-mon[31370]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:57.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:57 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3630953832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:57.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:57 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:57.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:57 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:58.186 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:56:58.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.401+0000 7f6c9a800640 1 -- 172.21.15.103:0/426218664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94104b60 msgr2=0x7f6c94106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:58.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.401+0000 7f6c9a800640 1 --2- 172.21.15.103:0/426218664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94104b60 0x7f6c94106f50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009900 tx=0x7f6c8002a410 comp rx=0 tx=0).stop 2024-05-22T23:56:58.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.402+0000 7f6c9a800640 1 -- 172.21.15.103:0/426218664 shutdown_connections 2024-05-22T23:56:58.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.402+0000 7f6c9a800640 1 --2- 172.21.15.103:0/426218664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94107490 0x7f6c94109880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.402+0000 7f6c9a800640 1 --2- 172.21.15.103:0/426218664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94104b60 0x7f6c94106f50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.402+0000 7f6c9a800640 1 --2- 172.21.15.103:0/426218664 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94102230 0x7f6c94104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.402+0000 7f6c9a800640 1 -- 172.21.15.103:0/426218664 >> 172.21.15.103:0/426218664 conn(0x7f6c940fbb40 msgr2=0x7f6c940fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:58.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.402+0000 7f6c9a800640 1 -- 172.21.15.103:0/426218664 shutdown_connections 2024-05-22T23:56:58.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.402+0000 7f6c9a800640 1 -- 172.21.15.103:0/426218664 wait complete. 2024-05-22T23:56:58.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c9a800640 1 Processor -- start 2024-05-22T23:56:58.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c9a800640 1 -- start start 2024-05-22T23:56:58.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c9a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94102230 0x7f6c94101ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:58.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c9a800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94104b60 0x7f6c941001e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c9a800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94107490 0x7f6c94100720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c9a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9410d3c0 con 0x7f6c94102230 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c9a800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9410d240 con 0x7f6c94107490 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c9a800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c9410d540 con 0x7f6c94104b60 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c99400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94102230 0x7f6c94101ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c99400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94102230 0x7f6c94101ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43606/0 (socket says 172.21.15.103:43606) 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c99400640 1 -- 172.21.15.103:0/293794476 learned_addr learned my addr 172.21.15.103:0/293794476 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c98a00640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94104b60 0x7f6c941001e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.403+0000 7f6c99e00640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94107490 0x7f6c94100720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:58.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c99400640 1 -- 172.21.15.103:0/293794476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94104b60 msgr2=0x7f6c941001e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c99400640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94104b60 0x7f6c941001e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c99400640 1 -- 172.21.15.103:0/293794476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94107490 msgr2=0x7f6c94100720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c99400640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94107490 0x7f6c94100720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c99400640 1 -- 172.21.15.103:0/293794476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c94100fb0 con 0x7f6c94102230 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c99400640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94102230 0x7f6c94101ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800e9f0 tx=0x7f6c8800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c92000640 1 -- 172.21.15.103:0/293794476 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800cd80 con 0x7f6c94102230 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c92000640 1 -- 172.21.15.103:0/293794476 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c88005500 con 0x7f6c94102230 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c92000640 1 -- 172.21.15.103:0/293794476 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c88010690 con 0x7f6c94102230 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.404+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c941a1b30 con 0x7f6c94102230 2024-05-22T23:56:58.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.405+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c941a2050 con 0x7f6c94102230 2024-05-22T23:56:58.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.406+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c940685b0 con 0x7f6c94102230 2024-05-22T23:56:58.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.408+0000 7f6c92000640 1 -- 172.21.15.103:0/293794476 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f6c880046c0 con 0x7f6c94102230 2024-05-22T23:56:58.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.409+0000 7f6c92000640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6c7c0a2c10 0x7f6c7c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:56:58.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.409+0000 7f6c92000640 1 -- 172.21.15.103:0/293794476 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f6c880cefe0 con 0x7f6c94102230 2024-05-22T23:56:58.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.409+0000 7f6c98a00640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6c7c0a2c10 0x7f6c7c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:56:58.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.410+0000 7f6c98a00640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6c7c0a2c10 0x7f6c7c0a50d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6c8002aa00 tx=0x7f6c800315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:56:58.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.413+0000 7f6c92000640 1 -- 172.21.15.103:0/293794476 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6c8809c360 con 0x7f6c94102230 2024-05-22T23:56:58.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:58 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.103:6834/1622172738,v1:172.21.15.103:6835/1622172738] up:boot 2024-05-22T23:56:58.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:58 smithi105 ceph-mon[36426]: fsmap 3 up:standby 2024-05-22T23:56:58.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:58 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-22T23:56:58.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:58 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:58.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:58 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:58.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:58 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:58.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:58 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:58.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:58 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.103:6834/1622172738,v1:172.21.15.103:6835/1622172738] up:boot 2024-05-22T23:56:58.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:58 smithi103 ceph-mon[31370]: fsmap 3 up:standby 2024-05-22T23:56:58.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:58 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-22T23:56:58.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:58 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:58.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:58 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:58.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:58 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:58.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:58 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:58.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:58 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.103:6834/1622172738,v1:172.21.15.103:6835/1622172738] up:boot 2024-05-22T23:56:58.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:58 smithi174 ceph-mon[36179]: fsmap 3 up:standby 2024-05-22T23:56:58.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:58 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-22T23:56:58.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:58 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:56:58.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:58 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:56:58.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:58 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:56:58.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:58 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:56:58.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.662+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c94062470 con 0x7f6c94102230 2024-05-22T23:56:58.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.663+0000 7f6c92000640 1 -- 172.21.15.103:0/293794476 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7f6c8809bc60 con 0x7f6c94102230 2024-05-22T23:56:58.667 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:56:58.667 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-05-22T23:56:58.667 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 4 2024-05-22T23:56:58.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6c7c0a2c10 msgr2=0x7f6c7c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:58.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6c7c0a2c10 0x7f6c7c0a50d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6c8002aa00 tx=0x7f6c800315b0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94102230 msgr2=0x7f6c94101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:56:58.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94102230 0x7f6c94101ba0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800e9f0 tx=0x7f6c8800edf0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 shutdown_connections 2024-05-22T23:56:58.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6c7c0a2c10 0x7f6c7c0a50d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c94107490 0x7f6c94100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c94104b60 0x7f6c941001e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 --2- 172.21.15.103:0/293794476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c94102230 0x7f6c94101ba0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:56:58.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 >> 172.21.15.103:0/293794476 conn(0x7f6c940fbb40 msgr2=0x7f6c94102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:56:58.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 shutdown_connections 2024-05-22T23:56:58.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:56:58.666+0000 7f6c9a800640 1 -- 172.21.15.103:0/293794476 wait complete. 2024-05-22T23:56:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:59 smithi105 ceph-mon[36426]: Deploying daemon mds.d on smithi103 2024-05-22T23:56:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:59 smithi105 ceph-mon[36426]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:56:59 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/293794476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:59.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:59 smithi103 ceph-mon[31370]: Deploying daemon mds.d on smithi103 2024-05-22T23:56:59.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:59 smithi103 ceph-mon[31370]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:59.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:56:59 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/293794476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:56:59.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:59 smithi174 ceph-mon[36179]: Deploying daemon mds.d on smithi103 2024-05-22T23:56:59.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:59 smithi174 ceph-mon[36179]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:56:59.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:56:59 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/293794476' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:00.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:00.748 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:00 smithi103 systemd[1]: Starting Ceph mds.d for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:00.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 -- 172.21.15.103:0/2265701711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664107e80 msgr2=0x7f56641082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:00.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 --2- 172.21.15.103:0/2265701711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664107e80 0x7f56641082a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5654009860 tx=0x7f565402a340 comp rx=0 tx=0).stop 2024-05-22T23:57:00.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 -- 172.21.15.103:0/2265701711 shutdown_connections 2024-05-22T23:57:00.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 --2- 172.21.15.103:0/2265701711 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5664109a20 0x7f5664110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:00.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 --2- 172.21.15.103:0/2265701711 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5664109060 0x7f56641094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:00.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 --2- 172.21.15.103:0/2265701711 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664107e80 0x7f56641082a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:00.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 -- 172.21.15.103:0/2265701711 >> 172.21.15.103:0/2265701711 conn(0x7f5664076e90 msgr2=0x7f56640792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:00.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 -- 172.21.15.103:0/2265701711 shutdown_connections 2024-05-22T23:57:00.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.872+0000 7f566a000640 1 -- 172.21.15.103:0/2265701711 wait complete. 2024-05-22T23:57:00.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.873+0000 7f566a000640 1 Processor -- start 2024-05-22T23:57:00.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.873+0000 7f566a000640 1 -- start start 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f566a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5664107e80 0x7f566419e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f566a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664109060 0x7f566419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f566a000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5664109a20 0x7f56641a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f566a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5664113b70 con 0x7f5664107e80 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f566a000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f56641139f0 con 0x7f5664109060 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f566a000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5664113cf0 con 0x7f5664109a20 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5663400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5664107e80 0x7f566419e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5662a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664109060 0x7f566419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5662a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664109060 0x7f566419e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:56254/0 (socket says 172.21.15.103:56254) 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5662a00640 1 -- 172.21.15.103:0/1631053672 learned_addr learned my addr 172.21.15.103:0/1631053672 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:00.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5663e00640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5664109a20 0x7f56641a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5662a00640 1 -- 172.21.15.103:0/1631053672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5664109a20 msgr2=0x7f56641a57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5662a00640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5664109a20 0x7f56641a57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5662a00640 1 -- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5664107e80 msgr2=0x7f566419e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5662a00640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5664107e80 0x7f566419e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.874+0000 7f5662a00640 1 -- 172.21.15.103:0/1631053672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56641a5e30 con 0x7f5664109060 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.875+0000 7f5662a00640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664109060 0x7f566419e750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f565000e9f0 tx=0x7f565000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.875+0000 7f565be00640 1 -- 172.21.15.103:0/1631053672 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565000cd80 con 0x7f5664109060 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.875+0000 7f565be00640 1 -- 172.21.15.103:0/1631053672 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5650005500 con 0x7f5664109060 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.875+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56641a6120 con 0x7f5664109060 2024-05-22T23:57:00.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.875+0000 7f565be00640 1 -- 172.21.15.103:0/1631053672 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5650010690 con 0x7f5664109060 2024-05-22T23:57:00.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.875+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56641a6620 con 0x7f5664109060 2024-05-22T23:57:00.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.876+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f566410f220 con 0x7f5664109060 2024-05-22T23:57:00.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.879+0000 7f565be00640 1 -- 172.21.15.103:0/1631053672 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f56500046c0 con 0x7f5664109060 2024-05-22T23:57:00.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.880+0000 7f565be00640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f56480a2b90 0x7f56480a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:00.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.880+0000 7f565be00640 1 -- 172.21.15.103:0/1631053672 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f56500cefe0 con 0x7f5664109060 2024-05-22T23:57:00.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.880+0000 7f5663400640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f56480a2b90 0x7f56480a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:00.885 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.881+0000 7f5663400640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f56480a2b90 0x7f56480a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5654009830 tx=0x7f5654030570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:00.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:00.885+0000 7f565be00640 1 -- 172.21.15.103:0/1631053672 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f565009c380 con 0x7f5664109060 2024-05-22T23:57:01.023 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:00 smithi103 podman[72355]: 2024-05-22 23:57:00.674733629 +0000 UTC m=+0.037379303 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:01.023 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:00 smithi103 podman[72355]: 2024-05-22 23:57:00.840379215 +0000 UTC m=+0.203024930 container create ae06e094c8ba4b82c8a7cf4390af1b5af491b30f739b641214fdd47f46f36ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-22T23:57:01.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.146+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f566402d200 con 0x7f5664109060 2024-05-22T23:57:01.151 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.147+0000 7f565be00640 1 -- 172.21.15.103:0/1631053672 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3052 (secure 0 0 0) 0x7f565009bc80 con 0x7f5664109060 2024-05-22T23:57:01.151 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:01.152 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-05-22T23:57:01.152 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 4 2024-05-22T23:57:01.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f56480a2b90 msgr2=0x7f56480a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:01.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f56480a2b90 0x7f56480a5050 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5654009830 tx=0x7f5654030570 comp rx=0 tx=0).stop 2024-05-22T23:57:01.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664109060 msgr2=0x7f566419e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:01.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664109060 0x7f566419e750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f565000e9f0 tx=0x7f565000edf0 comp rx=0 tx=0).stop 2024-05-22T23:57:01.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 shutdown_connections 2024-05-22T23:57:01.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f56480a2b90 0x7f56480a5050 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:01.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5664109a20 0x7f56641a57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:01.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5664109060 0x7f566419e750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:01.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 --2- 172.21.15.103:0/1631053672 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5664107e80 0x7f566419e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:01.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 >> 172.21.15.103:0/1631053672 conn(0x7f5664076e90 msgr2=0x7f566410c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:01.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 shutdown_connections 2024-05-22T23:57:01.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:01.150+0000 7f566a000640 1 -- 172.21.15.103:0/1631053672 wait complete. 2024-05-22T23:57:01.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:01 smithi105 ceph-mon[36426]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:01.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:01 smithi103 ceph-mon[31370]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:01.581 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:01 smithi103 podman[72355]: 2024-05-22 23:57:01.127646846 +0000 UTC m=+0.490292527 container init ae06e094c8ba4b82c8a7cf4390af1b5af491b30f739b641214fdd47f46f36ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-22T23:57:01.581 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:01 smithi103 podman[72355]: 2024-05-22 23:57:01.139766962 +0000 UTC m=+0.502412633 container start ae06e094c8ba4b82c8a7cf4390af1b5af491b30f739b641214fdd47f46f36ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-22T23:57:01.581 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:01 smithi103 ceph-mds[72413]: -- 172.21.15.103:0/1911234471 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5562110744e0 con 0x55621104ac00 2024-05-22T23:57:01.581 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:01 smithi103 ceph-mds[72413]: -- 172.21.15.103:0/1911234471 <== mon.0 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556211077a40 con 0x55621104ac00 2024-05-22T23:57:01.581 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d[72409]: starting mds.d at 2024-05-22T23:57:01.581 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:01 smithi103 bash[72355]: ae06e094c8ba4b82c8a7cf4390af1b5af491b30f739b641214fdd47f46f36ae5 2024-05-22T23:57:01.581 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 22 23:57:01 smithi103 systemd[1]: Started Ceph mds.d for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:01.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:01 smithi174 ceph-mon[36179]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:02.405 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:02 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1631053672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:02.405 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:02 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.405 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:02 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.406 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:02 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.406 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:02 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:02.406 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:02 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:02.406 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:02 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:02.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:02 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1631053672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:02 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:02 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:02 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:02 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:02 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:02.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:02 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:02 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1631053672' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:02 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:02 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:02 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:02 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:02 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:02.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:02 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:03.174 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.380+0000 7f3b89600640 1 -- 172.21.15.103:0/111788497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b841098a0 msgr2=0x7f3b84110130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.380+0000 7f3b89600640 1 --2- 172.21.15.103:0/111788497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b841098a0 0x7f3b84110130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c009ca0 tx=0x7f3b7c02a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.381+0000 7f3b89600640 1 -- 172.21.15.103:0/111788497 shutdown_connections 2024-05-22T23:57:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.381+0000 7f3b89600640 1 --2- 172.21.15.103:0/111788497 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b841098a0 0x7f3b84110130 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.381+0000 7f3b89600640 1 --2- 172.21.15.103:0/111788497 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b84108ee0 0x7f3b84109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.381+0000 7f3b89600640 1 --2- 172.21.15.103:0/111788497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b84107cf0 0x7f3b84108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.381+0000 7f3b89600640 1 -- 172.21.15.103:0/111788497 >> 172.21.15.103:0/111788497 conn(0x7f3b84076cb0 msgr2=0x7f3b840790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:03.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.381+0000 7f3b89600640 1 -- 172.21.15.103:0/111788497 shutdown_connections 2024-05-22T23:57:03.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.381+0000 7f3b89600640 1 -- 172.21.15.103:0/111788497 wait complete. 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.382+0000 7f3b89600640 1 Processor -- start 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.382+0000 7f3b89600640 1 -- start start 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.382+0000 7f3b89600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b84107cf0 0x7f3b8419e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.382+0000 7f3b89600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b84108ee0 0x7f3b8419e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.382+0000 7f3b89600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b841098a0 0x7f3b841a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.382+0000 7f3b89600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3b84113970 con 0x7f3b84108ee0 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.382+0000 7f3b89600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b841137f0 con 0x7f3b84107cf0 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.382+0000 7f3b89600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3b84113af0 con 0x7f3b841098a0 2024-05-22T23:57:03.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b84107cf0 0x7f3b8419e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b7be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b84108ee0 0x7f3b8419e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b84107cf0 0x7f3b8419e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:56286/0 (socket says 172.21.15.103:56286) 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 -- 172.21.15.103:0/1989092626 learned_addr learned my addr 172.21.15.103:0/1989092626 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b88c00640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b841098a0 0x7f3b841a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 -- 172.21.15.103:0/1989092626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b841098a0 msgr2=0x7f3b841a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b841098a0 0x7f3b841a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 -- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b84108ee0 msgr2=0x7f3b8419e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b84108ee0 0x7f3b8419e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 -- 172.21.15.103:0/1989092626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b841a5be0 con 0x7f3b84107cf0 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.383+0000 7f3b83e00640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b84107cf0 0x7f3b8419e010 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400e9c0 tx=0x7f3b7400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.384+0000 7f3b81600640 1 -- 172.21.15.103:0/1989092626 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7400cde0 con 0x7f3b84107cf0 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.384+0000 7f3b81600640 1 -- 172.21.15.103:0/1989092626 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b74010910 con 0x7f3b84107cf0 2024-05-22T23:57:03.389 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.384+0000 7f3b81600640 1 -- 172.21.15.103:0/1989092626 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74018a00 con 0x7f3b84107cf0 2024-05-22T23:57:03.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.384+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b841a5ed0 con 0x7f3b84107cf0 2024-05-22T23:57:03.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.384+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b841a6390 con 0x7f3b84107cf0 2024-05-22T23:57:03.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.385+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b8410f010 con 0x7f3b84107cf0 2024-05-22T23:57:03.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.387+0000 7f3b81600640 1 -- 172.21.15.103:0/1989092626 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3b7401f020 con 0x7f3b84107cf0 2024-05-22T23:57:03.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.389+0000 7f3b81600640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b680a2c10 0x7f3b680a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:03.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.389+0000 7f3b81600640 1 -- 172.21.15.103:0/1989092626 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f3b740cedb0 con 0x7f3b84107cf0 2024-05-22T23:57:03.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.389+0000 7f3b7be00640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b680a2c10 0x7f3b680a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:03.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.390+0000 7f3b7be00640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b680a2c10 0x7f3b680a50d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3b8419f4d0 tx=0x7f3b6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:03.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.393+0000 7f3b81600640 1 -- 172.21.15.103:0/1989092626 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3b740d3070 con 0x7f3b84107cf0 2024-05-22T23:57:03.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:03 smithi105 ceph-mon[36426]: Deploying daemon mds.e on smithi105 2024-05-22T23:57:03.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:03 smithi105 ceph-mon[36426]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:03.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:03 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.103:6836/1586986651,v1:172.21.15.103:6837/1586986651] up:boot 2024-05-22T23:57:03.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:03 smithi105 ceph-mon[36426]: fsmap 4 up:standby 2024-05-22T23:57:03.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:03 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-22T23:57:03.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:03 smithi103 ceph-mon[31370]: Deploying daemon mds.e on smithi105 2024-05-22T23:57:03.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:03 smithi103 ceph-mon[31370]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:03.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:03 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.103:6836/1586986651,v1:172.21.15.103:6837/1586986651] up:boot 2024-05-22T23:57:03.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:03 smithi103 ceph-mon[31370]: fsmap 4 up:standby 2024-05-22T23:57:03.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:03 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-22T23:57:03.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:03 smithi174 ceph-mon[36179]: Deploying daemon mds.e on smithi105 2024-05-22T23:57:03.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:03 smithi174 ceph-mon[36179]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:03.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:03 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.103:6836/1586986651,v1:172.21.15.103:6837/1586986651] up:boot 2024-05-22T23:57:03.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:03 smithi174 ceph-mon[36179]: fsmap 4 up:standby 2024-05-22T23:57:03.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:03 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-22T23:57:03.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.632+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b8402d200 con 0x7f3b84107cf0 2024-05-22T23:57:03.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.633+0000 7f3b81600640 1 -- 172.21.15.103:0/1989092626 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3880 (secure 0 0 0) 0x7f3b7409c110 con 0x7f3b84107cf0 2024-05-22T23:57:03.638 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:03.638 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-05-22T23:57:03.638 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 5 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.637+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b680a2c10 msgr2=0x7f3b680a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.637+0000 7f3b89600640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b680a2c10 0x7f3b680a50d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3b8419f4d0 tx=0x7f3b6c005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.637+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b84107cf0 msgr2=0x7f3b8419e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.637+0000 7f3b89600640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b84107cf0 0x7f3b8419e010 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400e9c0 tx=0x7f3b7400ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.638+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 shutdown_connections 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.638+0000 7f3b89600640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b680a2c10 0x7f3b680a50d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.638+0000 7f3b89600640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b841098a0 0x7f3b841a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.638+0000 7f3b89600640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b84108ee0 0x7f3b8419e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.638+0000 7f3b89600640 1 --2- 172.21.15.103:0/1989092626 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b84107cf0 0x7f3b8419e010 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:03.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.638+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 >> 172.21.15.103:0/1989092626 conn(0x7f3b84076cb0 msgr2=0x7f3b8410c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:03.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.638+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 shutdown_connections 2024-05-22T23:57:03.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:03.638+0000 7f3b89600640 1 -- 172.21.15.103:0/1989092626 wait complete. 2024-05-22T23:57:03.809 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:03 smithi105 systemd[1]: Starting Ceph mds.e for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:04.150 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:03 smithi105 podman[66344]: 2024-05-22 23:57:03.792608581 +0000 UTC m=+0.017428691 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:04.150 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:03 smithi105 podman[66344]: 2024-05-22 23:57:03.909245188 +0000 UTC m=+0.134065283 container create d880e172c126df554eb131ef51eb15a1214a1465a17d6cfaa42f7c819075fdf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-22T23:57:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:04 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1989092626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:04.559 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:04 smithi105 podman[66344]: 2024-05-22 23:57:04.153528186 +0000 UTC m=+0.378348288 container init d880e172c126df554eb131ef51eb15a1214a1465a17d6cfaa42f7c819075fdf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-05-22T23:57:04.559 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:04 smithi105 podman[66344]: 2024-05-22 23:57:04.15612006 +0000 UTC m=+0.380940161 container start d880e172c126df554eb131ef51eb15a1214a1465a17d6cfaa42f7c819075fdf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-22T23:57:04.559 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:04 smithi105 ceph-mds[66364]: -- 172.21.15.105:0/1140693387 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559859879a40 con 0x55985984d000 2024-05-22T23:57:04.559 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:04 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e[66360]: starting mds.e at 2024-05-22T23:57:04.560 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:04 smithi105 bash[66344]: d880e172c126df554eb131ef51eb15a1214a1465a17d6cfaa42f7c819075fdf2 2024-05-22T23:57:04.560 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 22 23:57:04 smithi105 systemd[1]: Started Ceph mds.e for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:04.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:04 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1989092626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:04 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1989092626' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:05.291 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:05 smithi174 ceph-mon[36179]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:05.291 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.291 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.292 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.292 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:05.292 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:05.292 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:05 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:05.292 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:05 smithi174 ceph-mon[36179]: Deploying daemon mds.f on smithi174 2024-05-22T23:57:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:05 smithi105 ceph-mon[36426]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:05 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:05 smithi105 ceph-mon[36426]: Deploying daemon mds.f on smithi174 2024-05-22T23:57:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:05 smithi103 ceph-mon[31370]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:05 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:05 smithi103 ceph-mon[31370]: Deploying daemon mds.f on smithi174 2024-05-22T23:57:05.661 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:05.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 -- 172.21.15.103:0/1754365190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb84107cf0 msgr2=0x7fdb84108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:05.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 --2- 172.21.15.103:0/1754365190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb84107cf0 0x7fdb84108110 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c009860 tx=0x7fdb6c02a340 comp rx=0 tx=0).stop 2024-05-22T23:57:05.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 -- 172.21.15.103:0/1754365190 shutdown_connections 2024-05-22T23:57:05.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 --2- 172.21.15.103:0/1754365190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb841098a0 0x7fdb84110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:05.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 --2- 172.21.15.103:0/1754365190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb84108ee0 0x7fdb84109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:05.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 --2- 172.21.15.103:0/1754365190 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb84107cf0 0x7fdb84108110 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:05.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 -- 172.21.15.103:0/1754365190 >> 172.21.15.103:0/1754365190 conn(0x7fdb84076cb0 msgr2=0x7fdb840790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:05.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 -- 172.21.15.103:0/1754365190 shutdown_connections 2024-05-22T23:57:05.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.868+0000 7fdb8a000640 1 -- 172.21.15.103:0/1754365190 wait complete. 2024-05-22T23:57:05.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb8a000640 1 Processor -- start 2024-05-22T23:57:05.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb8a000640 1 -- start start 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb8a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb84107cf0 0x7fdb8419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb8a000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb84108ee0 0x7fdb8419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb8a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb841098a0 0x7fdb841a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb8a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84113970 con 0x7fdb84107cf0 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb8a000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdb841137f0 con 0x7fdb841098a0 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb8a000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdb84113af0 con 0x7fdb84108ee0 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb88c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb84107cf0 0x7fdb8419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.869+0000 7fdb88c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb84107cf0 0x7fdb8419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45022/0 (socket says 172.21.15.103:45022) 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb88c00640 1 -- 172.21.15.103:0/493135354 learned_addr learned my addr 172.21.15.103:0/493135354 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb89600640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb841098a0 0x7fdb841a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:05.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb7be00640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb84108ee0 0x7fdb8419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:05.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb88c00640 1 -- 172.21.15.103:0/493135354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb84108ee0 msgr2=0x7fdb8419e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:05.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb88c00640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb84108ee0 0x7fdb8419e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:05.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb88c00640 1 -- 172.21.15.103:0/493135354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb841098a0 msgr2=0x7fdb841a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:05.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb88c00640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb841098a0 0x7fdb841a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:05.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb88c00640 1 -- 172.21.15.103:0/493135354 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb841a5c40 con 0x7fdb84107cf0 2024-05-22T23:57:05.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb89600640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb841098a0 0x7fdb841a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:05.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb88c00640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb84107cf0 0x7fdb8419e020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0305b0 tx=0x7fdb6c033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:05.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb82000640 1 -- 172.21.15.103:0/493135354 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c03f020 con 0x7fdb84107cf0 2024-05-22T23:57:05.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb82000640 1 -- 172.21.15.103:0/493135354 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb6c030850 con 0x7fdb84107cf0 2024-05-22T23:57:05.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb841a5ed0 con 0x7fdb84107cf0 2024-05-22T23:57:05.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.870+0000 7fdb82000640 1 -- 172.21.15.103:0/493135354 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c037040 con 0x7fdb84107cf0 2024-05-22T23:57:05.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.871+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb841a62e0 con 0x7fdb84107cf0 2024-05-22T23:57:05.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.872+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb8410f090 con 0x7fdb84107cf0 2024-05-22T23:57:05.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.874+0000 7fdb82000640 1 -- 172.21.15.103:0/493135354 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fdb6c042020 con 0x7fdb84107cf0 2024-05-22T23:57:05.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.875+0000 7fdb82000640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdb640a2b90 0x7fdb640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:05.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.875+0000 7fdb82000640 1 -- 172.21.15.103:0/493135354 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fdb6c0eeff0 con 0x7fdb84107cf0 2024-05-22T23:57:05.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.875+0000 7fdb7be00640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdb640a2b90 0x7fdb640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:05.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.876+0000 7fdb7be00640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdb640a2b90 0x7fdb640a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdb8419f530 tx=0x7fdb74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:05.883 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:05.879+0000 7fdb82000640 1 -- 172.21.15.103:0/493135354 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdb6c049020 con 0x7fdb84107cf0 2024-05-22T23:57:06.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.118+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb840625b0 con 0x7fdb84107cf0 2024-05-22T23:57:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.120+0000 7fdb82000640 1 -- 172.21.15.103:0/493135354 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4708 (secure 0 0 0) 0x7fdb6c0bc370 con 0x7fdb84107cf0 2024-05-22T23:57:06.124 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:06.124 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-22T23:57:06.124 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 6 2024-05-22T23:57:06.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdb640a2b90 msgr2=0x7fdb640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:06.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdb640a2b90 0x7fdb640a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdb8419f530 tx=0x7fdb74009260 comp rx=0 tx=0).stop 2024-05-22T23:57:06.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb84107cf0 msgr2=0x7fdb8419e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:06.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb84107cf0 0x7fdb8419e020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c0305b0 tx=0x7fdb6c033d90 comp rx=0 tx=0).stop 2024-05-22T23:57:06.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 shutdown_connections 2024-05-22T23:57:06.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdb640a2b90 0x7fdb640a5050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:06.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb841098a0 0x7fdb841a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:06.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb84108ee0 0x7fdb8419e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:06.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 --2- 172.21.15.103:0/493135354 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb84107cf0 0x7fdb8419e020 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:06.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 >> 172.21.15.103:0/493135354 conn(0x7fdb84076cb0 msgr2=0x7fdb8410c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:06.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 shutdown_connections 2024-05-22T23:57:06.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:06.123+0000 7fdb8a000640 1 -- 172.21.15.103:0/493135354 wait complete. 2024-05-22T23:57:06.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:06 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.105:6834/1500378287,v1:172.21.15.105:6835/1500378287] up:boot 2024-05-22T23:57:06.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:06 smithi174 ceph-mon[36179]: fsmap 5 up:standby 2024-05-22T23:57:06.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:06 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-22T23:57:06.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:06 smithi174 ceph-mon[36179]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:06.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:06 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/493135354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:06 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.105:6834/1500378287,v1:172.21.15.105:6835/1500378287] up:boot 2024-05-22T23:57:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:06 smithi105 ceph-mon[36426]: fsmap 5 up:standby 2024-05-22T23:57:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:06 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-22T23:57:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:06 smithi105 ceph-mon[36426]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:06 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/493135354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:06.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:06 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.105:6834/1500378287,v1:172.21.15.105:6835/1500378287] up:boot 2024-05-22T23:57:06.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:06 smithi103 ceph-mon[31370]: fsmap 5 up:standby 2024-05-22T23:57:06.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:06 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-22T23:57:06.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:06 smithi103 ceph-mon[31370]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:06.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:06 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/493135354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:06.868 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:06 smithi174 systemd[1]: Starting Ceph mds.f for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:07.174 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:06 smithi174 podman[66783]: 2024-05-22 23:57:06.775974042 +0000 UTC m=+0.017178984 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:07.174 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:06 smithi174 podman[66783]: 2024-05-22 23:57:06.910021524 +0000 UTC m=+0.151226444 container create 89db9aeff43f88c11ca6de9d8ad0bfd2ea1615eb54a1db25fabcf14ddf73e14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-22T23:57:07.174 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:07 smithi174 podman[66783]: 2024-05-22 23:57:07.171325554 +0000 UTC m=+0.412530475 container init 89db9aeff43f88c11ca6de9d8ad0bfd2ea1615eb54a1db25fabcf14ddf73e14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-22T23:57:07.618 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:07 smithi174 podman[66783]: 2024-05-22 23:57:07.174011587 +0000 UTC m=+0.415216514 container start 89db9aeff43f88c11ca6de9d8ad0bfd2ea1615eb54a1db25fabcf14ddf73e14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-22T23:57:07.618 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:07 smithi174 ceph-mds[66802]: -- 172.21.15.174:0/2015458329 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aca76cda40 con 0x55aca76a0c00 2024-05-22T23:57:07.618 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:07 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f[66798]: starting mds.f at 2024-05-22T23:57:07.618 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:07 smithi174 bash[66783]: 89db9aeff43f88c11ca6de9d8ad0bfd2ea1615eb54a1db25fabcf14ddf73e14c 2024-05-22T23:57:07.618 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 22 23:57:07 smithi174 systemd[1]: Started Ceph mds.f for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:08.149 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:08.303 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6834/3839469517,v1:172.21.15.174:6835/3839469517] up:boot 2024-05-22T23:57:08.303 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: fsmap 6 up:standby 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: Deploying daemon mds.c on smithi174 2024-05-22T23:57:08.304 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:08 smithi174 ceph-mon[36179]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:08.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 -- 172.21.15.103:0/1959754872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b48104ce0 msgr2=0x7f3b48105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:08.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 --2- 172.21.15.103:0/1959754872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b48104ce0 0x7f3b48105160 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b34009900 tx=0x7f3b3402a400 comp rx=0 tx=0).stop 2024-05-22T23:57:08.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 -- 172.21.15.103:0/1959754872 shutdown_connections 2024-05-22T23:57:08.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 --2- 172.21.15.103:0/1959754872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b481056a0 0x7f3b4810bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 --2- 172.21.15.103:0/1959754872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b48104ce0 0x7f3b48105160 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 --2- 172.21.15.103:0/1959754872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b48103af0 0x7f3b48103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 -- 172.21.15.103:0/1959754872 >> 172.21.15.103:0/1959754872 conn(0x7f3b480ff060 msgr2=0x7f3b481014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:08.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 -- 172.21.15.103:0/1959754872 shutdown_connections 2024-05-22T23:57:08.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.358+0000 7f3b4da00640 1 -- 172.21.15.103:0/1959754872 wait complete. 2024-05-22T23:57:08.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.359+0000 7f3b4da00640 1 Processor -- start 2024-05-22T23:57:08.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.359+0000 7f3b4da00640 1 -- start start 2024-05-22T23:57:08.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.359+0000 7f3b4da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b48103af0 0x7f3b48199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:08.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.359+0000 7f3b4da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b48104ce0 0x7f3b4819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:08.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.359+0000 7f3b4da00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b481056a0 0x7f3b481a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:08.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b4da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4810f750 con 0x7f3b48103af0 2024-05-22T23:57:08.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b4da00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4810f5d0 con 0x7f3b48104ce0 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b4da00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4810f8d0 con 0x7f3b481056a0 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b46a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b48103af0 0x7f3b48199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b46a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b48103af0 0x7f3b48199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45030/0 (socket says 172.21.15.103:45030) 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b46a00640 1 -- 172.21.15.103:0/2069907687 learned_addr learned my addr 172.21.15.103:0/2069907687 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b47400640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b481056a0 0x7f3b481a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b46000640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b48104ce0 0x7f3b4819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b47400640 1 -- 172.21.15.103:0/2069907687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b48104ce0 msgr2=0x7f3b4819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b47400640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b48104ce0 0x7f3b4819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b47400640 1 -- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b48103af0 msgr2=0x7f3b48199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b47400640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b48103af0 0x7f3b48199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b47400640 1 -- 172.21.15.103:0/2069907687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b481a1a30 con 0x7f3b481056a0 2024-05-22T23:57:08.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.360+0000 7f3b46a00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b48103af0 0x7f3b48199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:08.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.361+0000 7f3b47400640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b481056a0 0x7f3b481a13c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000ec50 tx=0x7f3b4000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:08.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.361+0000 7f3b3f400640 1 -- 172.21.15.103:0/2069907687 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b40003b70 con 0x7f3b481056a0 2024-05-22T23:57:08.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.361+0000 7f3b3f400640 1 -- 172.21.15.103:0/2069907687 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b40003d10 con 0x7f3b481056a0 2024-05-22T23:57:08.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.361+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b481a1d20 con 0x7f3b481056a0 2024-05-22T23:57:08.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.361+0000 7f3b3f400640 1 -- 172.21.15.103:0/2069907687 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b40010890 con 0x7f3b481056a0 2024-05-22T23:57:08.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.361+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b481a2190 con 0x7f3b481056a0 2024-05-22T23:57:08.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.362+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b4810aee0 con 0x7f3b481056a0 2024-05-22T23:57:08.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.364+0000 7f3b3f400640 1 -- 172.21.15.103:0/2069907687 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3b40003fc0 con 0x7f3b481056a0 2024-05-22T23:57:08.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.365+0000 7f3b3f400640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b2c0a2b90 0x7f3b2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:08.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.366+0000 7f3b3f400640 1 -- 172.21.15.103:0/2069907687 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f3b400cf1f0 con 0x7f3b481056a0 2024-05-22T23:57:08.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.366+0000 7f3b46a00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b2c0a2b90 0x7f3b2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:08.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.366+0000 7f3b46a00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b2c0a2b90 0x7f3b2c0a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3b380095f0 tx=0x7f3b38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:08.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.369+0000 7f3b3f400640 1 -- 172.21.15.103:0/2069907687 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3b4009c550 con 0x7f3b481056a0 2024-05-22T23:57:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6834/3839469517,v1:172.21.15.174:6835/3839469517] up:boot 2024-05-22T23:57:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: fsmap 6 up:standby 2024-05-22T23:57:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-22T23:57:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:08.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:08.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: Deploying daemon mds.c on smithi174 2024-05-22T23:57:08.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:08 smithi105 ceph-mon[36426]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:08.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6834/3839469517,v1:172.21.15.174:6835/3839469517] up:boot 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: fsmap 6 up:standby 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: Deploying daemon mds.c on smithi174 2024-05-22T23:57:08.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:08 smithi103 ceph-mon[31370]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:08.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.611+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b4802d200 con 0x7f3b481056a0 2024-05-22T23:57:08.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.612+0000 7f3b3f400640 1 -- 172.21.15.103:0/2069907687 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5536 (secure 0 0 0) 0x7f3b4009be50 con 0x7f3b481056a0 2024-05-22T23:57:08.616 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:08.616 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-05-22T23:57:08.617 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 7 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.614+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b2c0a2b90 msgr2=0x7f3b2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.614+0000 7f3b4da00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b2c0a2b90 0x7f3b2c0a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3b380095f0 tx=0x7f3b38005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.614+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b481056a0 msgr2=0x7f3b481a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.614+0000 7f3b4da00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b481056a0 0x7f3b481a13c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3b4000ec50 tx=0x7f3b4000c6a0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.615+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 shutdown_connections 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.615+0000 7f3b4da00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3b2c0a2b90 0x7f3b2c0a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.615+0000 7f3b4da00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b481056a0 0x7f3b481a13c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.615+0000 7f3b4da00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b48104ce0 0x7f3b4819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.615+0000 7f3b4da00640 1 --2- 172.21.15.103:0/2069907687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b48103af0 0x7f3b48199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:08.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.615+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 >> 172.21.15.103:0/2069907687 conn(0x7f3b480ff060 msgr2=0x7f3b481080b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:08.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.615+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 shutdown_connections 2024-05-22T23:57:08.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:08.615+0000 7f3b4da00640 1 -- 172.21.15.103:0/2069907687 wait complete. 2024-05-22T23:57:09.473 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:09 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2069907687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:09.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:09 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2069907687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:09.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:09 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2069907687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:09.868 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:09 smithi174 systemd[1]: Starting Ceph mds.c for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:10.156 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:09 smithi174 podman[67089]: 2024-05-22 23:57:09.775650264 +0000 UTC m=+0.017089348 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:10.156 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:09 smithi174 podman[67089]: 2024-05-22 23:57:09.891742173 +0000 UTC m=+0.133181249 container create 376efd3c5777993812fd21a83e819cc253d78012f69b757c2fa4ffe099f9be95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-2e1f762, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-22T23:57:10.156 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:10 smithi174 podman[67089]: 2024-05-22 23:57:10.153243343 +0000 UTC m=+0.394682427 container init 376efd3c5777993812fd21a83e819cc253d78012f69b757c2fa4ffe099f9be95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-22T23:57:10.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-mon[36179]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:10.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:10.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:10.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:10.619 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:10 smithi174 podman[67089]: 2024-05-22 23:57:10.155729261 +0000 UTC m=+0.397168343 container start 376efd3c5777993812fd21a83e819cc253d78012f69b757c2fa4ffe099f9be95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0) 2024-05-22T23:57:10.619 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-mds[67108]: -- 172.21.15.174:0/828263858 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55db294e9a40 con 0x55db294bcc00 2024-05-22T23:57:10.619 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:10 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c[67104]: starting mds.c at 2024-05-22T23:57:10.619 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:10 smithi174 bash[67089]: 376efd3c5777993812fd21a83e819cc253d78012f69b757c2fa4ffe099f9be95 2024-05-22T23:57:10.619 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 22 23:57:10 smithi174 systemd[1]: Started Ceph mds.c for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:10.640 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:10.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:10 smithi103 ceph-mon[31370]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:10.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.716 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.717 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:10.717 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:10.717 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:10 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:10 smithi105 ceph-mon[36426]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:10.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:10 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:10.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 -- 172.21.15.103:0/4137003180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b8108ee0 msgr2=0x7fe0b8109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 --2- 172.21.15.103:0/4137003180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b8108ee0 0x7fe0b8109360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0009900 tx=0x7fe0a002a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 -- 172.21.15.103:0/4137003180 shutdown_connections 2024-05-22T23:57:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 --2- 172.21.15.103:0/4137003180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0b81098a0 0x7fe0b8110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 --2- 172.21.15.103:0/4137003180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b8108ee0 0x7fe0b8109360 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 --2- 172.21.15.103:0/4137003180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b8107cf0 0x7fe0b8108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 -- 172.21.15.103:0/4137003180 >> 172.21.15.103:0/4137003180 conn(0x7fe0b8076cb0 msgr2=0x7fe0b80790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 -- 172.21.15.103:0/4137003180 shutdown_connections 2024-05-22T23:57:10.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.847+0000 7fe0b7200640 1 -- 172.21.15.103:0/4137003180 wait complete. 2024-05-22T23:57:10.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.848+0000 7fe0b7200640 1 Processor -- start 2024-05-22T23:57:10.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.848+0000 7fe0b7200640 1 -- start start 2024-05-22T23:57:10.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.848+0000 7fe0b7200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0b8107cf0 0x7fe0b819dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.848+0000 7fe0b7200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b8108ee0 0x7fe0b819e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.848+0000 7fe0b7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b81098a0 0x7fe0b81a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.848+0000 7fe0b7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8113950 con 0x7fe0b81098a0 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.848+0000 7fe0b7200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b81137d0 con 0x7fe0b8107cf0 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.848+0000 7fe0b7200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b8113ad0 con 0x7fe0b8108ee0 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0b8107cf0 0x7fe0b819dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0b8107cf0 0x7fe0b819dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:47284/0 (socket says 172.21.15.103:47284) 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b6800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b81098a0 0x7fe0b81a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 -- 172.21.15.103:0/3176044737 learned_addr learned my addr 172.21.15.103:0/3176044737 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0ad400640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b8108ee0 0x7fe0b819e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 -- 172.21.15.103:0/3176044737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b8108ee0 msgr2=0x7fe0b819e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:10.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b8108ee0 0x7fe0b819e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 -- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b81098a0 msgr2=0x7fe0b81a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b81098a0 0x7fe0b81a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 -- 172.21.15.103:0/3176044737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b81a5be0 con 0x7fe0b8107cf0 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0ad400640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b8108ee0 0x7fe0b819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.849+0000 7fe0b5e00640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0b8107cf0 0x7fe0b819dfc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800e9c0 tx=0x7fe0a800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.850+0000 7fe0af400640 1 -- 172.21.15.103:0/3176044737 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a800cde0 con 0x7fe0b8107cf0 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.850+0000 7fe0af400640 1 -- 172.21.15.103:0/3176044737 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a8010910 con 0x7fe0b8107cf0 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.850+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b81a5ed0 con 0x7fe0b8107cf0 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.850+0000 7fe0af400640 1 -- 172.21.15.103:0/3176044737 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8018a00 con 0x7fe0b8107cf0 2024-05-22T23:57:10.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.850+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b81a6330 con 0x7fe0b8107cf0 2024-05-22T23:57:10.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.851+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b810f060 con 0x7fe0b8107cf0 2024-05-22T23:57:10.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.853+0000 7fe0af400640 1 -- 172.21.15.103:0/3176044737 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe0a801f020 con 0x7fe0b8107cf0 2024-05-22T23:57:10.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.854+0000 7fe0af400640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe09c0a2b90 0x7fe09c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:10.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.854+0000 7fe0af400640 1 -- 172.21.15.103:0/3176044737 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fe0a80ceeb0 con 0x7fe0b8107cf0 2024-05-22T23:57:10.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.855+0000 7fe0ad400640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe09c0a2b90 0x7fe09c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:10.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.855+0000 7fe0ad400640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe09c0a2b90 0x7fe09c0a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0000c00 tx=0x7fe0a00095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:10.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:10.858+0000 7fe0af400640 1 -- 172.21.15.103:0/3176044737 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe0a809c210 con 0x7fe0b8107cf0 2024-05-22T23:57:11.104 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.100+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0b8062570 con 0x7fe0b8107cf0 2024-05-22T23:57:11.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.101+0000 7fe0af400640 1 -- 172.21.15.103:0/3176044737 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6364 (secure 0 0 0) 0x7fe0a809bb10 con 0x7fe0b8107cf0 2024-05-22T23:57:11.105 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:11.105 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-22T23:57:11.105 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 8 2024-05-22T23:57:11.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe09c0a2b90 msgr2=0x7fe09c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:11.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe09c0a2b90 0x7fe09c0a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe0a0000c00 tx=0x7fe0a00095f0 comp rx=0 tx=0).stop 2024-05-22T23:57:11.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0b8107cf0 msgr2=0x7fe0b819dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:11.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0b8107cf0 0x7fe0b819dfc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800e9c0 tx=0x7fe0a800ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:11.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 shutdown_connections 2024-05-22T23:57:11.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe09c0a2b90 0x7fe09c0a5050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:11.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0b81098a0 0x7fe0b81a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:11.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0b8108ee0 0x7fe0b819e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:11.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 --2- 172.21.15.103:0/3176044737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0b8107cf0 0x7fe0b819dfc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:11.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.103+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 >> 172.21.15.103:0/3176044737 conn(0x7fe0b8076cb0 msgr2=0x7fe0b810c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:11.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.104+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 shutdown_connections 2024-05-22T23:57:11.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:11.104+0000 7fe0b7200640 1 -- 172.21.15.103:0/3176044737 wait complete. 2024-05-22T23:57:11.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:11 smithi105 ceph-mon[36426]: Deploying daemon mds.h on smithi105 2024-05-22T23:57:11.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:11 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6836/2641083628,v1:172.21.15.174:6837/2641083628] up:boot 2024-05-22T23:57:11.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:11 smithi105 ceph-mon[36426]: fsmap 7 up:standby 2024-05-22T23:57:11.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:11 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-22T23:57:11.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:11 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3176044737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:11.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:11 smithi103 ceph-mon[31370]: Deploying daemon mds.h on smithi105 2024-05-22T23:57:11.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:11 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6836/2641083628,v1:172.21.15.174:6837/2641083628] up:boot 2024-05-22T23:57:11.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:11 smithi103 ceph-mon[31370]: fsmap 7 up:standby 2024-05-22T23:57:11.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:11 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-22T23:57:11.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:11 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3176044737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:11.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:11 smithi174 ceph-mon[36179]: Deploying daemon mds.h on smithi105 2024-05-22T23:57:11.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:11 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6836/2641083628,v1:172.21.15.174:6837/2641083628] up:boot 2024-05-22T23:57:11.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:11 smithi174 ceph-mon[36179]: fsmap 7 up:standby 2024-05-22T23:57:11.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:11 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-22T23:57:11.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:11 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3176044737' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:12.719 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:12 smithi105 ceph-mon[36426]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:12.719 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:12 smithi105 systemd[1]: Starting Ceph mds.h for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:12.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:12 smithi103 ceph-mon[31370]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:12.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:12 smithi174 ceph-mon[36179]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:13.012 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:12 smithi105 podman[66648]: 2024-05-22 23:57:12.618410078 +0000 UTC m=+0.016890835 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:13.012 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:12 smithi105 podman[66648]: 2024-05-22 23:57:12.739505268 +0000 UTC m=+0.137986027 container create bdac9b6bbd8f597bf97b66ce6959dcddc1d3f6fdc7e81dafd0e479c49428bf11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-22T23:57:13.012 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:13 smithi105 podman[66648]: 2024-05-22 23:57:13.009234679 +0000 UTC m=+0.407715436 container init bdac9b6bbd8f597bf97b66ce6959dcddc1d3f6fdc7e81dafd0e479c49428bf11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-22T23:57:13.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:13.309 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:13 smithi105 podman[66648]: 2024-05-22 23:57:13.011796477 +0000 UTC m=+0.410277237 container start bdac9b6bbd8f597bf97b66ce6959dcddc1d3f6fdc7e81dafd0e479c49428bf11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-22T23:57:13.309 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:13 smithi105 ceph-mds[66667]: -- 172.21.15.105:0/2138417347 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a18b477a40 con 0x55a18b44ac00 2024-05-22T23:57:13.309 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h[66663]: starting mds.h at 2024-05-22T23:57:13.309 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:13 smithi105 bash[66648]: bdac9b6bbd8f597bf97b66ce6959dcddc1d3f6fdc7e81dafd0e479c49428bf11 2024-05-22T23:57:13.309 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 22 23:57:13 smithi105 systemd[1]: Started Ceph mds.h for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:13.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 -- 172.21.15.103:0/1483085166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc104ce0 msgr2=0x7ff5bc105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:13.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 --2- 172.21.15.103:0/1483085166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc104ce0 0x7ff5bc105160 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff5a4009900 tx=0x7ff5a402a410 comp rx=0 tx=0).stop 2024-05-22T23:57:13.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 -- 172.21.15.103:0/1483085166 shutdown_connections 2024-05-22T23:57:13.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 --2- 172.21.15.103:0/1483085166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5bc1056a0 0x7ff5bc10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 --2- 172.21.15.103:0/1483085166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc104ce0 0x7ff5bc105160 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 --2- 172.21.15.103:0/1483085166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5bc103af0 0x7ff5bc103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 -- 172.21.15.103:0/1483085166 >> 172.21.15.103:0/1483085166 conn(0x7ff5bc0ff060 msgr2=0x7ff5bc1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:13.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 -- 172.21.15.103:0/1483085166 shutdown_connections 2024-05-22T23:57:13.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.330+0000 7ff5c3000640 1 -- 172.21.15.103:0/1483085166 wait complete. 2024-05-22T23:57:13.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.331+0000 7ff5c3000640 1 Processor -- start 2024-05-22T23:57:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.331+0000 7ff5c3000640 1 -- start start 2024-05-22T23:57:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.331+0000 7ff5c3000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5bc103af0 0x7ff5bc199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.331+0000 7ff5c3000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5bc104ce0 0x7ff5bc19a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.331+0000 7ff5c3000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc1056a0 0x7ff5bc1a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.331+0000 7ff5c3000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc10f7a0 con 0x7ff5bc1056a0 2024-05-22T23:57:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.331+0000 7ff5c3000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc10f620 con 0x7ff5bc104ce0 2024-05-22T23:57:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.331+0000 7ff5c3000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff5bc10f920 con 0x7ff5bc103af0 2024-05-22T23:57:13.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5bbe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5bc103af0 0x7ff5bc199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5c1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc1056a0 0x7ff5bc1a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5bbe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5bc103af0 0x7ff5bc199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:47436/0 (socket says 172.21.15.103:47436) 2024-05-22T23:57:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5bbe00640 1 -- 172.21.15.103:0/401351157 learned_addr learned my addr 172.21.15.103:0/401351157 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5b3400640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5bc104ce0 0x7ff5bc19a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5c1000640 1 -- 172.21.15.103:0/401351157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5bc103af0 msgr2=0x7ff5bc199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5c1000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5bc103af0 0x7ff5bc199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5c1000640 1 -- 172.21.15.103:0/401351157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5bc104ce0 msgr2=0x7ff5bc19a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:13.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5c1000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5bc104ce0 0x7ff5bc19a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5c1000640 1 -- 172.21.15.103:0/401351157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5bc1a1a10 con 0x7ff5bc1056a0 2024-05-22T23:57:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.332+0000 7ff5c1000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc1056a0 0x7ff5bc1a13a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009fd0 tx=0x7ff5b400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.333+0000 7ff5b9600640 1 -- 172.21.15.103:0/401351157 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b400ce20 con 0x7ff5bc1056a0 2024-05-22T23:57:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.333+0000 7ff5b9600640 1 -- 172.21.15.103:0/401351157 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5b4005020 con 0x7ff5bc1056a0 2024-05-22T23:57:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.333+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5bc1a1d00 con 0x7ff5bc1056a0 2024-05-22T23:57:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.333+0000 7ff5b9600640 1 -- 172.21.15.103:0/401351157 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5b4010720 con 0x7ff5bc1056a0 2024-05-22T23:57:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.333+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5bc1a21d0 con 0x7ff5bc1056a0 2024-05-22T23:57:13.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.334+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5bc10ae10 con 0x7ff5bc1056a0 2024-05-22T23:57:13.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.336+0000 7ff5b9600640 1 -- 172.21.15.103:0/401351157 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff5b4005540 con 0x7ff5bc1056a0 2024-05-22T23:57:13.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.338+0000 7ff5b9600640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff5a00a2b90 0x7ff5a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:13.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.339+0000 7ff5b9600640 1 -- 172.21.15.103:0/401351157 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff5b40cf2d0 con 0x7ff5bc1056a0 2024-05-22T23:57:13.343 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.339+0000 7ff5bbe00640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff5a00a2b90 0x7ff5a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:13.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.339+0000 7ff5bbe00640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff5a00a2b90 0x7ff5a00a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0058c0 tx=0x7ff5ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:13.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.341+0000 7ff5b9600640 1 -- 172.21.15.103:0/401351157 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff5b409c630 con 0x7ff5bc1056a0 2024-05-22T23:57:13.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.584+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5bc02d2a0 con 0x7ff5bc1056a0 2024-05-22T23:57:13.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.585+0000 7ff5b9600640 1 -- 172.21.15.103:0/401351157 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7192 (secure 0 0 0) 0x7ff5b409bf30 con 0x7ff5bc1056a0 2024-05-22T23:57:13.590 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:13.590 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-22T23:57:13.590 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 9 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff5a00a2b90 msgr2=0x7ff5a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff5a00a2b90 0x7ff5a00a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac0058c0 tx=0x7ff5ac005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc1056a0 msgr2=0x7ff5bc1a13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc1056a0 0x7ff5bc1a13a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009fd0 tx=0x7ff5b400ef10 comp rx=0 tx=0).stop 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 shutdown_connections 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff5a00a2b90 0x7ff5a00a5050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5bc1056a0 0x7ff5bc1a13a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff5bc104ce0 0x7ff5bc19a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 --2- 172.21.15.103:0/401351157 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff5bc103af0 0x7ff5bc199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:13.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 >> 172.21.15.103:0/401351157 conn(0x7ff5bc0ff060 msgr2=0x7ff5bc107fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:13.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 shutdown_connections 2024-05-22T23:57:13.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:13.588+0000 7ff5c3000640 1 -- 172.21.15.103:0/401351157 wait complete. 2024-05-22T23:57:14.531 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.531 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.531 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.531 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:14.531 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:14.532 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:14.532 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: Deploying daemon mds.i on smithi174 2024-05-22T23:57:14.532 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.105:6836/2433750399,v1:172.21.15.105:6837/2433750399] up:boot 2024-05-22T23:57:14.532 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: fsmap 8 up:standby 2024-05-22T23:57:14.532 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-22T23:57:14.532 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/401351157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:14.532 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:14 smithi174 ceph-mon[36179]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:14.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:14.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: Deploying daemon mds.i on smithi174 2024-05-22T23:57:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.105:6836/2433750399,v1:172.21.15.105:6837/2433750399] up:boot 2024-05-22T23:57:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: fsmap 8 up:standby 2024-05-22T23:57:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-22T23:57:14.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/401351157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:14.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:14 smithi105 ceph-mon[36426]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:14.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:14.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:14.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:14.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:14.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: Deploying daemon mds.i on smithi174 2024-05-22T23:57:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.105:6836/2433750399,v1:172.21.15.105:6837/2433750399] up:boot 2024-05-22T23:57:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: fsmap 8 up:standby 2024-05-22T23:57:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-22T23:57:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/401351157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:14.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:14 smithi103 ceph-mon[31370]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:15.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:15.618 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:15 smithi174 systemd[1]: Starting Ceph mds.i for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:15.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.819+0000 7f26c8400640 1 -- 172.21.15.103:0/1812444375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26c0104ce0 msgr2=0x7f26c0105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:15.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.819+0000 7f26c8400640 1 --2- 172.21.15.103:0/1812444375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26c0104ce0 0x7f26c0105160 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26ac009900 tx=0x7f26ac02a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:15.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.819+0000 7f26c8400640 1 -- 172.21.15.103:0/1812444375 shutdown_connections 2024-05-22T23:57:15.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.819+0000 7f26c8400640 1 --2- 172.21.15.103:0/1812444375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26c01056a0 0x7f26c010bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:15.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.819+0000 7f26c8400640 1 --2- 172.21.15.103:0/1812444375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26c0104ce0 0x7f26c0105160 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:15.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.819+0000 7f26c8400640 1 --2- 172.21.15.103:0/1812444375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c0103af0 0x7f26c0103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:15.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.819+0000 7f26c8400640 1 -- 172.21.15.103:0/1812444375 >> 172.21.15.103:0/1812444375 conn(0x7f26c00ff060 msgr2=0x7f26c01014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:15.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.819+0000 7f26c8400640 1 -- 172.21.15.103:0/1812444375 shutdown_connections 2024-05-22T23:57:15.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.820+0000 7f26c8400640 1 -- 172.21.15.103:0/1812444375 wait complete. 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.821+0000 7f26c8400640 1 Processor -- start 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.821+0000 7f26c8400640 1 -- start start 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.821+0000 7f26c8400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26c0103af0 0x7f26c0199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.821+0000 7f26c8400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c0104ce0 0x7f26c019a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.821+0000 7f26c8400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26c01056a0 0x7f26c01a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.821+0000 7f26c8400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26c010f770 con 0x7f26c0104ce0 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.821+0000 7f26c8400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f26c010f5f0 con 0x7f26c0103af0 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.821+0000 7f26c8400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f26c010f8f0 con 0x7f26c01056a0 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c6400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26c01056a0 0x7f26c01a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c5a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26c0103af0 0x7f26c0199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:15.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c0104ce0 0x7f26c019a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c6400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26c01056a0 0x7f26c01a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:48192/0 (socket says 172.21.15.103:48192) 2024-05-22T23:57:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c6400640 1 -- 172.21.15.103:0/2907641568 learned_addr learned my addr 172.21.15.103:0/2907641568 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c6400640 1 -- 172.21.15.103:0/2907641568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26c0103af0 msgr2=0x7f26c0199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c6400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26c0103af0 0x7f26c0199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c6400640 1 -- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c0104ce0 msgr2=0x7f26c019a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c6400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c0104ce0 0x7f26c019a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:15.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.822+0000 7f26c6400640 1 -- 172.21.15.103:0/2907641568 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26c01a1a30 con 0x7f26c01056a0 2024-05-22T23:57:15.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.850+0000 7f26c6400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26c01056a0 0x7f26c01a13c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26b800e970 tx=0x7f26b800ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:15.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.869+0000 7f26be000640 1 -- 172.21.15.103:0/2907641568 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b800ccc0 con 0x7f26c01056a0 2024-05-22T23:57:15.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.870+0000 7f26be000640 1 -- 172.21.15.103:0/2907641568 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26b800ce60 con 0x7f26c01056a0 2024-05-22T23:57:15.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.870+0000 7f26be000640 1 -- 172.21.15.103:0/2907641568 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b8010490 con 0x7f26c01056a0 2024-05-22T23:57:15.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.870+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26c01a1d20 con 0x7f26c01056a0 2024-05-22T23:57:15.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.870+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26c01a2190 con 0x7f26c01056a0 2024-05-22T23:57:15.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.872+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26c010ae10 con 0x7f26c01056a0 2024-05-22T23:57:15.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.873+0000 7f26be000640 1 -- 172.21.15.103:0/2907641568 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f26b8005020 con 0x7f26c01056a0 2024-05-22T23:57:15.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.875+0000 7f26be000640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26a40a2c10 0x7f26a40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:15.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.875+0000 7f26be000640 1 -- 172.21.15.103:0/2907641568 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f26b80cef20 con 0x7f26c01056a0 2024-05-22T23:57:15.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.875+0000 7f26c5a00640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26a40a2c10 0x7f26a40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:15.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.876+0000 7f26c5a00640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26a40a2c10 0x7f26a40a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f26c0104b40 tx=0x7f26b0009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:15.882 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:15.879+0000 7f26be000640 1 -- 172.21.15.103:0/2907641568 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f26b809c280 con 0x7f26c01056a0 2024-05-22T23:57:15.929 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:15 smithi174 podman[67394]: 2024-05-22 23:57:15.568948256 +0000 UTC m=+0.017129705 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:15.929 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:15 smithi174 podman[67394]: 2024-05-22 23:57:15.690090788 +0000 UTC m=+0.138272240 container create da829a48158a3f0e3011bd119371adc1eaa434d069560525260f8b6a5d7c3243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-22T23:57:16.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.126+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26c00624a0 con 0x7f26c01056a0 2024-05-22T23:57:16.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.128+0000 7f26be000640 1 -- 172.21.15.103:0/2907641568 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8022 (secure 0 0 0) 0x7f26b809bb80 con 0x7f26c01056a0 2024-05-22T23:57:16.131 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:16.131 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-22T23:57:16.132 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 10 2024-05-22T23:57:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.130+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26a40a2c10 msgr2=0x7f26a40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.130+0000 7f26c8400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26a40a2c10 0x7f26a40a50d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f26c0104b40 tx=0x7f26b0009720 comp rx=0 tx=0).stop 2024-05-22T23:57:16.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.130+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26c01056a0 msgr2=0x7f26c01a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:16.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.130+0000 7f26c8400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26c01056a0 0x7f26c01a13c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f26b800e970 tx=0x7f26b800ed70 comp rx=0 tx=0).stop 2024-05-22T23:57:16.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.131+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 shutdown_connections 2024-05-22T23:57:16.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.131+0000 7f26c8400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26a40a2c10 0x7f26a40a50d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:16.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.131+0000 7f26c8400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26c01056a0 0x7f26c01a13c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:16.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.131+0000 7f26c8400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26c0104ce0 0x7f26c019a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:16.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.131+0000 7f26c8400640 1 --2- 172.21.15.103:0/2907641568 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26c0103af0 0x7f26c0199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:16.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.131+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 >> 172.21.15.103:0/2907641568 conn(0x7f26c00ff060 msgr2=0x7f26c0107fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:16.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.131+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 shutdown_connections 2024-05-22T23:57:16.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:16.131+0000 7f26c8400640 1 -- 172.21.15.103:0/2907641568 wait complete. 2024-05-22T23:57:16.368 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:15 smithi174 podman[67394]: 2024-05-22 23:57:15.926428676 +0000 UTC m=+0.374610119 container init da829a48158a3f0e3011bd119371adc1eaa434d069560525260f8b6a5d7c3243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762) 2024-05-22T23:57:16.368 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:15 smithi174 podman[67394]: 2024-05-22 23:57:15.929138274 +0000 UTC m=+0.377319723 container start da829a48158a3f0e3011bd119371adc1eaa434d069560525260f8b6a5d7c3243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-22T23:57:16.368 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:15 smithi174 ceph-mds[67413]: -- 172.21.15.174:0/4125182583 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559b315b1a40 con 0x559b31584800 2024-05-22T23:57:16.368 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: starting mds.i at 2024-05-22T23:57:16.369 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:16 smithi174 bash[67394]: da829a48158a3f0e3011bd119371adc1eaa434d069560525260f8b6a5d7c3243 2024-05-22T23:57:16.369 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 22 23:57:16 smithi174 systemd[1]: Started Ceph mds.i for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:boot 2024-05-22T23:57:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: fsmap 9 up:standby 2024-05-22T23:57:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-22T23:57:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-22T23:57:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:17.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2907641568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:17.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:17.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:17.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:17 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:17.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:boot 2024-05-22T23:57:17.327 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: fsmap 9 up:standby 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2907641568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:17.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:17 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:boot 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: fsmap 9 up:standby 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2907641568' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:17.356 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:17.357 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:17 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:18.155 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:18.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:18 smithi103 ceph-mon[31370]: Deploying daemon mds.g on smithi103 2024-05-22T23:57:18.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 -- 172.21.15.103:0/1913929065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0750109060 msgr2=0x7f07501094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:18.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 --2- 172.21.15.103:0/1913929065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0750109060 0x7f07501094e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0738009860 tx=0x7f073802a340 comp rx=0 tx=0).stop 2024-05-22T23:57:18.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 -- 172.21.15.103:0/1913929065 shutdown_connections 2024-05-22T23:57:18.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 --2- 172.21.15.103:0/1913929065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0750109a20 0x7f0750110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 --2- 172.21.15.103:0/1913929065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0750109060 0x7f07501094e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 --2- 172.21.15.103:0/1913929065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0750107e80 0x7f07501082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 -- 172.21.15.103:0/1913929065 >> 172.21.15.103:0/1913929065 conn(0x7f0750076e90 msgr2=0x7f07500792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:18.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 -- 172.21.15.103:0/1913929065 shutdown_connections 2024-05-22T23:57:18.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.364+0000 7f0757200640 1 -- 172.21.15.103:0/1913929065 wait complete. 2024-05-22T23:57:18.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:18 smithi174 ceph-mon[36179]: Deploying daemon mds.g on smithi103 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.365+0000 7f0757200640 1 Processor -- start 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.365+0000 7f0757200640 1 -- start start 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f0757200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0750107e80 0x7f075019e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f0757200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0750109060 0x7f075019e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f0757200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0750109a20 0x7f07501a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f0757200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0750113b50 con 0x7f0750109a20 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f0757200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f07501139d0 con 0x7f0750109060 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f0757200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0750113cd0 con 0x7f0750107e80 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f074fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0750107e80 0x7f075019e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:18.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f0747400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0750109060 0x7f075019e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f0755200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0750109a20 0x7f07501a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f074fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0750107e80 0x7f075019e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:48202/0 (socket says 172.21.15.103:48202) 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.366+0000 7f074fe00640 1 -- 172.21.15.103:0/3234336180 learned_addr learned my addr 172.21.15.103:0/3234336180 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f0747400640 1 -- 172.21.15.103:0/3234336180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0750107e80 msgr2=0x7f075019e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f0747400640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0750107e80 0x7f075019e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f0747400640 1 -- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0750109a20 msgr2=0x7f07501a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f0747400640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0750109a20 0x7f07501a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f0747400640 1 -- 172.21.15.103:0/3234336180 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07501a5dc0 con 0x7f0750109060 2024-05-22T23:57:18.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f0747400640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0750109060 0x7f075019e6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0738035040 tx=0x7f0738033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:18.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f074d600640 1 -- 172.21.15.103:0/3234336180 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738037040 con 0x7f0750109060 2024-05-22T23:57:18.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f074d600640 1 -- 172.21.15.103:0/3234336180 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0738030850 con 0x7f0750109060 2024-05-22T23:57:18.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07501a6050 con 0x7f0750109060 2024-05-22T23:57:18.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f074d600640 1 -- 172.21.15.103:0/3234336180 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07380046c0 con 0x7f0750109060 2024-05-22T23:57:18.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.367+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07501a6510 con 0x7f0750109060 2024-05-22T23:57:18.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.369+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f075010f220 con 0x7f0750109060 2024-05-22T23:57:18.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.372+0000 7f074d600640 1 -- 172.21.15.103:0/3234336180 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f0738004050 con 0x7f0750109060 2024-05-22T23:57:18.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.374+0000 7f074d600640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f07340a2b90 0x7f07340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:18.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.374+0000 7f074d600640 1 -- 172.21.15.103:0/3234336180 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f07380eea40 con 0x7f0750109060 2024-05-22T23:57:18.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.374+0000 7f074fe00640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f07340a2b90 0x7f07340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:18.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.375+0000 7f074fe00640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f07340a2b90 0x7f07340a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f07400096c0 tx=0x7f0740009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:18.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.378+0000 7f074d600640 1 -- 172.21.15.103:0/3234336180 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f07380bbda0 con 0x7f0750109060 2024-05-22T23:57:18.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:18 smithi105 ceph-mon[36426]: Deploying daemon mds.g on smithi103 2024-05-22T23:57:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.621+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0750062470 con 0x7f0750109060 2024-05-22T23:57:18.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.622+0000 7f074d600640 1 -- 172.21.15.103:0/3234336180 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8022 (secure 0 0 0) 0x7f07380bb6a0 con 0x7f0750109060 2024-05-22T23:57:18.626 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:18.626 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-22T23:57:18.626 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 10 2024-05-22T23:57:18.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f07340a2b90 msgr2=0x7f07340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:18.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f07340a2b90 0x7f07340a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f07400096c0 tx=0x7f0740009260 comp rx=0 tx=0).stop 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0750109060 msgr2=0x7f075019e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0750109060 0x7f075019e6e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0738035040 tx=0x7f0738033d90 comp rx=0 tx=0).stop 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 shutdown_connections 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f07340a2b90 0x7f07340a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0750109a20 0x7f07501a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0750109060 0x7f075019e6e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 --2- 172.21.15.103:0/3234336180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0750107e80 0x7f075019e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 >> 172.21.15.103:0/3234336180 conn(0x7f0750076e90 msgr2=0x7f075010c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 shutdown_connections 2024-05-22T23:57:18.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:18.625+0000 7f0757200640 1 -- 172.21.15.103:0/3234336180 wait complete. 2024-05-22T23:57:19.250 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:19 smithi103 ceph-mon[31370]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:19.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:19 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3234336180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:19.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:19 smithi174 ceph-mon[36179]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:19.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:19 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3234336180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:19.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:19 smithi105 ceph-mon[36426]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:19.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:19 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3234336180' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:19.580 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:19 smithi103 systemd[1]: Starting Ceph mds.g for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:19.910 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:19 smithi103 podman[72990]: 2024-05-22 23:57:19.636382886 +0000 UTC m=+0.128367408 container create df066456361cf869bc6bdeecff7742dfc2b7b8d65ceedfec7471c9fd32c4ec0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521) 2024-05-22T23:57:19.910 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:19 smithi103 podman[72990]: 2024-05-22 23:57:19.543858043 +0000 UTC m=+0.035842533 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:19.910 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:19 smithi103 podman[72990]: 2024-05-22 23:57:19.90664902 +0000 UTC m=+0.398633521 container init df066456361cf869bc6bdeecff7742dfc2b7b8d65ceedfec7471c9fd32c4ec0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True) 2024-05-22T23:57:20.330 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:19 smithi103 podman[72990]: 2024-05-22 23:57:19.913853351 +0000 UTC m=+0.405837828 container start df066456361cf869bc6bdeecff7742dfc2b7b8d65ceedfec7471c9fd32c4ec0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-22T23:57:20.330 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:19 smithi103 ceph-mds[73014]: -- 172.21.15.103:0/424499335 <== mon.0 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561aecbcba40 con 0x561aecb9ec00 2024-05-22T23:57:20.330 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:19 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g[73008]: starting mds.g at 2024-05-22T23:57:20.330 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:19 smithi103 bash[72990]: df066456361cf869bc6bdeecff7742dfc2b7b8d65ceedfec7471c9fd32c4ec0d 2024-05-22T23:57:20.330 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 22 23:57:20 smithi103 systemd[1]: Started Ceph mds.g for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:20.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:20.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 -- 172.21.15.103:0/196856234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64104b00 msgr2=0x7fea64104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:20.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 --2- 172.21.15.103:0/196856234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64104b00 0x7fea64104f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea50009900 tx=0x7fea5002a410 comp rx=0 tx=0).stop 2024-05-22T23:57:20.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 -- 172.21.15.103:0/196856234 shutdown_connections 2024-05-22T23:57:20.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 --2- 172.21.15.103:0/196856234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea641054c0 0x7fea6410bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:20.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 --2- 172.21.15.103:0/196856234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64104b00 0x7fea64104f80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:20.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 --2- 172.21.15.103:0/196856234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea64103910 0x7fea64103d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:20.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 -- 172.21.15.103:0/196856234 >> 172.21.15.103:0/196856234 conn(0x7fea640fee80 msgr2=0x7fea641012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:20.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 -- 172.21.15.103:0/196856234 shutdown_connections 2024-05-22T23:57:20.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.841+0000 7fea6a600640 1 -- 172.21.15.103:0/196856234 wait complete. 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.842+0000 7fea6a600640 1 Processor -- start 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.842+0000 7fea6a600640 1 -- start start 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea6a600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64103910 0x7fea64199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea6a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea64104b00 0x7fea6419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea6a600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea641054c0 0x7fea641a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea6a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea6410f590 con 0x7fea64104b00 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea6a600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fea6410f410 con 0x7fea64103910 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea6a600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fea6410f710 con 0x7fea641054c0 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea63e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea64104b00 0x7fea6419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea69200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64103910 0x7fea64199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:20.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea69200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64103910 0x7fea64199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:34172/0 (socket says 172.21.15.103:34172) 2024-05-22T23:57:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea69200640 1 -- 172.21.15.103:0/3401697687 learned_addr learned my addr 172.21.15.103:0/3401697687 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.843+0000 7fea69c00640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea641054c0 0x7fea641a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea69200640 1 -- 172.21.15.103:0/3401697687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea641054c0 msgr2=0x7fea641a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea69200640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea641054c0 0x7fea641a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea69200640 1 -- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea64104b00 msgr2=0x7fea6419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea69200640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea64104b00 0x7fea6419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea69200640 1 -- 172.21.15.103:0/3401697687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea641a1890 con 0x7fea64103910 2024-05-22T23:57:20.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea69200640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64103910 0x7fea64199c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea5800e9c0 tx=0x7fea5800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:20.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea61600640 1 -- 172.21.15.103:0/3401697687 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5800cd80 con 0x7fea64103910 2024-05-22T23:57:20.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea641a1b80 con 0x7fea64103910 2024-05-22T23:57:20.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea61600640 1 -- 172.21.15.103:0/3401697687 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea580035f0 con 0x7fea64103910 2024-05-22T23:57:20.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea61600640 1 -- 172.21.15.103:0/3401697687 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea58010690 con 0x7fea64103910 2024-05-22T23:57:20.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.844+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea641a2080 con 0x7fea64103910 2024-05-22T23:57:20.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.845+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea6410ac80 con 0x7fea64103910 2024-05-22T23:57:20.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.849+0000 7fea61600640 1 -- 172.21.15.103:0/3401697687 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fea580037a0 con 0x7fea64103910 2024-05-22T23:57:20.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.850+0000 7fea61600640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea4c0a2b90 0x7fea4c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:20.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.850+0000 7fea61600640 1 -- 172.21.15.103:0/3401697687 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fea580cefa0 con 0x7fea64103910 2024-05-22T23:57:20.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.850+0000 7fea63e00640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea4c0a2b90 0x7fea4c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:20.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.851+0000 7fea63e00640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea4c0a2b90 0x7fea4c0a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fea500314e0 tx=0x7fea50000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:20.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:20.854+0000 7fea61600640 1 -- 172.21.15.103:0/3401697687 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fea5809c380 con 0x7fea64103910 2024-05-22T23:57:21.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.121+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea6402d200 con 0x7fea64103910 2024-05-22T23:57:21.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.122+0000 7fea61600640 1 -- 172.21.15.103:0/3401697687 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8851 (secure 0 0 0) 0x7fea5809bc80 con 0x7fea64103910 2024-05-22T23:57:21.126 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:21.126 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-22T23:57:21.126 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 11 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.125+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea4c0a2b90 msgr2=0x7fea4c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.125+0000 7fea6a600640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea4c0a2b90 0x7fea4c0a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fea500314e0 tx=0x7fea50000bc0 comp rx=0 tx=0).stop 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.125+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64103910 msgr2=0x7fea64199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.125+0000 7fea6a600640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64103910 0x7fea64199c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fea5800e9c0 tx=0x7fea5800ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.125+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 shutdown_connections 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.125+0000 7fea6a600640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea4c0a2b90 0x7fea4c0a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.125+0000 7fea6a600640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea641054c0 0x7fea641a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.126+0000 7fea6a600640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea64104b00 0x7fea6419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.126+0000 7fea6a600640 1 --2- 172.21.15.103:0/3401697687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea64103910 0x7fea64199c80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.126+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 >> 172.21.15.103:0/3401697687 conn(0x7fea640fee80 msgr2=0x7fea64107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.126+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 shutdown_connections 2024-05-22T23:57:21.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:21.126+0000 7fea6a600640 1 -- 172.21.15.103:0/3401697687 wait complete. 2024-05-22T23:57:21.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:21.339 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.103:6838/2610863022,v1:172.21.15.103:6839/2610863022] up:boot 2024-05-22T23:57:21.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: fsmap 10 up:standby 2024-05-22T23:57:21.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-22T23:57:21.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:21.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:21.340 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:21 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:21.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:21.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.103:6838/2610863022,v1:172.21.15.103:6839/2610863022] up:boot 2024-05-22T23:57:21.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: fsmap 10 up:standby 2024-05-22T23:57:21.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-22T23:57:21.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:21.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:21.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:21 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.103:6838/2610863022,v1:172.21.15.103:6839/2610863022] up:boot 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: fsmap 10 up:standby 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:21.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:21 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:22.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:22 smithi103 ceph-mon[31370]: Deploying daemon mds.a on smithi103 2024-05-22T23:57:22.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:22 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3401697687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:22.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:22 smithi174 ceph-mon[36179]: Deploying daemon mds.a on smithi103 2024-05-22T23:57:22.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:22 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3401697687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:22.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:22 smithi105 ceph-mon[36426]: Deploying daemon mds.a on smithi103 2024-05-22T23:57:22.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:22 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3401697687' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:23.147 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:23.222 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:23 smithi103 ceph-mon[31370]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:23.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 -- 172.21.15.103:0/614475911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 msgr2=0x7f814810bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:23.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 --2- 172.21.15.103:0/614475911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 0x7f814810bf30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8140009ca0 tx=0x7f814002a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 -- 172.21.15.103:0/614475911 shutdown_connections 2024-05-22T23:57:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 --2- 172.21.15.103:0/614475911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 0x7f814810bf30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 --2- 172.21.15.103:0/614475911 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8148104ce0 0x7f8148105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 --2- 172.21.15.103:0/614475911 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8148103af0 0x7f8148103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 -- 172.21.15.103:0/614475911 >> 172.21.15.103:0/614475911 conn(0x7f81480ff060 msgr2=0x7f81481014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 -- 172.21.15.103:0/614475911 shutdown_connections 2024-05-22T23:57:23.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.371+0000 7f814fa00640 1 -- 172.21.15.103:0/614475911 wait complete. 2024-05-22T23:57:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.372+0000 7f814fa00640 1 Processor -- start 2024-05-22T23:57:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.372+0000 7f814fa00640 1 -- start start 2024-05-22T23:57:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.372+0000 7f814fa00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8148103af0 0x7f8148199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.372+0000 7f814fa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8148104ce0 0x7f814819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.372+0000 7f814fa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 0x7f81481a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:23.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f814fa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f814810f7a0 con 0x7f81481056a0 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f814fa00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f814810f620 con 0x7f8148104ce0 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f814fa00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f814810f920 con 0x7f8148103af0 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f814da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 0x7f81481a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f814da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 0x7f81481a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43440/0 (socket says 172.21.15.103:43440) 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f814da00640 1 -- 172.21.15.103:0/2482895555 learned_addr learned my addr 172.21.15.103:0/2482895555 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f8147e00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8148104ce0 0x7f814819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f814d000640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8148103af0 0x7f8148199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f8147e00640 1 -- 172.21.15.103:0/2482895555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8148103af0 msgr2=0x7f8148199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:23.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f8147e00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8148103af0 0x7f8148199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f8147e00640 1 -- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 msgr2=0x7f81481a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f8147e00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 0x7f81481a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.373+0000 7f8147e00640 1 -- 172.21.15.103:0/2482895555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81481a1a30 con 0x7f8148104ce0 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.374+0000 7f8147e00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8148104ce0 0x7f814819a350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f813400e9f0 tx=0x7f813400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.374+0000 7f8145600640 1 -- 172.21.15.103:0/2482895555 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813400ccc0 con 0x7f8148104ce0 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.374+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81481a1d20 con 0x7f8148104ce0 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.374+0000 7f8145600640 1 -- 172.21.15.103:0/2482895555 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8134010910 con 0x7f8148104ce0 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.374+0000 7f8145600640 1 -- 172.21.15.103:0/2482895555 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8134018960 con 0x7f8148104ce0 2024-05-22T23:57:23.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.374+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81481a2260 con 0x7f8148104ce0 2024-05-22T23:57:23.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.375+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f814810ad40 con 0x7f8148104ce0 2024-05-22T23:57:23.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.378+0000 7f8145600640 1 -- 172.21.15.103:0/2482895555 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8134010ac0 con 0x7f8148104ce0 2024-05-22T23:57:23.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.379+0000 7f8145600640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81300a2c10 0x7f81300a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:23.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.380+0000 7f8145600640 1 -- 172.21.15.103:0/2482895555 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f81340ceed0 con 0x7f8148104ce0 2024-05-22T23:57:23.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.380+0000 7f814d000640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81300a2c10 0x7f81300a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:23.383 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.380+0000 7f814d000640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81300a2c10 0x7f81300a50d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f813c0095f0 tx=0x7f813c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:23.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.383+0000 7f8145600640 1 -- 172.21.15.103:0/2482895555 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f813409bbe0 con 0x7f8148104ce0 2024-05-22T23:57:23.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:23 smithi105 ceph-mon[36426]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:23.580 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:23 smithi103 systemd[1]: Starting Ceph mds.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:23.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:23 smithi174 ceph-mon[36179]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:23.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.634+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f814802d200 con 0x7f8148104ce0 2024-05-22T23:57:23.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.635+0000 7f8145600640 1 -- 172.21.15.103:0/2482895555 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8851 (secure 0 0 0) 0x7f813409fee0 con 0x7f8148104ce0 2024-05-22T23:57:23.639 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:23.639 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-22T23:57:23.639 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 11 2024-05-22T23:57:23.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81300a2c10 msgr2=0x7f81300a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81300a2c10 0x7f81300a50d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f813c0095f0 tx=0x7f813c005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8148104ce0 msgr2=0x7f814819a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8148104ce0 0x7f814819a350 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f813400e9f0 tx=0x7f813400eeb0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 shutdown_connections 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f81300a2c10 0x7f81300a50d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81481056a0 0x7f81481a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8148104ce0 0x7f814819a350 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 --2- 172.21.15.103:0/2482895555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8148103af0 0x7f8148199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.637+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 >> 172.21.15.103:0/2482895555 conn(0x7f81480ff060 msgr2=0x7f8148107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.638+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 shutdown_connections 2024-05-22T23:57:23.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:23.638+0000 7f814fa00640 1 -- 172.21.15.103:0/2482895555 wait complete. 2024-05-22T23:57:23.900 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:23 smithi103 podman[73377]: 2024-05-22 23:57:23.528382541 +0000 UTC m=+0.033192535 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:23.901 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:23 smithi103 podman[73377]: 2024-05-22 23:57:23.643247056 +0000 UTC m=+0.148057075 container create 4cf31a575965abee3965adfbd8e5ddd0d4ca6f95e433f6a490c4d371a4c3c560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0) 2024-05-22T23:57:23.901 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:23 smithi103 podman[73377]: 2024-05-22 23:57:23.897102511 +0000 UTC m=+0.401912574 container init 4cf31a575965abee3965adfbd8e5ddd0d4ca6f95e433f6a490c4d371a4c3c560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-22T23:57:24.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:24 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2482895555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:24.331 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:23 smithi103 podman[73377]: 2024-05-22 23:57:23.908351745 +0000 UTC m=+0.413161798 container start 4cf31a575965abee3965adfbd8e5ddd0d4ca6f95e433f6a490c4d371a4c3c560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-22T23:57:24.331 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:23 smithi103 ceph-mds[73398]: -- 172.21.15.103:0/372916459 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5628a3d1fa40 con 0x5628a3cf3000 2024-05-22T23:57:24.331 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:23 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a[73394]: starting mds.a at 2024-05-22T23:57:24.331 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:24 smithi103 bash[73377]: 4cf31a575965abee3965adfbd8e5ddd0d4ca6f95e433f6a490c4d371a4c3c560 2024-05-22T23:57:24.331 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 22 23:57:24 smithi103 systemd[1]: Started Ceph mds.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:24.459 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:24 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2482895555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:24.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:24 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2482895555' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:25.161 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] up:boot 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: fsmap 11 up:standby 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:25.162 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:25 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:25.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] up:boot 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: fsmap 11 up:standby 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:25.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:25 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:25.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:25.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] up:boot 2024-05-22T23:57:25.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: fsmap 11 up:standby 2024-05-22T23:57:25.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-22T23:57:25.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:25.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-22T23:57:25.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-22T23:57:25.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:25 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:25.660 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:25.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 -- 172.21.15.103:0/3759837448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba4103af0 msgr2=0x7faba4103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:25.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 --2- 172.21.15.103:0/3759837448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba4103af0 0x7faba4103f10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab98009860 tx=0x7fab9802a340 comp rx=0 tx=0).stop 2024-05-22T23:57:25.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 -- 172.21.15.103:0/3759837448 shutdown_connections 2024-05-22T23:57:25.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 --2- 172.21.15.103:0/3759837448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba41056a0 0x7faba410bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:25.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 --2- 172.21.15.103:0/3759837448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faba4104ce0 0x7faba4105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:25.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 --2- 172.21.15.103:0/3759837448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba4103af0 0x7faba4103f10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:25.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 -- 172.21.15.103:0/3759837448 >> 172.21.15.103:0/3759837448 conn(0x7faba40ff060 msgr2=0x7faba41014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:25.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 -- 172.21.15.103:0/3759837448 shutdown_connections 2024-05-22T23:57:25.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.870+0000 7fabab600640 1 -- 172.21.15.103:0/3759837448 wait complete. 2024-05-22T23:57:25.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.871+0000 7fabab600640 1 Processor -- start 2024-05-22T23:57:25.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.871+0000 7fabab600640 1 -- start start 2024-05-22T23:57:25.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.871+0000 7fabab600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba4103af0 0x7faba4199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:25.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7fabab600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faba4104ce0 0x7faba419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:25.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7fabab600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba41056a0 0x7faba41a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:25.876 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7fabab600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faba410f740 con 0x7faba4104ce0 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7fabab600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faba410f5c0 con 0x7faba41056a0 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7fabab600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faba410f8c0 con 0x7faba4103af0 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faba4104ce0 0x7faba419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba9600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba41056a0 0x7faba41a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba9600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba41056a0 0x7faba41a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:60512/0 (socket says 172.21.15.103:60512) 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba9600640 1 -- 172.21.15.103:0/493336315 learned_addr learned my addr 172.21.15.103:0/493336315 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba8c00640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba4103af0 0x7faba4199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba9600640 1 -- 172.21.15.103:0/493336315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba4103af0 msgr2=0x7faba4199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba9600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba4103af0 0x7faba4199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba9600640 1 -- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faba4104ce0 msgr2=0x7faba419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba9600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faba4104ce0 0x7faba419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:25.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba9600640 1 -- 172.21.15.103:0/493336315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faba41a1a30 con 0x7faba41056a0 2024-05-22T23:57:25.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.872+0000 7faba3e00640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faba4104ce0 0x7faba419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:25.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.873+0000 7faba9600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba41056a0 0x7faba41a13c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00eb10 tx=0x7fab9c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:25.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.873+0000 7faba1600640 1 -- 172.21.15.103:0/493336315 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c004b20 con 0x7faba41056a0 2024-05-22T23:57:25.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.873+0000 7faba1600640 1 -- 172.21.15.103:0/493336315 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab9c004cc0 con 0x7faba41056a0 2024-05-22T23:57:25.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.873+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faba41a1d20 con 0x7faba41056a0 2024-05-22T23:57:25.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.873+0000 7faba1600640 1 -- 172.21.15.103:0/493336315 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab9c010760 con 0x7faba41056a0 2024-05-22T23:57:25.878 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.873+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faba41a2210 con 0x7faba41056a0 2024-05-22T23:57:25.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.874+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba410ae90 con 0x7faba41056a0 2024-05-22T23:57:25.879 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.876+0000 7faba1600640 1 -- 172.21.15.103:0/493336315 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fab9c005020 con 0x7faba41056a0 2024-05-22T23:57:25.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.877+0000 7faba1600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fab8c0a2c10 0x7fab8c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:25.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.877+0000 7faba1600640 1 -- 172.21.15.103:0/493336315 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fab9c0cf170 con 0x7faba41056a0 2024-05-22T23:57:25.880 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.877+0000 7faba8c00640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fab8c0a2c10 0x7fab8c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:25.881 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.878+0000 7faba8c00640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fab8c0a2c10 0x7fab8c0a50d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fab98009830 tx=0x7fab98030570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:25.884 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:25.881+0000 7faba1600640 1 -- 172.21.15.103:0/493336315 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fab9c09c4d0 con 0x7faba41056a0 2024-05-22T23:57:26.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.123+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faba4062470 con 0x7faba41056a0 2024-05-22T23:57:26.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.125+0000 7faba1600640 1 -- 172.21.15.103:0/493336315 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9680 (secure 0 0 0) 0x7fab9c09bdd0 con 0x7faba41056a0 2024-05-22T23:57:26.128 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:26.128 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24563,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[]} 2024-05-22T23:57:26.128 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 12 2024-05-22T23:57:26.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fab8c0a2c10 msgr2=0x7fab8c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:26.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fab8c0a2c10 0x7fab8c0a50d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fab98009830 tx=0x7fab98030570 comp rx=0 tx=0).stop 2024-05-22T23:57:26.131 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba41056a0 msgr2=0x7faba41a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba41056a0 0x7faba41a13c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fab9c00eb10 tx=0x7fab9c00c2d0 comp rx=0 tx=0).stop 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 shutdown_connections 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fab8c0a2c10 0x7fab8c0a50d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faba41056a0 0x7faba41a13c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faba4104ce0 0x7faba419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 --2- 172.21.15.103:0/493336315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faba4103af0 0x7faba4199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 >> 172.21.15.103:0/493336315 conn(0x7faba40ff060 msgr2=0x7faba4108060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 shutdown_connections 2024-05-22T23:57:26.132 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:26.128+0000 7fabab600640 1 -- 172.21.15.103:0/493336315 wait complete. 2024-05-22T23:57:26.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:26 smithi105 ceph-mon[36426]: Deploying daemon mds.b on smithi105 2024-05-22T23:57:26.220 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:26 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/493336315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:26.559 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:26 smithi105 systemd[1]: Starting Ceph mds.b for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-22T23:57:26.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:26 smithi103 ceph-mon[31370]: Deploying daemon mds.b on smithi105 2024-05-22T23:57:26.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:26 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/493336315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:26.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:26 smithi174 ceph-mon[36179]: Deploying daemon mds.b on smithi105 2024-05-22T23:57:26.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:26 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/493336315' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:26.918 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:26 smithi105 podman[66961]: 2024-05-22 23:57:26.525618588 +0000 UTC m=+0.017378241 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-22T23:57:26.918 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:26 smithi105 podman[66961]: 2024-05-22 23:57:26.661883602 +0000 UTC m=+0.153643242 container create 76c9c4314656676e5f3e7908ae605ca11b6cd6542e1f0aa9255a02b5298e3c17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-22T23:57:26.918 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:26 smithi105 podman[66961]: 2024-05-22 23:57:26.914821183 +0000 UTC m=+0.406580826 container init 76c9c4314656676e5f3e7908ae605ca11b6cd6542e1f0aa9255a02b5298e3c17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-22T23:57:27.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:27 smithi105 ceph-mon[36426]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:27.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:27 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:27.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:27 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:27.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:27 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:27.310 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:26 smithi105 podman[66961]: 2024-05-22 23:57:26.917614341 +0000 UTC m=+0.409373990 container start 76c9c4314656676e5f3e7908ae605ca11b6cd6542e1f0aa9255a02b5298e3c17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-22T23:57:27.310 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:26 smithi105 ceph-mds[66980]: -- 172.21.15.105:0/3934320253 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d497be5a40 con 0x55d497bb8c00 2024-05-22T23:57:27.310 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:26 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b[66976]: starting mds.b at 2024-05-22T23:57:27.310 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:27 smithi105 bash[66961]: 76c9c4314656676e5f3e7908ae605ca11b6cd6542e1f0aa9255a02b5298e3c17 2024-05-22T23:57:27.310 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:27 smithi105 systemd[1]: Started Ceph mds.b for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-22T23:57:27.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:27 smithi103 ceph-mon[31370]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:27.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:27 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:27.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:27 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:27.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:27 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:27.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:27 smithi174 ceph-mon[36179]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:27.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:27 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:27.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:27 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:27.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:27 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:28.153 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:28.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.343+0000 7f3733200640 1 -- 172.21.15.103:0/3904348842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37341098a0 msgr2=0x7f3734110130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:28.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.343+0000 7f3733200640 1 --2- 172.21.15.103:0/3904348842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37341098a0 0x7f3734110130 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f372c009ca0 tx=0x7f372c02a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.343+0000 7f3733200640 1 -- 172.21.15.103:0/3904348842 shutdown_connections 2024-05-22T23:57:28.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.343+0000 7f3733200640 1 --2- 172.21.15.103:0/3904348842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37341098a0 0x7f3734110130 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.343+0000 7f3733200640 1 --2- 172.21.15.103:0/3904348842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3734108ee0 0x7f3734109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.343+0000 7f3733200640 1 --2- 172.21.15.103:0/3904348842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3734107cf0 0x7f3734108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.343+0000 7f3733200640 1 -- 172.21.15.103:0/3904348842 >> 172.21.15.103:0/3904348842 conn(0x7f3734076cb0 msgr2=0x7f37340790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:28.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.344+0000 7f3733200640 1 -- 172.21.15.103:0/3904348842 shutdown_connections 2024-05-22T23:57:28.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.344+0000 7f3733200640 1 -- 172.21.15.103:0/3904348842 wait complete. 2024-05-22T23:57:28.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3733200640 1 Processor -- start 2024-05-22T23:57:28.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3733200640 1 -- start start 2024-05-22T23:57:28.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3733200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3734107cf0 0x7f373419dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:28.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3733200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3734108ee0 0x7f373419e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:28.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3733200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37341098a0 0x7f37341a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:28.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3733200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f37341139c0 con 0x7f37341098a0 2024-05-22T23:57:28.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3733200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3734113840 con 0x7f3734108ee0 2024-05-22T23:57:28.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3733200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3734113b40 con 0x7f3734107cf0 2024-05-22T23:57:28.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f372be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3734108ee0 0x7f373419e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3731e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3734107cf0 0x7f373419dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3732800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37341098a0 0x7f37341a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3731e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3734107cf0 0x7f373419dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:50962/0 (socket says 172.21.15.103:50962) 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.345+0000 7f3731e00640 1 -- 172.21.15.103:0/3123297890 learned_addr learned my addr 172.21.15.103:0/3123297890 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f3731e00640 1 -- 172.21.15.103:0/3123297890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3734108ee0 msgr2=0x7f373419e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f3731e00640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3734108ee0 0x7f373419e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f3731e00640 1 -- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37341098a0 msgr2=0x7f37341a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f3731e00640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37341098a0 0x7f37341a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f3731e00640 1 -- 172.21.15.103:0/3123297890 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37341a5bc0 con 0x7f3734107cf0 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f372be00640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3734108ee0 0x7f373419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f3731e00640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3734107cf0 0x7f373419dfa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f372400e9c0 tx=0x7f372400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:28.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f372aa00640 1 -- 172.21.15.103:0/3123297890 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372400cd80 con 0x7f3734107cf0 2024-05-22T23:57:28.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f372aa00640 1 -- 172.21.15.103:0/3123297890 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37240035f0 con 0x7f3734107cf0 2024-05-22T23:57:28.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.346+0000 7f372aa00640 1 -- 172.21.15.103:0/3123297890 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3724010690 con 0x7f3734107cf0 2024-05-22T23:57:28.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.347+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37341a5eb0 con 0x7f3734107cf0 2024-05-22T23:57:28.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.347+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37341a6260 con 0x7f3734107cf0 2024-05-22T23:57:28.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.348+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f373410f010 con 0x7f3734107cf0 2024-05-22T23:57:28.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.350+0000 7f372aa00640 1 -- 172.21.15.103:0/3123297890 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f37240037a0 con 0x7f3734107cf0 2024-05-22T23:57:28.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.351+0000 7f372aa00640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37180a2b90 0x7f37180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:28.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.351+0000 7f372aa00640 1 -- 172.21.15.103:0/3123297890 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f37240ceff0 con 0x7f3734107cf0 2024-05-22T23:57:28.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.351+0000 7f372be00640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37180a2b90 0x7f37180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:28.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.352+0000 7f372be00640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37180a2b90 0x7f37180a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f373419f4b0 tx=0x7f371c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:28.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.353+0000 7f372aa00640 1 -- 172.21.15.103:0/3123297890 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f372409c390 con 0x7f3734107cf0 2024-05-22T23:57:28.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:28 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:28.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:28 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] up:boot 2024-05-22T23:57:28.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:28 smithi105 ceph-mon[36426]: fsmap 12 up:standby 2024-05-22T23:57:28.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:28 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-22T23:57:28.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:28 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:28.489 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:28 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:57:28.489 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:28 smithi105 ceph-mon[36426]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:28.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:28 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:28.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:28 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] up:boot 2024-05-22T23:57:28.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:28 smithi103 ceph-mon[31370]: fsmap 12 up:standby 2024-05-22T23:57:28.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:28 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-22T23:57:28.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:28 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:28.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:28 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:57:28.582 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:28 smithi103 ceph-mon[31370]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:28.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:28 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:28.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:28 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] up:boot 2024-05-22T23:57:28.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:28 smithi174 ceph-mon[36179]: fsmap 12 up:standby 2024-05-22T23:57:28.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:28 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-22T23:57:28.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:28 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:28.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:28 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:57:28.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:28 smithi174 ceph-mon[36179]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:28.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.625+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f373402d200 con 0x7f3734107cf0 2024-05-22T23:57:28.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.626+0000 7f372aa00640 1 -- 172.21.15.103:0/3123297890 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10506 (secure 0 0 0) 0x7f37240b3920 con 0x7f3734107cf0 2024-05-22T23:57:28.629 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:28.630 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24563,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-22T23:57:28.630 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 13 2024-05-22T23:57:28.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37180a2b90 msgr2=0x7f37180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:28.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37180a2b90 0x7f37180a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f373419f4b0 tx=0x7f371c005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:28.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3734107cf0 msgr2=0x7f373419dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:28.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3734107cf0 0x7f373419dfa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f372400e9c0 tx=0x7f372400ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:28.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 shutdown_connections 2024-05-22T23:57:28.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f37180a2b90 0x7f37180a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37341098a0 0x7f37341a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3734108ee0 0x7f373419e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 --2- 172.21.15.103:0/3123297890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3734107cf0 0x7f373419dfa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 >> 172.21.15.103:0/3123297890 conn(0x7f3734076cb0 msgr2=0x7f373410c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:28.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 shutdown_connections 2024-05-22T23:57:28.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.629+0000 7f3733200640 1 -- 172.21.15.103:0/3123297890 wait complete. 2024-05-22T23:57:28.651 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.850+0000 7fc68fe00640 1 -- 172.21.15.103:0/481382945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 msgr2=0x7fc690100a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.850+0000 7fc68fe00640 1 --2- 172.21.15.103:0/481382945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 0x7fc690100a10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc680009ca0 tx=0x7fc68002a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.850+0000 7fc68fe00640 1 -- 172.21.15.103:0/481382945 shutdown_connections 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.850+0000 7fc68fe00640 1 --2- 172.21.15.103:0/481382945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6901021a0 0x7fc690108a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.850+0000 7fc68fe00640 1 --2- 172.21.15.103:0/481382945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6901017e0 0x7fc690101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.850+0000 7fc68fe00640 1 --2- 172.21.15.103:0/481382945 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 0x7fc690100a10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.850+0000 7fc68fe00640 1 -- 172.21.15.103:0/481382945 >> 172.21.15.103:0/481382945 conn(0x7fc6900fbb40 msgr2=0x7fc6900fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.850+0000 7fc68fe00640 1 -- 172.21.15.103:0/481382945 shutdown_connections 2024-05-22T23:57:28.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.851+0000 7fc68fe00640 1 -- 172.21.15.103:0/481382945 wait complete. 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.851+0000 7fc68fe00640 1 Processor -- start 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68fe00640 1 -- start start 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 0x7fc690199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6901017e0 0x7fc69019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6901021a0 0x7fc6901a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc69010d3f0 con 0x7fc6901005f0 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68fe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc69010d270 con 0x7fc6901017e0 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68fe00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc69010d570 con 0x7fc6901021a0 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 0x7fc690199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 0x7fc690199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43062/0 (socket says 172.21.15.103:43062) 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68ea00640 1 -- 172.21.15.103:0/3698583239 learned_addr learned my addr 172.21.15.103:0/3698583239 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:28.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc686000640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6901017e0 0x7fc69019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.852+0000 7fc68f400640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6901021a0 0x7fc6901a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc68ea00640 1 -- 172.21.15.103:0/3698583239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6901021a0 msgr2=0x7fc6901a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc68ea00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6901021a0 0x7fc6901a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc68ea00640 1 -- 172.21.15.103:0/3698583239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6901017e0 msgr2=0x7fc69019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc68ea00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6901017e0 0x7fc69019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc68ea00640 1 -- 172.21.15.103:0/3698583239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6901a1860 con 0x7fc6901005f0 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc686000640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6901017e0 0x7fc69019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc68ea00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 0x7fc690199c50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc680000f80 tx=0x7fc6800319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc694a00640 1 -- 172.21.15.103:0/3698583239 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680003b70 con 0x7fc6901005f0 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc694a00640 1 -- 172.21.15.103:0/3698583239 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc680003d10 con 0x7fc6901005f0 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc694a00640 1 -- 172.21.15.103:0/3698583239 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68002fc50 con 0x7fc6901005f0 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6901a1af0 con 0x7fc6901005f0 2024-05-22T23:57:28.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.853+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6901a1ff0 con 0x7fc6901005f0 2024-05-22T23:57:28.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.855+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6901a2220 con 0x7fc6901005f0 2024-05-22T23:57:28.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.857+0000 7fc694a00640 1 -- 172.21.15.103:0/3698583239 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc680004040 con 0x7fc6901005f0 2024-05-22T23:57:28.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.858+0000 7fc694a00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc6740a2c10 0x7fc6740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:28.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.858+0000 7fc694a00640 1 -- 172.21.15.103:0/3698583239 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7fc6800eff50 con 0x7fc6901005f0 2024-05-22T23:57:28.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.858+0000 7fc686000640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc6740a2c10 0x7fc6740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:28.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.859+0000 7fc686000640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc6740a2c10 0x7fc6740a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc6780095f0 tx=0x7fc678005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:28.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:28.863+0000 7fc694a00640 1 -- 172.21.15.103:0/3698583239 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc6800bd350 con 0x7fc6901005f0 2024-05-22T23:57:29.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.073+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc6901a25b0 con 0x7fc6901005f0 2024-05-22T23:57:29.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.076+0000 7fc694a00640 1 -- 172.21.15.103:0/3698583239 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+18515 (secure 0 0 0) 0x7fc6800bcc50 con 0x7fc6901005f0 2024-05-22T23:57:29.080 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:29.080 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":83,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:56:35.739343+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:57:29.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc6740a2c10 msgr2=0x7fc6740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc6740a2c10 0x7fc6740a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc6780095f0 tx=0x7fc678005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:29.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 msgr2=0x7fc690199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 0x7fc690199c50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc680000f80 tx=0x7fc6800319f0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 shutdown_connections 2024-05-22T23:57:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc6740a2c10 0x7fc6740a50d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6901021a0 0x7fc6901a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6901017e0 0x7fc69019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 --2- 172.21.15.103:0/3698583239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc6901005f0 0x7fc690199c50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.078+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 >> 172.21.15.103:0/3698583239 conn(0x7fc6900fbb40 msgr2=0x7fc6901093f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.079+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 shutdown_connections 2024-05-22T23:57:29.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.079+0000 7fc68fe00640 1 -- 172.21.15.103:0/3698583239 wait complete. 2024-05-22T23:57:29.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-22T23:57:29.099 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-22T23:57:29.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 -- 172.21.15.103:0/2937203064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6748102350 msgr2=0x7f6748104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 --2- 172.21.15.103:0/2937203064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6748102350 0x7f6748104740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6738009900 tx=0x7f673802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 -- 172.21.15.103:0/2937203064 shutdown_connections 2024-05-22T23:57:29.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 --2- 172.21.15.103:0/2937203064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67481075b0 0x7f6748109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 --2- 172.21.15.103:0/2937203064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6748104c80 0x7f6748107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 --2- 172.21.15.103:0/2937203064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6748102350 0x7f6748104740 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 -- 172.21.15.103:0/2937203064 >> 172.21.15.103:0/2937203064 conn(0x7f67480fbd40 msgr2=0x7f67480fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:29.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 -- 172.21.15.103:0/2937203064 shutdown_connections 2024-05-22T23:57:29.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.292+0000 7f674e200640 1 -- 172.21.15.103:0/2937203064 wait complete. 2024-05-22T23:57:29.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.293+0000 7f674e200640 1 Processor -- start 2024-05-22T23:57:29.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.293+0000 7f674e200640 1 -- start start 2024-05-22T23:57:29.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.293+0000 7f674e200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6748102350 0x7f6748101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:29.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.293+0000 7f674e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6748104c80 0x7f6748100300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:29.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f674e200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67481075b0 0x7f6748100840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:29.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f674e200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f674810d5c0 con 0x7f6748104c80 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f674e200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f674810d440 con 0x7f6748102350 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f674e200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f674810d740 con 0x7f67481075b0 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6746a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6748104c80 0x7f6748100300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6746a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6748104c80 0x7f6748100300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43090/0 (socket says 172.21.15.103:43090) 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6746a00640 1 -- 172.21.15.103:0/2559540595 learned_addr learned my addr 172.21.15.103:0/2559540595 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6747e00640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67481075b0 0x7f6748100840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6747400640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6748102350 0x7f6748101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6746a00640 1 -- 172.21.15.103:0/2559540595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67481075b0 msgr2=0x7f6748100840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6746a00640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67481075b0 0x7f6748100840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6746a00640 1 -- 172.21.15.103:0/2559540595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6748102350 msgr2=0x7f6748101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6746a00640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6748102350 0x7f6748101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6746a00640 1 -- 172.21.15.103:0/2559540595 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67481010d0 con 0x7f6748104c80 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6747e00640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67481075b0 0x7f6748100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.294+0000 7f6747400640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6748102350 0x7f6748101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.295+0000 7f6746a00640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6748104c80 0x7f6748100300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f673400e9c0 tx=0x7f673400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.295+0000 7f673fe00640 1 -- 172.21.15.103:0/2559540595 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673400cd80 con 0x7f6748104c80 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.295+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67481a1d50 con 0x7f6748104c80 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.295+0000 7f673fe00640 1 -- 172.21.15.103:0/2559540595 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6734005500 con 0x7f6748104c80 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.295+0000 7f673fe00640 1 -- 172.21.15.103:0/2559540595 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6734010690 con 0x7f6748104c80 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.295+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67481a2260 con 0x7f6748104c80 2024-05-22T23:57:29.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.296+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67480fd830 con 0x7f6748104c80 2024-05-22T23:57:29.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.300+0000 7f673fe00640 1 -- 172.21.15.103:0/2559540595 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f67340046c0 con 0x7f6748104c80 2024-05-22T23:57:29.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.302+0000 7f673fe00640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f672c0a2b90 0x7f672c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:29.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.302+0000 7f673fe00640 1 -- 172.21.15.103:0/2559540595 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f67340cefe0 con 0x7f6748104c80 2024-05-22T23:57:29.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.302+0000 7f6747400640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f672c0a2b90 0x7f672c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:29.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.303+0000 7f6747400640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f672c0a2b90 0x7f672c0a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6738031440 tx=0x7f67380315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:29.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.307+0000 7f673fe00640 1 -- 172.21.15.103:0/2559540595 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f673409c380 con 0x7f6748104c80 2024-05-22T23:57:29.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:29 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3123297890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:29.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:29 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3698583239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:29.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.514+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67480624a0 con 0x7f6748104c80 2024-05-22T23:57:29.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.515+0000 7f673fe00640 1 -- 172.21.15.103:0/2559540595 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) ==== 93+0+10 (secure 0 0 0) 0x7f673409bc80 con 0x7f6748104c80 2024-05-22T23:57:29.518 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-22T23:57:29.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f672c0a2b90 msgr2=0x7f672c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f672c0a2b90 0x7f672c0a5050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6738031440 tx=0x7f67380315b0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6748104c80 msgr2=0x7f6748100300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6748104c80 0x7f6748100300 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f673400e9c0 tx=0x7f673400ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 shutdown_connections 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f672c0a2b90 0x7f672c0a5050 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67481075b0 0x7f6748100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6748104c80 0x7f6748100300 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 --2- 172.21.15.103:0/2559540595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6748102350 0x7f6748101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 >> 172.21.15.103:0/2559540595 conn(0x7f67480fbd40 msgr2=0x7f6748102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 shutdown_connections 2024-05-22T23:57:29.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.518+0000 7f674e200640 1 -- 172.21.15.103:0/2559540595 wait complete. 2024-05-22T23:57:29.539 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-22T23:57:29.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:29 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3123297890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:29.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:29 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3698583239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:29.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:29 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3123297890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:29.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:29 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3698583239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:29.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:29.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 -- 172.21.15.103:0/1641057600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0102340 msgr2=0x7f60c0104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 --2- 172.21.15.103:0/1641057600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0102340 0x7f60c0104730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009be0 tx=0x7f60b002a6c0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 -- 172.21.15.103:0/1641057600 shutdown_connections 2024-05-22T23:57:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 --2- 172.21.15.103:0/1641057600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60c01075a0 0x7f60c01099f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 --2- 172.21.15.103:0/1641057600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0104c70 0x7f60c0107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 --2- 172.21.15.103:0/1641057600 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0102340 0x7f60c0104730 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 -- 172.21.15.103:0/1641057600 >> 172.21.15.103:0/1641057600 conn(0x7f60c00fbd50 msgr2=0x7f60c00fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 -- 172.21.15.103:0/1641057600 shutdown_connections 2024-05-22T23:57:29.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.729+0000 7f60c5a00640 1 -- 172.21.15.103:0/1641057600 wait complete. 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.730+0000 7f60c5a00640 1 Processor -- start 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60c5a00640 1 -- start start 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60c5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0102340 0x7f60c0199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60c5a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0104c70 0x7f60c019a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60c5a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60c01075a0 0x7f60c01a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60c5a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60c010d560 con 0x7f60c0102340 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60c5a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f60c010d3e0 con 0x7f60c0104c70 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60c5a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f60c010d6e0 con 0x7f60c01075a0 2024-05-22T23:57:29.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60b6000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0104c70 0x7f60c019a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60b6000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0104c70 0x7f60c019a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:60576/0 (socket says 172.21.15.103:60576) 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60b6000640 1 -- 172.21.15.103:0/2790377353 learned_addr learned my addr 172.21.15.103:0/2790377353 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60bf400640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60c01075a0 0x7f60c01a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.731+0000 7f60bea00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0102340 0x7f60c0199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b6000640 1 -- 172.21.15.103:0/2790377353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60c01075a0 msgr2=0x7f60c01a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b6000640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60c01075a0 0x7f60c01a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b6000640 1 -- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0102340 msgr2=0x7f60c0199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b6000640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0102340 0x7f60c0199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b6000640 1 -- 172.21.15.103:0/2790377353 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60c01a1a10 con 0x7f60c0104c70 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b6000640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0104c70 0x7f60c019a330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f60a800e9f0 tx=0x7f60a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b7e00640 1 -- 172.21.15.103:0/2790377353 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a800ccc0 con 0x7f60c0104c70 2024-05-22T23:57:29.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b7e00640 1 -- 172.21.15.103:0/2790377353 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60a8010910 con 0x7f60c0104c70 2024-05-22T23:57:29.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60b7e00640 1 -- 172.21.15.103:0/2790377353 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a8018960 con 0x7f60c0104c70 2024-05-22T23:57:29.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.732+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60c01a1d00 con 0x7f60c0104c70 2024-05-22T23:57:29.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.733+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60c01a2240 con 0x7f60c0104c70 2024-05-22T23:57:29.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.734+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60c01090f0 con 0x7f60c0104c70 2024-05-22T23:57:29.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.736+0000 7f60b7e00640 1 -- 172.21.15.103:0/2790377353 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f60a8010ac0 con 0x7f60c0104c70 2024-05-22T23:57:29.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.737+0000 7f60b7e00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f60a40a2c10 0x7f60a40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:29.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.737+0000 7f60b7e00640 1 -- 172.21.15.103:0/2790377353 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7f60a80cee90 con 0x7f60c0104c70 2024-05-22T23:57:29.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.737+0000 7f60bea00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f60a40a2c10 0x7f60a40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:29.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.738+0000 7f60bea00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f60a40a2c10 0x7f60a40a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f60b002acb0 tx=0x7f60b0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:29.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.741+0000 7f60b7e00640 1 -- 172.21.15.103:0/2790377353 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f60a809bba0 con 0x7f60c0104c70 2024-05-22T23:57:29.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.959+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f60c0062470 con 0x7f60c0104c70 2024-05-22T23:57:29.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.960+0000 7f60b7e00640 1 -- 172.21.15.103:0/2790377353 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v83) ==== 84+0+65 (secure 0 0 0) 0x7f60a80ce090 con 0x7f60c0104c70 2024-05-22T23:57:29.963 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:29.963 INFO:teuthology.orchestra.run.smithi103.stdout:[ 2024-05-22T23:57:29.963 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-22T23:57:29.964 INFO:teuthology.orchestra.run.smithi103.stdout: "poolnum": 1, 2024-05-22T23:57:29.964 INFO:teuthology.orchestra.run.smithi103.stdout: "poolname": ".mgr" 2024-05-22T23:57:29.964 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-22T23:57:29.964 INFO:teuthology.orchestra.run.smithi103.stdout:] 2024-05-22T23:57:29.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f60a40a2c10 msgr2=0x7f60a40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f60a40a2c10 0x7f60a40a50d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f60b002acb0 tx=0x7f60b0005fd0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0104c70 msgr2=0x7f60c019a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:29.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0104c70 0x7f60c019a330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f60a800e9f0 tx=0x7f60a800eeb0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 shutdown_connections 2024-05-22T23:57:29.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f60a40a2c10 0x7f60a40a50d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60c01075a0 0x7f60c01a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60c0104c70 0x7f60c019a330 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 --2- 172.21.15.103:0/2790377353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60c0102340 0x7f60c0199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:29.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 >> 172.21.15.103:0/2790377353 conn(0x7f60c00fbd50 msgr2=0x7f60c0101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:29.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 shutdown_connections 2024-05-22T23:57:29.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:29.964+0000 7f60c5a00640 1 -- 172.21.15.103:0/2790377353 wait complete. 2024-05-22T23:57:29.985 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-22T23:57:29.986 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 -- 172.21.15.103:0/3358870628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154074270 msgr2=0x7ff1540746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 --2- 172.21.15.103:0/3358870628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154074270 0x7ff1540746d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff140009900 tx=0x7ff14002a480 comp rx=0 tx=0).stop 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 -- 172.21.15.103:0/3358870628 shutdown_connections 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 --2- 172.21.15.103:0/3358870628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff154109630 0x7ff15410bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 --2- 172.21.15.103:0/3358870628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154074270 0x7ff1540746d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 --2- 172.21.15.103:0/3358870628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff154075850 0x7ff154073ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 -- 172.21.15.103:0/3358870628 >> 172.21.15.103:0/3358870628 conn(0x7ff1540fcd10 msgr2=0x7ff1540ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 -- 172.21.15.103:0/3358870628 shutdown_connections 2024-05-22T23:57:30.168 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.164+0000 7ff159800640 1 -- 172.21.15.103:0/3358870628 wait complete. 2024-05-22T23:57:30.170 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.165+0000 7ff159800640 1 Processor -- start 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.165+0000 7ff159800640 1 -- start start 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff159800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff154074270 0x7ff154199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff159800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154075850 0x7ff15419a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff159800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff154109630 0x7ff1541a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff159800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff15410f670 con 0x7ff154109630 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff159800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff15410f4f0 con 0x7ff154074270 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff159800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff15410f7f0 con 0x7ff154075850 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff158e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff154109630 0x7ff1541a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff153400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154075850 0x7ff15419a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:30.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff153400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154075850 0x7ff15419a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:51028/0 (socket says 172.21.15.103:51028) 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff153400640 1 -- 172.21.15.103:0/1749249188 learned_addr learned my addr 172.21.15.103:0/1749249188 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.166+0000 7ff153e00640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff154074270 0x7ff154199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff153400640 1 -- 172.21.15.103:0/1749249188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff154074270 msgr2=0x7ff154199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff153400640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff154074270 0x7ff154199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff153400640 1 -- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff154109630 msgr2=0x7ff1541a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff153400640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff154109630 0x7ff1541a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff153400640 1 -- 172.21.15.103:0/1749249188 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1541a18e0 con 0x7ff154075850 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff153400640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154075850 0x7ff15419a1e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff140031440 tx=0x7ff140032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff150c00640 1 -- 172.21.15.103:0/1749249188 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14003f030 con 0x7ff154075850 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff150c00640 1 -- 172.21.15.103:0/1749249188 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff140003ce0 con 0x7ff154075850 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1541a1b70 con 0x7ff154075850 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff150c00640 1 -- 172.21.15.103:0/1749249188 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff140035070 con 0x7ff154075850 2024-05-22T23:57:30.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.167+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1541a20b0 con 0x7ff154075850 2024-05-22T23:57:30.173 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.169+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1541083f0 con 0x7ff154075850 2024-05-22T23:57:30.173 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.170+0000 7ff150c00640 1 -- 172.21.15.103:0/1749249188 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff140042030 con 0x7ff154075850 2024-05-22T23:57:30.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.173+0000 7ff150c00640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff1380a2c10 0x7ff1380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:30.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.173+0000 7ff150c00640 1 -- 172.21.15.103:0/1749249188 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7127+0+0 (secure 0 0 0) 0x7ff1400ef0d0 con 0x7ff154075850 2024-05-22T23:57:30.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.173+0000 7ff153e00640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff1380a2c10 0x7ff1380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:30.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.173+0000 7ff153e00640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff1380a2c10 0x7ff1380a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff15419ab70 tx=0x7ff144005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:30.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.176+0000 7ff150c00640 1 -- 172.21.15.103:0/1749249188 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff1400bbde0 con 0x7ff154075850 2024-05-22T23:57:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:30 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2559540595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:30 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2790377353' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-22T23:57:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:30 smithi103 ceph-mon[31370]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:30.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:30.385+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff154062470 con 0x7ff154075850 2024-05-22T23:57:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:30 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2559540595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:30 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2790377353' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-22T23:57:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:30 smithi105 ceph-mon[36426]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:30.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:30 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2559540595' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:30.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:30 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2790377353' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-22T23:57:30.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:30.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:30 smithi174 ceph-mon[36179]: pgmap v313: 1 pgs: 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:31.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.296+0000 7ff150c00640 1 -- 172.21.15.103:0/1749249188 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v84) ==== 138+0+0 (secure 0 0 0) 0x7ff1400c00e0 con 0x7ff154075850 2024-05-22T23:57:31.300 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'cephfs_metadata' created 2024-05-22T23:57:31.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.301+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff1380a2c10 msgr2=0x7ff1380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:31.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.301+0000 7ff159800640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff1380a2c10 0x7ff1380a50d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff15419ab70 tx=0x7ff144005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:31.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.301+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154075850 msgr2=0x7ff15419a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:31.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.301+0000 7ff159800640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154075850 0x7ff15419a1e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff140031440 tx=0x7ff140032ce0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.302+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 shutdown_connections 2024-05-22T23:57:31.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.302+0000 7ff159800640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff1380a2c10 0x7ff1380a50d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.302+0000 7ff159800640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff154109630 0x7ff1541a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.302+0000 7ff159800640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff154075850 0x7ff15419a1e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.302+0000 7ff159800640 1 --2- 172.21.15.103:0/1749249188 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff154074270 0x7ff154199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.302+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 >> 172.21.15.103:0/1749249188 conn(0x7ff1540fcd10 msgr2=0x7ff1541055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:31.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.302+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 shutdown_connections 2024-05-22T23:57:31.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.303+0000 7ff159800640 1 -- 172.21.15.103:0/1749249188 wait complete. 2024-05-22T23:57:31.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-22T23:57:31.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 -- 172.21.15.103:0/72936660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c102390 msgr2=0x7fe16c104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:31.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 --2- 172.21.15.103:0/72936660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c102390 0x7fe16c104780 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe15c009900 tx=0x7fe15c02a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 -- 172.21.15.103:0/72936660 shutdown_connections 2024-05-22T23:57:31.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 --2- 172.21.15.103:0/72936660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe16c1075f0 0x7fe16c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 --2- 172.21.15.103:0/72936660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe16c104cc0 0x7fe16c1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 --2- 172.21.15.103:0/72936660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c102390 0x7fe16c104780 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 -- 172.21.15.103:0/72936660 >> 172.21.15.103:0/72936660 conn(0x7fe16c0fbd20 msgr2=0x7fe16c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:31.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 -- 172.21.15.103:0/72936660 shutdown_connections 2024-05-22T23:57:31.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.512+0000 7fe173200640 1 -- 172.21.15.103:0/72936660 wait complete. 2024-05-22T23:57:31.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.513+0000 7fe173200640 1 Processor -- start 2024-05-22T23:57:31.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.513+0000 7fe173200640 1 -- start start 2024-05-22T23:57:31.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.513+0000 7fe173200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe16c102390 0x7fe16c199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:31.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.513+0000 7fe173200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c104cc0 0x7fe16c19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:31.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.513+0000 7fe173200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe16c1075f0 0x7fe16c1a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:31.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.513+0000 7fe173200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe16c10d570 con 0x7fe16c1075f0 2024-05-22T23:57:31.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.513+0000 7fe173200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe16c10d3f0 con 0x7fe16c102390 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.513+0000 7fe173200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe16c10d6f0 con 0x7fe16c104cc0 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c104cc0 0x7fe16c19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c104cc0 0x7fe16c19a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:51036/0 (socket says 172.21.15.103:51036) 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 -- 172.21.15.103:0/1982291416 learned_addr learned my addr 172.21.15.103:0/1982291416 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe16be00640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe16c102390 0x7fe16c199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe171200640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe16c1075f0 0x7fe16c1a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 -- 172.21.15.103:0/1982291416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe16c102390 msgr2=0x7fe16c199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe16c102390 0x7fe16c199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 -- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe16c1075f0 msgr2=0x7fe16c1a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe16c1075f0 0x7fe16c1a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 -- 172.21.15.103:0/1982291416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16c1a1a40 con 0x7fe16c104cc0 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe171200640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe16c1075f0 0x7fe16c1a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:57:31.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.514+0000 7fe163e00640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c104cc0 0x7fe16c19a360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe15400e9a0 tx=0x7fe15400ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:31.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.515+0000 7fe169600640 1 -- 172.21.15.103:0/1982291416 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe15400cd80 con 0x7fe16c104cc0 2024-05-22T23:57:31.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.515+0000 7fe169600640 1 -- 172.21.15.103:0/1982291416 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1540035f0 con 0x7fe16c104cc0 2024-05-22T23:57:31.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.515+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe16c1a1d30 con 0x7fe16c104cc0 2024-05-22T23:57:31.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.515+0000 7fe169600640 1 -- 172.21.15.103:0/1982291416 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe154010670 con 0x7fe16c104cc0 2024-05-22T23:57:31.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.515+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe16c1a2270 con 0x7fe16c104cc0 2024-05-22T23:57:31.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.516+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe16c1090e0 con 0x7fe16c104cc0 2024-05-22T23:57:31.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.518+0000 7fe169600640 1 -- 172.21.15.103:0/1982291416 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fe1540037a0 con 0x7fe16c104cc0 2024-05-22T23:57:31.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.520+0000 7fe169600640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe1500a2c10 0x7fe1500a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:31.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.520+0000 7fe169600640 1 -- 172.21.15.103:0/1982291416 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7497+0+0 (secure 0 0 0) 0x7fe1540cefd0 con 0x7fe16c104cc0 2024-05-22T23:57:31.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.520+0000 7fe16be00640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe1500a2c10 0x7fe1500a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:31.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.521+0000 7fe16be00640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe1500a2c10 0x7fe1500a50d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe15c031440 tx=0x7fe15c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:31.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.522+0000 7fe169600640 1 -- 172.21.15.103:0/1982291416 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe15409c1c0 con 0x7fe16c104cc0 2024-05-22T23:57:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:57:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:31 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1749249188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-22T23:57:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:31 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-22T23:57:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:57:31.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:31.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:57:31.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:31 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1749249188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-22T23:57:31.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:31 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-22T23:57:31.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:57:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:57:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:31 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1749249188' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-22T23:57:31.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:31 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-22T23:57:31.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:57:31.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:31.736+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fe16c02d200 con 0x7fe16c104cc0 2024-05-22T23:57:32.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.303+0000 7fe169600640 1 -- 172.21.15.103:0/1982291416 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v85) ==== 171+0+0 (secure 0 0 0) 0x7fe15409bac0 con 0x7fe16c104cc0 2024-05-22T23:57:32.306 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'cephfs_data' created 2024-05-22T23:57:32.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.312+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe1500a2c10 msgr2=0x7fe1500a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:32.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.312+0000 7fe173200640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe1500a2c10 0x7fe1500a50d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe15c031440 tx=0x7fe15c0315b0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.312+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c104cc0 msgr2=0x7fe16c19a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:32.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.312+0000 7fe173200640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c104cc0 0x7fe16c19a360 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe15400e9a0 tx=0x7fe15400ee60 comp rx=0 tx=0).stop 2024-05-22T23:57:32.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.313+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 shutdown_connections 2024-05-22T23:57:32.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.313+0000 7fe173200640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe1500a2c10 0x7fe1500a50d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.313+0000 7fe173200640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe16c1075f0 0x7fe16c1a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.313+0000 7fe173200640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe16c104cc0 0x7fe16c19a360 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.313+0000 7fe173200640 1 --2- 172.21.15.103:0/1982291416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe16c102390 0x7fe16c199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.313+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 >> 172.21.15.103:0/1982291416 conn(0x7fe16c0fbd20 msgr2=0x7fe16c101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:32.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.314+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 shutdown_connections 2024-05-22T23:57:32.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.314+0000 7fe173200640 1 -- 172.21.15.103:0/1982291416 wait complete. 2024-05-22T23:57:32.348 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 -- 172.21.15.103:0/1810500285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6001018c0 msgr2=0x7fd600101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 --2- 172.21.15.103:0/1810500285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6001018c0 0x7fd600101ce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd5f4009900 tx=0x7fd5f402a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 -- 172.21.15.103:0/1810500285 shutdown_connections 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 --2- 172.21.15.103:0/1810500285 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd600103470 0x7fd600109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 --2- 172.21.15.103:0/1810500285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd600102ab0 0x7fd600102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 --2- 172.21.15.103:0/1810500285 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6001018c0 0x7fd600101ce0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 -- 172.21.15.103:0/1810500285 >> 172.21.15.103:0/1810500285 conn(0x7fd6000fce50 msgr2=0x7fd6000ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 -- 172.21.15.103:0/1810500285 shutdown_connections 2024-05-22T23:57:32.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.516+0000 7fd608400640 1 -- 172.21.15.103:0/1810500285 wait complete. 2024-05-22T23:57:32.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.517+0000 7fd608400640 1 Processor -- start 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd608400640 1 -- start start 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd608400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6001018c0 0x7fd600197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd608400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd600102ab0 0x7fd600198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd608400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd600103470 0x7fd60019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd608400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010d560 con 0x7fd6001018c0 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd608400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010d3e0 con 0x7fd600103470 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd608400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd60010d6e0 con 0x7fd600102ab0 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd605000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd600102ab0 0x7fd600198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd606400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd600103470 0x7fd60019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd605000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd600102ab0 0x7fd600198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:51056/0 (socket says 172.21.15.103:51056) 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd606400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd600103470 0x7fd60019f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:60628/0 (socket says 172.21.15.103:60628) 2024-05-22T23:57:32.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd605000640 1 -- 172.21.15.103:0/1916386988 learned_addr learned my addr 172.21.15.103:0/1916386988 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.518+0000 7fd605a00640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6001018c0 0x7fd600197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd605000640 1 -- 172.21.15.103:0/1916386988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd600103470 msgr2=0x7fd60019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd605000640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd600103470 0x7fd60019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd605000640 1 -- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6001018c0 msgr2=0x7fd600197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd605000640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6001018c0 0x7fd600197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd605000640 1 -- 172.21.15.103:0/1916386988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd60019f810 con 0x7fd600102ab0 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd605000640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd600102ab0 0x7fd600198130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00e9f0 tx=0x7fd5ec00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd5fe000640 1 -- 172.21.15.103:0/1916386988 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec00cd80 con 0x7fd600102ab0 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd600073b00 con 0x7fd600102ab0 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd5fe000640 1 -- 172.21.15.103:0/1916386988 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5ec010910 con 0x7fd600102ab0 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd5fe000640 1 -- 172.21.15.103:0/1916386988 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec018990 con 0x7fd600102ab0 2024-05-22T23:57:32.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.519+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd600074050 con 0x7fd600102ab0 2024-05-22T23:57:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.520+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd600108c80 con 0x7fd600102ab0 2024-05-22T23:57:32.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.523+0000 7fd5fe000640 1 -- 172.21.15.103:0/1916386988 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd5ec010ac0 con 0x7fd600102ab0 2024-05-22T23:57:32.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.524+0000 7fd5fe000640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5e80a2b90 0x7fd5e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:32.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.524+0000 7fd5fe000640 1 -- 172.21.15.103:0/1916386988 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7879+0+0 (secure 0 0 0) 0x7fd5ec0cf240 con 0x7fd600102ab0 2024-05-22T23:57:32.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.524+0000 7fd605a00640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5e80a2b90 0x7fd5e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:32.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.525+0000 7fd605a00640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5e80a2b90 0x7fd5e80a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd5f402a9e0 tx=0x7fd5f40315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:32.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.530+0000 7fd5fe000640 1 -- 172.21.15.103:0/1916386988 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd5ec09bc60 con 0x7fd600102ab0 2024-05-22T23:57:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:32 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-22T23:57:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:32 smithi105 ceph-mon[36426]: osdmap e84: 12 total, 12 up, 12 in 2024-05-22T23:57:32.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:32 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1982291416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-22T23:57:32.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:32 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-22T23:57:32.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:32 smithi105 ceph-mon[36426]: pgmap v315: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:32.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:32 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-22T23:57:32.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:32 smithi103 ceph-mon[31370]: osdmap e84: 12 total, 12 up, 12 in 2024-05-22T23:57:32.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:32 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1982291416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-22T23:57:32.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:32 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-22T23:57:32.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:32 smithi103 ceph-mon[31370]: pgmap v315: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:32.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:32 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-22T23:57:32.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:32 smithi174 ceph-mon[36179]: osdmap e84: 12 total, 12 up, 12 in 2024-05-22T23:57:32.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:32 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1982291416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-22T23:57:32.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:32 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-22T23:57:32.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:32 smithi174 ceph-mon[36179]: pgmap v315: 33 pgs: 9 creating+peering, 23 unknown, 1 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:32.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:32.802+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd60002d200 con 0x7fd600102ab0 2024-05-22T23:57:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-22T23:57:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-22T23:57:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: osdmap e85: 12 total, 12 up, 12 in 2024-05-22T23:57:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1916386988' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-22T23:57:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-22T23:57:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:33 smithi105 ceph-mon[36426]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-22T23:57:33.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-22T23:57:33.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-22T23:57:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: osdmap e85: 12 total, 12 up, 12 in 2024-05-22T23:57:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1916386988' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-22T23:57:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-22T23:57:33.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:33 smithi103 ceph-mon[31370]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-22T23:57:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-22T23:57:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-22T23:57:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: osdmap e85: 12 total, 12 up, 12 in 2024-05-22T23:57:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1916386988' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-22T23:57:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-22T23:57:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:33 smithi174 ceph-mon[36179]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-22T23:57:33.715 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.712+0000 7fd5fe000640 1 -- 172.21.15.103:0/1916386988 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fd5ec09ff60 con 0x7fd600102ab0 2024-05-22T23:57:33.715 INFO:teuthology.orchestra.run.smithi103.stderr:new fs with metadata pool 2 and data pool 3 2024-05-22T23:57:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.715+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5e80a2b90 msgr2=0x7fd5e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.715+0000 7fd608400640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5e80a2b90 0x7fd5e80a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd5f402a9e0 tx=0x7fd5f40315b0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.715+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd600102ab0 msgr2=0x7fd600198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.715+0000 7fd608400640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd600102ab0 0x7fd600198130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00e9f0 tx=0x7fd5ec00eeb0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.716+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 shutdown_connections 2024-05-22T23:57:33.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.716+0000 7fd608400640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5e80a2b90 0x7fd5e80a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.716+0000 7fd608400640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd600103470 0x7fd60019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.716+0000 7fd608400640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd600102ab0 0x7fd600198130 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.716+0000 7fd608400640 1 --2- 172.21.15.103:0/1916386988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6001018c0 0x7fd600197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.716+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 >> 172.21.15.103:0/1916386988 conn(0x7fd6000fce50 msgr2=0x7fd600105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:33.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.717+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 shutdown_connections 2024-05-22T23:57:33.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.717+0000 7fd608400640 1 -- 172.21.15.103:0/1916386988 wait complete. 2024-05-22T23:57:33.742 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-22T23:57:33.742 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-22T23:57:33.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 -- 172.21.15.103:0/959619194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388109960 msgr2=0x7f83881101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:33.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 --2- 172.21.15.103:0/959619194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388109960 0x7f83881101f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8380009be0 tx=0x7f838002a6c0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 -- 172.21.15.103:0/959619194 shutdown_connections 2024-05-22T23:57:33.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 --2- 172.21.15.103:0/959619194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388109960 0x7f83881101f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 --2- 172.21.15.103:0/959619194 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8388108fa0 0x7f8388109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 --2- 172.21.15.103:0/959619194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388107db0 0x7f83881081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 -- 172.21.15.103:0/959619194 >> 172.21.15.103:0/959619194 conn(0x7f8388076d40 msgr2=0x7f8388079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:33.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 -- 172.21.15.103:0/959619194 shutdown_connections 2024-05-22T23:57:33.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.921+0000 7f838ce00640 1 -- 172.21.15.103:0/959619194 wait complete. 2024-05-22T23:57:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.922+0000 7f838ce00640 1 Processor -- start 2024-05-22T23:57:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.922+0000 7f838ce00640 1 -- start start 2024-05-22T23:57:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f838ce00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388107db0 0x7f838819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f838ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8388108fa0 0x7f838819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f838ce00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388109960 0x7f83881a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f838ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f83881139e0 con 0x7f8388108fa0 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f838ce00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8388113860 con 0x7f8388107db0 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f838ce00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8388113b60 con 0x7f8388109960 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388107db0 0x7f838819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388107db0 0x7f838819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:60648/0 (socket says 172.21.15.103:60648) 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387400640 1 -- 172.21.15.103:0/3101049956 learned_addr learned my addr 172.21.15.103:0/3101049956 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8386a00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8388108fa0 0x7f838819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387e00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388109960 0x7f83881a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387400640 1 -- 172.21.15.103:0/3101049956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388109960 msgr2=0x7f83881a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387400640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388109960 0x7f83881a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387400640 1 -- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8388108fa0 msgr2=0x7f838819e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387400640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8388108fa0 0x7f838819e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.923+0000 7f8387400640 1 -- 172.21.15.103:0/3101049956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83881a5c20 con 0x7f8388107db0 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.924+0000 7f8386a00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8388108fa0 0x7f838819e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:57:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.924+0000 7f8387400640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388107db0 0x7f838819e000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f837800e9c0 tx=0x7f837800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:33.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.924+0000 7f837fe00640 1 -- 172.21.15.103:0/3101049956 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837800cd80 con 0x7f8388107db0 2024-05-22T23:57:33.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.924+0000 7f837fe00640 1 -- 172.21.15.103:0/3101049956 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83780035f0 con 0x7f8388107db0 2024-05-22T23:57:33.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.924+0000 7f837fe00640 1 -- 172.21.15.103:0/3101049956 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378010690 con 0x7f8388107db0 2024-05-22T23:57:33.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.924+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83881a5f10 con 0x7f8388107db0 2024-05-22T23:57:33.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.924+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83881a6320 con 0x7f8388107db0 2024-05-22T23:57:33.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.925+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8388068a00 con 0x7f8388107db0 2024-05-22T23:57:33.930 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.927+0000 7f837fe00640 1 -- 172.21.15.103:0/3101049956 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f83780037a0 con 0x7f8388107db0 2024-05-22T23:57:33.931 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.929+0000 7f837fe00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f836c0a2b90 0x7f836c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:33.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.929+0000 7f837fe00640 1 -- 172.21.15.103:0/3101049956 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f83780cf3d0 con 0x7f8388107db0 2024-05-22T23:57:33.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.929+0000 7f8386a00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f836c0a2b90 0x7f836c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:33.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.930+0000 7f8386a00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f836c0a2b90 0x7f836c0a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f838819f510 tx=0x7f8374009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:33.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:33.931+0000 7f837fe00640 1 -- 172.21.15.103:0/3101049956 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f837809c3e0 con 0x7f8388107db0 2024-05-22T23:57:34.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.140+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f838810f120 con 0x7f8388107db0 2024-05-22T23:57:34.325 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.321+0000 7f837fe00640 1 -- 172.21.15.103:0/3101049956 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v87) ==== 157+0+0 (secure 0 0 0) 0x7f838810f120 con 0x7f8388107db0 2024-05-22T23:57:34.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f836c0a2b90 msgr2=0x7f836c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:34.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f836c0a2b90 0x7f836c0a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f838819f510 tx=0x7f8374009260 comp rx=0 tx=0).stop 2024-05-22T23:57:34.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388107db0 msgr2=0x7f838819e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:34.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388107db0 0x7f838819e000 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f837800e9c0 tx=0x7f837800ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:34.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 shutdown_connections 2024-05-22T23:57:34.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f836c0a2b90 0x7f836c0a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388109960 0x7f83881a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8388108fa0 0x7f838819e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 --2- 172.21.15.103:0/3101049956 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388107db0 0x7f838819e000 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 >> 172.21.15.103:0/3101049956 conn(0x7f8388076d40 msgr2=0x7f838810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:34.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 shutdown_connections 2024-05-22T23:57:34.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.325+0000 7f838ce00640 1 -- 172.21.15.103:0/3101049956 wait complete. 2024-05-22T23:57:34.353 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-22T23:57:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: osdmap e86: 12 total, 12 up, 12 in 2024-05-22T23:57:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:57:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-22T23:57:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-22T23:57:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-22T23:57:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: pgmap v318: 97 pgs: 17 creating+peering, 61 unknown, 19 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3101049956' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-22T23:57:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:34 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-22T23:57:34.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 -- 172.21.15.103:0/2084830867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c1032c0 msgr2=0x7f184c109b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:34.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 --2- 172.21.15.103:0/2084830867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c1032c0 0x7f184c109b50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1844009ca0 tx=0x7f184402a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 -- 172.21.15.103:0/2084830867 shutdown_connections 2024-05-22T23:57:34.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 --2- 172.21.15.103:0/2084830867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c1032c0 0x7f184c109b50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 --2- 172.21.15.103:0/2084830867 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f184c102900 0x7f184c102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 --2- 172.21.15.103:0/2084830867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f184c101710 0x7f184c101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 -- 172.21.15.103:0/2084830867 >> 172.21.15.103:0/2084830867 conn(0x7f184c0fcc80 msgr2=0x7f184c0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:34.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 -- 172.21.15.103:0/2084830867 shutdown_connections 2024-05-22T23:57:34.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.560+0000 7f184be00640 1 -- 172.21.15.103:0/2084830867 wait complete. 2024-05-22T23:57:34.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.561+0000 7f184be00640 1 Processor -- start 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184be00640 1 -- start start 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f184c101710 0x7f184c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c102900 0x7f184c197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f184c1032c0 0x7f184c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f184c10d390 con 0x7f184c102900 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f184c10d210 con 0x7f184c1032c0 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184be00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f184c10d510 con 0x7f184c101710 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f1842000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c102900 0x7f184c197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f1842000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c102900 0x7f184c197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47982/0 (socket says 172.21.15.103:47982) 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f1842000640 1 -- 172.21.15.103:0/3061521935 learned_addr learned my addr 172.21.15.103:0/3061521935 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184b400640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f184c1032c0 0x7f184c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:34.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.562+0000 7f184aa00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f184c101710 0x7f184c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:34.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1842000640 1 -- 172.21.15.103:0/3061521935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f184c101710 msgr2=0x7f184c1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:34.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1842000640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f184c101710 0x7f184c1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1842000640 1 -- 172.21.15.103:0/3061521935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f184c1032c0 msgr2=0x7f184c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:34.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1842000640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f184c1032c0 0x7f184c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:34.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1842000640 1 -- 172.21.15.103:0/3061521935 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f184c19f5c0 con 0x7f184c102900 2024-05-22T23:57:34.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1842000640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c102900 0x7f184c197ee0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f183400ea60 tx=0x7f183400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:34.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1850a00640 1 -- 172.21.15.103:0/3061521935 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f183400cde0 con 0x7f184c102900 2024-05-22T23:57:34.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1850a00640 1 -- 172.21.15.103:0/3061521935 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18340035f0 con 0x7f184c102900 2024-05-22T23:57:34.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f184c19f8b0 con 0x7f184c102900 2024-05-22T23:57:34.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.563+0000 7f1850a00640 1 -- 172.21.15.103:0/3061521935 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1834010670 con 0x7f184c102900 2024-05-22T23:57:34.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.564+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f184c19fe00 con 0x7f184c102900 2024-05-22T23:57:34.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.565+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f184c0685b0 con 0x7f184c102900 2024-05-22T23:57:34.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.567+0000 7f1850a00640 1 -- 172.21.15.103:0/3061521935 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f18340037a0 con 0x7f184c102900 2024-05-22T23:57:34.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.569+0000 7f1850a00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18300a2b90 0x7f18300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:34.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.570+0000 7f1850a00640 1 -- 172.21.15.103:0/3061521935 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8277+0+0 (secure 0 0 0) 0x7f18340cf380 con 0x7f184c102900 2024-05-22T23:57:34.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.570+0000 7f184aa00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18300a2b90 0x7f18300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:34.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.570+0000 7f184aa00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18300a2b90 0x7f18300a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f184c102760 tx=0x7f183c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:34.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.575+0000 7f1850a00640 1 -- 172.21.15.103:0/3061521935 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f183409c250 con 0x7f184c102900 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: osdmap e86: 12 total, 12 up, 12 in 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-22T23:57:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: pgmap v318: 97 pgs: 17 creating+peering, 61 unknown, 19 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:34.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3101049956' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-22T23:57:34.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:34 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-22T23:57:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: osdmap e86: 12 total, 12 up, 12 in 2024-05-22T23:57:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:57:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:57:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:57:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-22T23:57:34.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-22T23:57:34.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-22T23:57:34.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: pgmap v318: 97 pgs: 17 creating+peering, 61 unknown, 19 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:34.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3101049956' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-22T23:57:34.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:34 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-22T23:57:34.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:34.775+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f184c108a30 con 0x7f184c102900 2024-05-22T23:57:35.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:35 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-22T23:57:35.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:35 smithi103 ceph-mon[31370]: osdmap e87: 12 total, 12 up, 12 in 2024-05-22T23:57:35.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:35 smithi103 ceph-mon[31370]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-22T23:57:35.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:35 smithi103 ceph-mon[31370]: Cluster is now healthy 2024-05-22T23:57:35.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:35 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] up:active 2024-05-22T23:57:35.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:35 smithi103 ceph-mon[31370]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:35.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:35 smithi103 ceph-mon[31370]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:35.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:35 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3061521935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-22T23:57:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:35 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-22T23:57:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:35 smithi174 ceph-mon[36179]: osdmap e87: 12 total, 12 up, 12 in 2024-05-22T23:57:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:35 smithi174 ceph-mon[36179]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-22T23:57:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:35 smithi174 ceph-mon[36179]: Cluster is now healthy 2024-05-22T23:57:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:35 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] up:active 2024-05-22T23:57:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:35 smithi174 ceph-mon[36179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:35 smithi174 ceph-mon[36179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:35.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:35 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3061521935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-22T23:57:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:35 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-22T23:57:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:35 smithi105 ceph-mon[36426]: osdmap e87: 12 total, 12 up, 12 in 2024-05-22T23:57:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:35 smithi105 ceph-mon[36426]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-22T23:57:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:35 smithi105 ceph-mon[36426]: Cluster is now healthy 2024-05-22T23:57:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:35 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] up:active 2024-05-22T23:57:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:35 smithi105 ceph-mon[36426]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:35 smithi105 ceph-mon[36426]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:35.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:35 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3061521935' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-22T23:57:36.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.751+0000 7f1850a00640 1 -- 172.21.15.103:0/3061521935 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v89) ==== 228+0+0 (secure 0 0 0) 0x7f183409bb50 con 0x7f184c102900 2024-05-22T23:57:36.755 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'cephfs_data_ec' created 2024-05-22T23:57:36.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.756+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18300a2b90 msgr2=0x7f18300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:36.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.756+0000 7f184be00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18300a2b90 0x7f18300a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f184c102760 tx=0x7f183c005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:36.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.756+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c102900 msgr2=0x7f184c197ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:36.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.756+0000 7f184be00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c102900 0x7f184c197ee0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f183400ea60 tx=0x7f183400ef20 comp rx=0 tx=0).stop 2024-05-22T23:57:36.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.757+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 shutdown_connections 2024-05-22T23:57:36.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.757+0000 7f184be00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18300a2b90 0x7f18300a5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.757+0000 7f184be00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f184c1032c0 0x7f184c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.757+0000 7f184be00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f184c102900 0x7f184c197ee0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.757+0000 7f184be00640 1 --2- 172.21.15.103:0/3061521935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f184c101710 0x7f184c1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.757+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 >> 172.21.15.103:0/3061521935 conn(0x7f184c0fcc80 msgr2=0x7f184c105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:36.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.757+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 shutdown_connections 2024-05-22T23:57:36.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.758+0000 7f184be00640 1 -- 172.21.15.103:0/3061521935 wait complete. 2024-05-22T23:57:36.789 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 -- 172.21.15.103:0/2632885812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290102ab0 msgr2=0x7f0290102f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 --2- 172.21.15.103:0/2632885812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290102ab0 0x7f0290102f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0280009ca0 tx=0x7f028002a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 -- 172.21.15.103:0/2632885812 shutdown_connections 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 --2- 172.21.15.103:0/2632885812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0290103470 0x7f0290109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 --2- 172.21.15.103:0/2632885812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0290102ab0 0x7f0290102f30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 --2- 172.21.15.103:0/2632885812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f02901018c0 0x7f0290101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 -- 172.21.15.103:0/2632885812 >> 172.21.15.103:0/2632885812 conn(0x7f02900fce50 msgr2=0x7f02900ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 -- 172.21.15.103:0/2632885812 shutdown_connections 2024-05-22T23:57:36.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.969+0000 7f0298200640 1 -- 172.21.15.103:0/2632885812 wait complete. 2024-05-22T23:57:36.975 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.970+0000 7f0298200640 1 Processor -- start 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0298200640 1 -- start start 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0298200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02901018c0 0x7f0290199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0298200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0290102ab0 0x7f029019a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0298200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0290103470 0x7f02901a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0298200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f029010d520 con 0x7f0290103470 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0298200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f029010d3a0 con 0x7f02901018c0 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0298200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f029010d6a0 con 0x7f0290102ab0 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0296200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0290103470 0x7f02901a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0296200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0290103470 0x7f02901a1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48012/0 (socket says 172.21.15.103:48012) 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0296200640 1 -- 172.21.15.103:0/2758083076 learned_addr learned my addr 172.21.15.103:0/2758083076 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:36.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0295800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02901018c0 0x7f0290199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.971+0000 7f0294e00640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0290102ab0 0x7f029019a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f0296200640 1 -- 172.21.15.103:0/2758083076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0290102ab0 msgr2=0x7f029019a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f0296200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0290102ab0 0x7f029019a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f0296200640 1 -- 172.21.15.103:0/2758083076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02901018c0 msgr2=0x7f0290199e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f0296200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02901018c0 0x7f0290199e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f0296200640 1 -- 172.21.15.103:0/2758083076 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02901a1a90 con 0x7f0290103470 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f0295800640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02901018c0 0x7f0290199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f0296200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0290103470 0x7f02901a1420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f028800d950 tx=0x7f028800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f028e000640 1 -- 172.21.15.103:0/2758083076 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288017030 con 0x7f0290103470 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02901a1d80 con 0x7f0290103470 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f028e000640 1 -- 172.21.15.103:0/2758083076 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0288004020 con 0x7f0290103470 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.972+0000 7f028e000640 1 -- 172.21.15.103:0/2758083076 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0288003c00 con 0x7f0290103470 2024-05-22T23:57:36.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.973+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02901a2220 con 0x7f0290103470 2024-05-22T23:57:36.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.974+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02901a24b0 con 0x7f0290103470 2024-05-22T23:57:36.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.976+0000 7f028e000640 1 -- 172.21.15.103:0/2758083076 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f028801a070 con 0x7f0290103470 2024-05-22T23:57:36.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.977+0000 7f028e000640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f02780a2b90 0x7f02780a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:36.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.977+0000 7f028e000640 1 -- 172.21.15.103:0/2758083076 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8770+0+0 (secure 0 0 0) 0x7f02880cedd0 con 0x7f0290103470 2024-05-22T23:57:36.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.978+0000 7f0295800640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f02780a2b90 0x7f02780a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:36.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.978+0000 7f0295800640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f02780a2b90 0x7f02780a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0290102910 tx=0x7f0284005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:36.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:36.981+0000 7f028e000640 1 -- 172.21.15.103:0/2758083076 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0288021070 con 0x7f0290103470 2024-05-22T23:57:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:36 smithi105 ceph-mon[36426]: pgmap v321: 97 pgs: 17 creating+peering, 24 unknown, 56 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-05-22T23:57:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:36 smithi105 ceph-mon[36426]: osdmap e88: 12 total, 12 up, 12 in 2024-05-22T23:57:37.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:36 smithi103 ceph-mon[31370]: pgmap v321: 97 pgs: 17 creating+peering, 24 unknown, 56 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-05-22T23:57:37.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:36 smithi103 ceph-mon[31370]: osdmap e88: 12 total, 12 up, 12 in 2024-05-22T23:57:37.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:36 smithi174 ceph-mon[36179]: pgmap v321: 97 pgs: 17 creating+peering, 24 unknown, 56 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-05-22T23:57:37.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:36 smithi174 ceph-mon[36179]: osdmap e88: 12 total, 12 up, 12 in 2024-05-22T23:57:37.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.183+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f02901a2680 con 0x7f0290103470 2024-05-22T23:57:37.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.755+0000 7f028e000640 1 -- 172.21.15.103:0/2758083076 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v90) ==== 169+0+0 (secure 0 0 0) 0x7f028809bab0 con 0x7f0290103470 2024-05-22T23:57:37.759 INFO:teuthology.orchestra.run.smithi103.stderr:set pool 4 allow_ec_overwrites to true 2024-05-22T23:57:37.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.760+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f02780a2b90 msgr2=0x7f02780a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:37.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.760+0000 7f0298200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f02780a2b90 0x7f02780a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0290102910 tx=0x7f0284005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:37.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.760+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0290103470 msgr2=0x7f02901a1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:37.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.760+0000 7f0298200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0290103470 0x7f02901a1420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f028800d950 tx=0x7f028800de10 comp rx=0 tx=0).stop 2024-05-22T23:57:37.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.761+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 shutdown_connections 2024-05-22T23:57:37.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.761+0000 7f0298200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f02780a2b90 0x7f02780a5050 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0290102910 tx=0x7f0284005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:37.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.761+0000 7f0298200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0290103470 0x7f02901a1420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.761+0000 7f0298200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0290102ab0 0x7f029019a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.761+0000 7f0298200640 1 --2- 172.21.15.103:0/2758083076 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02901018c0 0x7f0290199e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.761+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 >> 172.21.15.103:0/2758083076 conn(0x7f02900fce50 msgr2=0x7f0290073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:37.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.764+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 shutdown_connections 2024-05-22T23:57:37.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.764+0000 7f0298200640 1 -- 172.21.15.103:0/2758083076 wait complete. 2024-05-22T23:57:37.803 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-22T23:57:37.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.979+0000 7f4657e00640 1 -- 172.21.15.103:0/3504892569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4658109960 msgr2=0x7f46581101f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:37.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.979+0000 7f4657e00640 1 --2- 172.21.15.103:0/3504892569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4658109960 0x7f46581101f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4650009ca0 tx=0x7f465002a8b0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.980+0000 7f4657e00640 1 -- 172.21.15.103:0/3504892569 shutdown_connections 2024-05-22T23:57:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.980+0000 7f4657e00640 1 --2- 172.21.15.103:0/3504892569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4658109960 0x7f46581101f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.980+0000 7f4657e00640 1 --2- 172.21.15.103:0/3504892569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4658108fa0 0x7f4658109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.980+0000 7f4657e00640 1 --2- 172.21.15.103:0/3504892569 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4658107db0 0x7f46581081d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.980+0000 7f4657e00640 1 -- 172.21.15.103:0/3504892569 >> 172.21.15.103:0/3504892569 conn(0x7f4658076d40 msgr2=0x7f4658079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.980+0000 7f4657e00640 1 -- 172.21.15.103:0/3504892569 shutdown_connections 2024-05-22T23:57:37.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.980+0000 7f4657e00640 1 -- 172.21.15.103:0/3504892569 wait complete. 2024-05-22T23:57:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657e00640 1 Processor -- start 2024-05-22T23:57:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657e00640 1 -- start start 2024-05-22T23:57:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4658107db0 0x7f465819e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4658108fa0 0x7f465819e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4658109960 0x7f46581a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4658113a30 con 0x7f4658109960 2024-05-22T23:57:37.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f46581138b0 con 0x7f4658108fa0 2024-05-22T23:57:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4658113bb0 con 0x7f4658107db0 2024-05-22T23:57:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4657400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4658109960 0x7f46581a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.981+0000 7f4656a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4658107db0 0x7f465819e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4657400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4658109960 0x7f46581a55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48036/0 (socket says 172.21.15.103:48036) 2024-05-22T23:57:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4657400640 1 -- 172.21.15.103:0/3417520008 learned_addr learned my addr 172.21.15.103:0/3417520008 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4656a00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4658107db0 0x7f465819e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59710/0 (socket says 172.21.15.103:59710) 2024-05-22T23:57:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f464e000640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4658108fa0 0x7f465819e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:37.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4656a00640 1 -- 172.21.15.103:0/3417520008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4658108fa0 msgr2=0x7f465819e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4656a00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4658108fa0 0x7f465819e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4656a00640 1 -- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4658109960 msgr2=0x7f46581a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4656a00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4658109960 0x7f46581a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4656a00640 1 -- 172.21.15.103:0/3417520008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46581a5c40 con 0x7f4658107db0 2024-05-22T23:57:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f4656a00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4658107db0 0x7f465819e020 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f464800e9c0 tx=0x7f464800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.982+0000 7f464fe00640 1 -- 172.21.15.103:0/3417520008 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464800ccc0 con 0x7f4658107db0 2024-05-22T23:57:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.983+0000 7f464fe00640 1 -- 172.21.15.103:0/3417520008 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46480035f0 con 0x7f4658107db0 2024-05-22T23:57:37.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.983+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46581a5f30 con 0x7f4658107db0 2024-05-22T23:57:37.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.983+0000 7f464fe00640 1 -- 172.21.15.103:0/3417520008 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464800ce60 con 0x7f4658107db0 2024-05-22T23:57:37.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.983+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46581a6390 con 0x7f4658107db0 2024-05-22T23:57:37.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.984+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4658068a00 con 0x7f4658107db0 2024-05-22T23:57:37.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.986+0000 7f464fe00640 1 -- 172.21.15.103:0/3417520008 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f46480037a0 con 0x7f4658107db0 2024-05-22T23:57:37.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.987+0000 7f464fe00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f463c0a2b90 0x7f463c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:37.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.987+0000 7f464fe00640 1 -- 172.21.15.103:0/3417520008 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8770+0+0 (secure 0 0 0) 0x7f46480cf650 con 0x7f4658107db0 2024-05-22T23:57:37.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.987+0000 7f464e000640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f463c0a2b90 0x7f463c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:37.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.988+0000 7f464e000640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f463c0a2b90 0x7f463c0a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f465819f530 tx=0x7f4640005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:37.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:37.991+0000 7f464fe00640 1 -- 172.21.15.103:0/3417520008 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f464809c330 con 0x7f4658107db0 2024-05-22T23:57:38.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:37 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3061521935' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-22T23:57:38.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:37 smithi105 ceph-mon[36426]: osdmap e89: 12 total, 12 up, 12 in 2024-05-22T23:57:38.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:37 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2758083076' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-22T23:57:38.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:37 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3061521935' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-22T23:57:38.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:37 smithi103 ceph-mon[31370]: osdmap e89: 12 total, 12 up, 12 in 2024-05-22T23:57:38.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:37 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2758083076' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-22T23:57:38.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:37 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3061521935' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-22T23:57:38.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:37 smithi174 ceph-mon[36179]: osdmap e89: 12 total, 12 up, 12 in 2024-05-22T23:57:38.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:37 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2758083076' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-22T23:57:38.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.241+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f46580624a0 con 0x7f4658107db0 2024-05-22T23:57:38.549 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:57:38 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b[66976]: 2024-05-22T23:57:38.160+0000 7f633d485700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-22T23:57:38.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.790+0000 7f464fe00640 1 -- 172.21.15.103:0/3417520008 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f464809bc30 con 0x7f4658107db0 2024-05-22T23:57:38.794 INFO:teuthology.orchestra.run.smithi103.stderr:added data pool 4 to fsmap 2024-05-22T23:57:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.794+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f463c0a2b90 msgr2=0x7f463c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.794+0000 7f4657e00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f463c0a2b90 0x7f463c0a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f465819f530 tx=0x7f4640005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4658107db0 msgr2=0x7f465819e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4658107db0 0x7f465819e020 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f464800e9c0 tx=0x7f464800ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 shutdown_connections 2024-05-22T23:57:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f463c0a2b90 0x7f463c0a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4658109960 0x7f46581a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:38.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4658108fa0 0x7f465819e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:38.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 --2- 172.21.15.103:0/3417520008 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4658107db0 0x7f465819e020 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:38.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 >> 172.21.15.103:0/3417520008 conn(0x7f4658076d40 msgr2=0x7f465810c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:38.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 shutdown_connections 2024-05-22T23:57:38.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:38.795+0000 7f4657e00640 1 -- 172.21.15.103:0/3417520008 wait complete. 2024-05-22T23:57:38.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:38 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2758083076' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-22T23:57:38.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:38 smithi105 ceph-mon[36426]: osdmap e90: 12 total, 12 up, 12 in 2024-05-22T23:57:38.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:38 smithi105 ceph-mon[36426]: pgmap v324: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-05-22T23:57:38.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:38 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3417520008' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-22T23:57:38.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:38 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-22T23:57:38.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:38 smithi105 ceph-mon[36426]: osdmap e91: 12 total, 12 up, 12 in 2024-05-22T23:57:38.818 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:39.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 -- 172.21.15.103:0/814840352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 msgr2=0x7fddd8109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 --2- 172.21.15.103:0/814840352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 0x7fddd8109360 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fddc0009900 tx=0x7fddc002a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 -- 172.21.15.103:0/814840352 shutdown_connections 2024-05-22T23:57:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 --2- 172.21.15.103:0/814840352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddd81098a0 0x7fddd8110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 --2- 172.21.15.103:0/814840352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 0x7fddd8109360 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 --2- 172.21.15.103:0/814840352 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddd8107cf0 0x7fddd8108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 -- 172.21.15.103:0/814840352 >> 172.21.15.103:0/814840352 conn(0x7fddd8076cb0 msgr2=0x7fddd80790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 -- 172.21.15.103:0/814840352 shutdown_connections 2024-05-22T23:57:39.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.013+0000 7fddddc00640 1 -- 172.21.15.103:0/814840352 wait complete. 2024-05-22T23:57:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.014+0000 7fddddc00640 1 Processor -- start 2024-05-22T23:57:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddddc00640 1 -- start start 2024-05-22T23:57:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddddc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddd8107cf0 0x7fddd819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddddc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 0x7fddd819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddddc00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddd81098a0 0x7fddd81a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddddc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fddd81139c0 con 0x7fddd8108ee0 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddddc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8113840 con 0x7fddd8107cf0 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddddc00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fddd8113b40 con 0x7fddd81098a0 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddcf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 0x7fddd819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fdddd200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddd81098a0 0x7fddd81a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddcf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 0x7fddd819e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48062/0 (socket says 172.21.15.103:48062) 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddcf400640 1 -- 172.21.15.103:0/331616722 learned_addr learned my addr 172.21.15.103:0/331616722 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.015+0000 7fddd7e00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddd8107cf0 0x7fddd819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd7e00640 1 -- 172.21.15.103:0/331616722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddd81098a0 msgr2=0x7fddd81a55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd7e00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddd81098a0 0x7fddd81a55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd7e00640 1 -- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 msgr2=0x7fddd819e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd7e00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 0x7fddd819e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd7e00640 1 -- 172.21.15.103:0/331616722 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddd81a5c30 con 0x7fddd8107cf0 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fdddd200640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddd81098a0 0x7fddd81a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd7e00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddd8107cf0 0x7fddd819e060 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fddc800e9c0 tx=0x7fddc800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd5600640 1 -- 172.21.15.103:0/331616722 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc800ccc0 con 0x7fddd8107cf0 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd5600640 1 -- 172.21.15.103:0/331616722 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddc80035f0 con 0x7fddd8107cf0 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddd81a5f20 con 0x7fddd8107cf0 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.016+0000 7fddd5600640 1 -- 172.21.15.103:0/331616722 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddc800ce60 con 0x7fddd8107cf0 2024-05-22T23:57:39.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.017+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddd81a6390 con 0x7fddd8107cf0 2024-05-22T23:57:39.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.017+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddd810efb0 con 0x7fddd8107cf0 2024-05-22T23:57:39.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.019+0000 7fddd5600640 1 -- 172.21.15.103:0/331616722 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fddc80037a0 con 0x7fddd8107cf0 2024-05-22T23:57:39.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.020+0000 7fddd5600640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddbc0a2c10 0x7fddbc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.021+0000 7fddd5600640 1 -- 172.21.15.103:0/331616722 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fddc80cf6a0 con 0x7fddd8107cf0 2024-05-22T23:57:39.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.021+0000 7fddcf400640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddbc0a2c10 0x7fddbc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.021+0000 7fddcf400640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddbc0a2c10 0x7fddbc0a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fddd819f520 tx=0x7fddc00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:39.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.025+0000 7fddd5600640 1 -- 172.21.15.103:0/331616722 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fddc809c360 con 0x7fddd8107cf0 2024-05-22T23:57:39.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:38 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2758083076' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-22T23:57:39.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:38 smithi103 ceph-mon[31370]: osdmap e90: 12 total, 12 up, 12 in 2024-05-22T23:57:39.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:38 smithi103 ceph-mon[31370]: pgmap v324: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-05-22T23:57:39.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:38 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3417520008' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-22T23:57:39.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:38 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-22T23:57:39.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:38 smithi103 ceph-mon[31370]: osdmap e91: 12 total, 12 up, 12 in 2024-05-22T23:57:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:38 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2758083076' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-22T23:57:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:38 smithi174 ceph-mon[36179]: osdmap e90: 12 total, 12 up, 12 in 2024-05-22T23:57:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:38 smithi174 ceph-mon[36179]: pgmap v324: 129 pgs: 6 creating+peering, 26 unknown, 97 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-05-22T23:57:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:38 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3417520008' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-22T23:57:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:38 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-22T23:57:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:38 smithi174 ceph-mon[36179]: osdmap e91: 12 total, 12 up, 12 in 2024-05-22T23:57:39.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.248+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddd8062510 con 0x7fddd8107cf0 2024-05-22T23:57:39.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.249+0000 7fddd5600640 1 -- 172.21.15.103:0/331616722 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11824 (secure 0 0 0) 0x7fddc809bc60 con 0x7fddd8107cf0 2024-05-22T23:57:39.253 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:39.253 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24563,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-22T23:57:38.243778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24575,"qdb_cluster":[24575]},"id":1}]} 2024-05-22T23:57:39.253 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 17 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddbc0a2c10 msgr2=0x7fddbc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddbc0a2c10 0x7fddbc0a50d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fddd819f520 tx=0x7fddc00315b0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddd8107cf0 msgr2=0x7fddd819e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddd8107cf0 0x7fddd819e060 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fddc800e9c0 tx=0x7fddc800ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 shutdown_connections 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddbc0a2c10 0x7fddbc0a50d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fddd81098a0 0x7fddd81a55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fddd8108ee0 0x7fddd819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 --2- 172.21.15.103:0/331616722 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddd8107cf0 0x7fddd819e060 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 >> 172.21.15.103:0/331616722 conn(0x7fddd8076cb0 msgr2=0x7fddd810c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:39.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 shutdown_connections 2024-05-22T23:57:39.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.252+0000 7fddddc00640 1 -- 172.21.15.103:0/331616722 wait complete. 2024-05-22T23:57:39.274 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:57:39.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.463+0000 7f8f8a200640 1 -- 172.21.15.103:0/3855477631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f84104ae0 msgr2=0x7f8f84104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.463+0000 7f8f8a200640 1 --2- 172.21.15.103:0/3855477631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f84104ae0 0x7f8f84104f60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009900 tx=0x7f8f7402a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.464+0000 7f8f8a200640 1 -- 172.21.15.103:0/3855477631 shutdown_connections 2024-05-22T23:57:39.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.464+0000 7f8f8a200640 1 --2- 172.21.15.103:0/3855477631 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f841054a0 0x7f8f8410bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.464+0000 7f8f8a200640 1 --2- 172.21.15.103:0/3855477631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f84104ae0 0x7f8f84104f60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.464+0000 7f8f8a200640 1 --2- 172.21.15.103:0/3855477631 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f841038f0 0x7f8f84103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.464+0000 7f8f8a200640 1 -- 172.21.15.103:0/3855477631 >> 172.21.15.103:0/3855477631 conn(0x7f8f840fee80 msgr2=0x7f8f841012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.464+0000 7f8f8a200640 1 -- 172.21.15.103:0/3855477631 shutdown_connections 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.464+0000 7f8f8a200640 1 -- 172.21.15.103:0/3855477631 wait complete. 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.465+0000 7f8f8a200640 1 Processor -- start 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.465+0000 7f8f8a200640 1 -- start start 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.465+0000 7f8f8a200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f841038f0 0x7f8f84199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.465+0000 7f8f8a200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f84104ae0 0x7f8f8419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.465+0000 7f8f8a200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f841054a0 0x7f8f841a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.465+0000 7f8f8a200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8410f5c0 con 0x7f8f841038f0 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.465+0000 7f8f8a200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8410f440 con 0x7f8f841054a0 2024-05-22T23:57:39.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.465+0000 7f8f8a200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8f8410f740 con 0x7f8f84104ae0 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f88e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f841038f0 0x7f8f84199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f841054a0 0x7f8f841a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f83e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f84104ae0 0x7f8f8419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f841054a0 0x7f8f841a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48512/0 (socket says 172.21.15.103:48512) 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 -- 172.21.15.103:0/2836143989 learned_addr learned my addr 172.21.15.103:0/2836143989 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f88e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f841038f0 0x7f8f84199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48084/0 (socket says 172.21.15.103:48084) 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 -- 172.21.15.103:0/2836143989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f84104ae0 msgr2=0x7f8f8419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f84104ae0 0x7f8f8419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 -- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f841038f0 msgr2=0x7f8f84199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f841038f0 0x7f8f84199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 -- 172.21.15.103:0/2836143989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f841a1830 con 0x7f8f841054a0 2024-05-22T23:57:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.466+0000 7f8f89800640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f841054a0 0x7f8f841a1190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00eb10 tx=0x7f8f7c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.467+0000 7f8f81600640 1 -- 172.21.15.103:0/2836143989 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c004b20 con 0x7f8f841054a0 2024-05-22T23:57:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.467+0000 7f8f81600640 1 -- 172.21.15.103:0/2836143989 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f7c004cc0 con 0x7f8f841054a0 2024-05-22T23:57:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.467+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f841a1b20 con 0x7f8f841054a0 2024-05-22T23:57:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.467+0000 7f8f81600640 1 -- 172.21.15.103:0/2836143989 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c010760 con 0x7f8f841054a0 2024-05-22T23:57:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.467+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f841a2020 con 0x7f8f841054a0 2024-05-22T23:57:39.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.468+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8410ac10 con 0x7f8f841054a0 2024-05-22T23:57:39.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.470+0000 7f8f81600640 1 -- 172.21.15.103:0/2836143989 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8f7c010900 con 0x7f8f841054a0 2024-05-22T23:57:39.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.471+0000 7f8f81600640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8f6c0a2b90 0x7f8f6c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.471+0000 7f8f81600640 1 -- 172.21.15.103:0/2836143989 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f8f7c0cfa90 con 0x7f8f841054a0 2024-05-22T23:57:39.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.471+0000 7f8f88e00640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8f6c0a2b90 0x7f8f6c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.475 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.472+0000 7f8f88e00640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8f6c0a2b90 0x7f8f6c0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8f8419b0f0 tx=0x7f8f78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:39.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.473+0000 7f8f81600640 1 -- 172.21.15.103:0/2836143989 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8f7c053850 con 0x7f8f841054a0 2024-05-22T23:57:39.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.660+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f8402d3d0 con 0x7f8f841054a0 2024-05-22T23:57:39.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.663+0000 7f8f81600640 1 -- 172.21.15.103:0/2836143989 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7f8f7c09c7b0 con 0x7f8f841054a0 2024-05-22T23:57:39.667 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:39.667 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:57:39.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8f6c0a2b90 msgr2=0x7f8f6c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8f6c0a2b90 0x7f8f6c0a5050 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8f8419b0f0 tx=0x7f8f78005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:39.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f841054a0 msgr2=0x7f8f841a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f841054a0 0x7f8f841a1190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00eb10 tx=0x7f8f7c00c2d0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 shutdown_connections 2024-05-22T23:57:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8f6c0a2b90 0x7f8f6c0a5050 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8f841054a0 0x7f8f841a1190 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f84104ae0 0x7f8f8419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 --2- 172.21.15.103:0/2836143989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f841038f0 0x7f8f84199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 >> 172.21.15.103:0/2836143989 conn(0x7f8f840fee80 msgr2=0x7f8f84107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 shutdown_connections 2024-05-22T23:57:39.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.665+0000 7f8f8a200640 1 -- 172.21.15.103:0/2836143989 wait complete. 2024-05-22T23:57:39.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:57:39.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 -- 172.21.15.103:0/1852565248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8102900 msgr2=0x7feaa8102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 --2- 172.21.15.103:0/1852565248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8102900 0x7feaa8102d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fea98009900 tx=0x7fea9802a410 comp rx=0 tx=0).stop 2024-05-22T23:57:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 -- 172.21.15.103:0/1852565248 shutdown_connections 2024-05-22T23:57:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 --2- 172.21.15.103:0/1852565248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa81032c0 0x7feaa8109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 --2- 172.21.15.103:0/1852565248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8102900 0x7feaa8102d80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 --2- 172.21.15.103:0/1852565248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feaa8101710 0x7feaa8101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 -- 172.21.15.103:0/1852565248 >> 172.21.15.103:0/1852565248 conn(0x7feaa80fcc80 msgr2=0x7feaa80ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 -- 172.21.15.103:0/1852565248 shutdown_connections 2024-05-22T23:57:39.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.882+0000 7feaa7200640 1 -- 172.21.15.103:0/1852565248 wait complete. 2024-05-22T23:57:39.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.883+0000 7feaa7200640 1 Processor -- start 2024-05-22T23:57:39.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.883+0000 7feaa7200640 1 -- start start 2024-05-22T23:57:39.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.883+0000 7feaa7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8101710 0x7feaa8075620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.883+0000 7feaa7200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feaa8102900 0x7feaa8073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.883+0000 7feaa7200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa81032c0 0x7feaa80741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.883+0000 7feaa7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feaa810d3e0 con 0x7feaa8101710 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.883+0000 7feaa7200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feaa810d260 con 0x7feaa8102900 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.883+0000 7feaa7200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feaa810d560 con 0x7feaa81032c0 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8101710 0x7feaa8075620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8101710 0x7feaa8075620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48100/0 (socket says 172.21.15.103:48100) 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 -- 172.21.15.103:0/1221337743 learned_addr learned my addr 172.21.15.103:0/1221337743 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5400640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feaa8102900 0x7feaa8073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa6800640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa81032c0 0x7feaa80741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 -- 172.21.15.103:0/1221337743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa81032c0 msgr2=0x7feaa80741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa81032c0 0x7feaa80741a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 -- 172.21.15.103:0/1221337743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feaa8102900 msgr2=0x7feaa8073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feaa8102900 0x7feaa8073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 -- 172.21.15.103:0/1221337743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa8074a30 con 0x7feaa8101710 2024-05-22T23:57:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.884+0000 7feaa5e00640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8101710 0x7feaa8075620 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fea9400e9c0 tx=0x7fea9400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.885+0000 7fea9ea00640 1 -- 172.21.15.103:0/1221337743 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9400cde0 con 0x7feaa8101710 2024-05-22T23:57:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.885+0000 7fea9ea00640 1 -- 172.21.15.103:0/1221337743 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea94010910 con 0x7feaa8101710 2024-05-22T23:57:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.885+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaa81a1b50 con 0x7feaa8101710 2024-05-22T23:57:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.885+0000 7fea9ea00640 1 -- 172.21.15.103:0/1221337743 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea940189c0 con 0x7feaa8101710 2024-05-22T23:57:39.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.885+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaa81a2070 con 0x7feaa8101710 2024-05-22T23:57:39.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.886+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaa8108a30 con 0x7feaa8101710 2024-05-22T23:57:39.892 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.888+0000 7fea9ea00640 1 -- 172.21.15.103:0/1221337743 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fea94010ac0 con 0x7feaa8101710 2024-05-22T23:57:39.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.889+0000 7fea9ea00640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea8c0a2b90 0x7fea8c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:39.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.889+0000 7fea9ea00640 1 -- 172.21.15.103:0/1221337743 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fea940cf990 con 0x7feaa8101710 2024-05-22T23:57:39.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.889+0000 7feaa5400640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea8c0a2b90 0x7fea8c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:39.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.890+0000 7feaa5400640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea8c0a2b90 0x7fea8c0a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fea98006010 tx=0x7fea98000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:39.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:39.893+0000 7fea9ea00640 1 -- 172.21.15.103:0/1221337743 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fea94053850 con 0x7feaa8101710 2024-05-22T23:57:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:39 smithi105 ceph-mon[36426]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-22T23:57:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:39 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-22T23:57:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:39 smithi105 ceph-mon[36426]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:39 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/331616722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:39 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2836143989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:40.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:39 smithi103 ceph-mon[31370]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-22T23:57:40.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:39 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-22T23:57:40.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:39 smithi103 ceph-mon[31370]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:40.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:39 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/331616722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:40.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:39 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2836143989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.096+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feaa8062470 con 0x7feaa8101710 2024-05-22T23:57:40.104 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.100+0000 7fea9ea00640 1 -- 172.21.15.103:0/1221337743 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7fea9409c650 con 0x7feaa8101710 2024-05-22T23:57:40.105 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:40.105 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:57:40.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea8c0a2b90 msgr2=0x7fea8c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea8c0a2b90 0x7fea8c0a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fea98006010 tx=0x7fea98000bc0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8101710 msgr2=0x7feaa8075620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8101710 0x7feaa8075620 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fea9400e9c0 tx=0x7fea9400ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:40.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 shutdown_connections 2024-05-22T23:57:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fea8c0a2b90 0x7fea8c0a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaa81032c0 0x7feaa80741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feaa8102900 0x7feaa8073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 --2- 172.21.15.103:0/1221337743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feaa8101710 0x7feaa8075620 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 >> 172.21.15.103:0/1221337743 conn(0x7feaa80fcc80 msgr2=0x7feaa8105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 shutdown_connections 2024-05-22T23:57:40.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.103+0000 7feaa7200640 1 -- 172.21.15.103:0/1221337743 wait complete. 2024-05-22T23:57:40.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:39 smithi174 ceph-mon[36179]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-22T23:57:40.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:39 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-22T23:57:40.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:39 smithi174 ceph-mon[36179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:57:40.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:39 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/331616722' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:40.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:39 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2836143989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:40.143 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:57:40.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 -- 172.21.15.103:0/1648547792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 msgr2=0x7f0474073ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 --2- 172.21.15.103:0/1648547792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 0x7f0474073ca0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0468009900 tx=0x7f046802a480 comp rx=0 tx=0).stop 2024-05-22T23:57:40.347 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 -- 172.21.15.103:0/1648547792 shutdown_connections 2024-05-22T23:57:40.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 --2- 172.21.15.103:0/1648547792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0474109750 0x7f047410bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 --2- 172.21.15.103:0/1648547792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0474074270 0x7f04740746d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 --2- 172.21.15.103:0/1648547792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 0x7f0474073ca0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 -- 172.21.15.103:0/1648547792 >> 172.21.15.103:0/1648547792 conn(0x7f04740fce50 msgr2=0x7f04740ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:40.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 -- 172.21.15.103:0/1648547792 shutdown_connections 2024-05-22T23:57:40.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.343+0000 7f047c600640 1 -- 172.21.15.103:0/1648547792 wait complete. 2024-05-22T23:57:40.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.344+0000 7f047c600640 1 Processor -- start 2024-05-22T23:57:40.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.344+0000 7f047c600640 1 -- start start 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f047c600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0474074270 0x7f0474199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f047c600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 0x7f047419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f047c600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0474109750 0x7f04741a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f047c600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f047410f780 con 0x7f0474075850 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f047c600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f047410f600 con 0x7f0474109750 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f047c600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f047410f900 con 0x7f0474074270 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 0x7f047419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 0x7f047419a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48118/0 (socket says 172.21.15.103:48118) 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479200640 1 -- 172.21.15.103:0/512890754 learned_addr learned my addr 172.21.15.103:0/512890754 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479c00640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0474074270 0x7f0474199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f047a600640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0474109750 0x7f04741a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479200640 1 -- 172.21.15.103:0/512890754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0474074270 msgr2=0x7f0474199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479200640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0474074270 0x7f0474199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479200640 1 -- 172.21.15.103:0/512890754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0474109750 msgr2=0x7f04741a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479200640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0474109750 0x7f04741a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479200640 1 -- 172.21.15.103:0/512890754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04741a19f0 con 0x7f0474075850 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.345+0000 7f0479c00640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0474074270 0x7f0474199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.346+0000 7f0479200640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 0x7f047419a310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f046000ea60 tx=0x7f046000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.346+0000 7f0472000640 1 -- 172.21.15.103:0/512890754 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046000cde0 con 0x7f0474075850 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.346+0000 7f0472000640 1 -- 172.21.15.103:0/512890754 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04600035f0 con 0x7f0474075850 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.346+0000 7f0472000640 1 -- 172.21.15.103:0/512890754 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0460010670 con 0x7f0474075850 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.346+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04741a1ce0 con 0x7f0474075850 2024-05-22T23:57:40.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.346+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04741a2180 con 0x7f0474075850 2024-05-22T23:57:40.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.347+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0474108510 con 0x7f0474075850 2024-05-22T23:57:40.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.349+0000 7f0472000640 1 -- 172.21.15.103:0/512890754 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f04600037a0 con 0x7f0474075850 2024-05-22T23:57:40.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.351+0000 7f0472000640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f045c0a2c10 0x7f045c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:40.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.351+0000 7f0472000640 1 -- 172.21.15.103:0/512890754 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f04600cf550 con 0x7f0474075850 2024-05-22T23:57:40.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.351+0000 7f0479c00640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f045c0a2c10 0x7f045c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:40.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.352+0000 7f0479c00640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f045c0a2c10 0x7f045c0a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0468031440 tx=0x7f04680315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:40.358 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.355+0000 7f0472000640 1 -- 172.21.15.103:0/512890754 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f046009c210 con 0x7f0474075850 2024-05-22T23:57:40.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.593+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0474062510 con 0x7f0474075850 2024-05-22T23:57:40.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.595+0000 7f0472000640 1 -- 172.21.15.103:0/512890754 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11824 (secure 0 0 0) 0x7f046009bb10 con 0x7f0474075850 2024-05-22T23:57:40.599 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:40.599 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24559,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24563,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-22T23:57:38.243778+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24575,"qdb_cluster":[24575]},"id":1}]} 2024-05-22T23:57:40.599 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 17 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.597+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f045c0a2c10 msgr2=0x7f045c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.597+0000 7f047c600640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f045c0a2c10 0x7f045c0a50d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0468031440 tx=0x7f04680315b0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.597+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 msgr2=0x7f047419a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.597+0000 7f047c600640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 0x7f047419a310 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f046000ea60 tx=0x7f046000ef20 comp rx=0 tx=0).stop 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.598+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 shutdown_connections 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.598+0000 7f047c600640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f045c0a2c10 0x7f045c0a50d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.598+0000 7f047c600640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0474109750 0x7f04741a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.598+0000 7f047c600640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0474075850 0x7f047419a310 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.598+0000 7f047c600640 1 --2- 172.21.15.103:0/512890754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0474074270 0x7f0474199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.598+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 >> 172.21.15.103:0/512890754 conn(0x7f04740fce50 msgr2=0x7f04741056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:40.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.598+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 shutdown_connections 2024-05-22T23:57:40.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.598+0000 7f047c600640 1 -- 172.21.15.103:0/512890754 wait complete. 2024-05-22T23:57:40.621 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:57:40.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 -- 172.21.15.103:0/2964709738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4102170 msgr2=0x7fd4c4104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 --2- 172.21.15.103:0/2964709738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4102170 0x7fd4c4104560 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc009ca0 tx=0x7fd4bc02a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 -- 172.21.15.103:0/2964709738 shutdown_connections 2024-05-22T23:57:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 --2- 172.21.15.103:0/2964709738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4c41073d0 0x7fd4c4109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 --2- 172.21.15.103:0/2964709738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4c4104aa0 0x7fd4c4106e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 --2- 172.21.15.103:0/2964709738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4102170 0x7fd4c4104560 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 -- 172.21.15.103:0/2964709738 >> 172.21.15.103:0/2964709738 conn(0x7fd4c40fbb40 msgr2=0x7fd4c40fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 -- 172.21.15.103:0/2964709738 shutdown_connections 2024-05-22T23:57:40.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.819+0000 7fd4c9800640 1 -- 172.21.15.103:0/2964709738 wait complete. 2024-05-22T23:57:40.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.820+0000 7fd4c9800640 1 Processor -- start 2024-05-22T23:57:40.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.820+0000 7fd4c9800640 1 -- start start 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c9800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4c4102170 0x7fd4c4197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c9800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4104aa0 0x7fd4c4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c9800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4c41073d0 0x7fd4c419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c9800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c410d3e0 con 0x7fd4c41073d0 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c9800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c410d260 con 0x7fd4c4102170 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c9800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd4c410d560 con 0x7fd4c4104aa0 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c8e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4c41073d0 0x7fd4c419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c8e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4c41073d0 0x7fd4c419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48130/0 (socket says 172.21.15.103:48130) 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c8e00640 1 -- 172.21.15.103:0/992486879 learned_addr learned my addr 172.21.15.103:0/992486879 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c3400640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4104aa0 0x7fd4c4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:40.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c3e00640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4c4102170 0x7fd4c4197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c8e00640 1 -- 172.21.15.103:0/992486879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4104aa0 msgr2=0x7fd4c4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c8e00640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4104aa0 0x7fd4c4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c8e00640 1 -- 172.21.15.103:0/992486879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4c4102170 msgr2=0x7fd4c4197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.821+0000 7fd4c8e00640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4c4102170 0x7fd4c4197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.822+0000 7fd4c8e00640 1 -- 172.21.15.103:0/992486879 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4c419f640 con 0x7fd4c41073d0 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.822+0000 7fd4c3400640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4104aa0 0x7fd4c4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.822+0000 7fd4c8e00640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4c41073d0 0x7fd4c419efd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4b400e970 tx=0x7fd4b400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.822+0000 7fd4c0c00640 1 -- 172.21.15.103:0/992486879 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b400ccc0 con 0x7fd4c41073d0 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.822+0000 7fd4c0c00640 1 -- 172.21.15.103:0/992486879 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd4b400ce60 con 0x7fd4c41073d0 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.822+0000 7fd4c0c00640 1 -- 172.21.15.103:0/992486879 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4b4010490 con 0x7fd4c41073d0 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.822+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4c419f930 con 0x7fd4c41073d0 2024-05-22T23:57:40.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.822+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4c419fe30 con 0x7fd4c41073d0 2024-05-22T23:57:40.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.823+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4c40fd630 con 0x7fd4c41073d0 2024-05-22T23:57:40.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.825+0000 7fd4c0c00640 1 -- 172.21.15.103:0/992486879 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd4b4005000 con 0x7fd4c41073d0 2024-05-22T23:57:40.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.827+0000 7fd4c0c00640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4a80a2b90 0x7fd4a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:40.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.827+0000 7fd4c0c00640 1 -- 172.21.15.103:0/992486879 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd4b40cf920 con 0x7fd4c41073d0 2024-05-22T23:57:40.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.827+0000 7fd4c3e00640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4a80a2b90 0x7fd4a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:40.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.828+0000 7fd4c3e00640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4a80a2b90 0x7fd4a80a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc005e30 tx=0x7fd4bc005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:40.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:40.829+0000 7fd4c0c00640 1 -- 172.21.15.103:0/992486879 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd4b4053850 con 0x7fd4c41073d0 2024-05-22T23:57:41.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.031+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4c4062470 con 0x7fd4c41073d0 2024-05-22T23:57:41.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.035+0000 7fd4c0c00640 1 -- 172.21.15.103:0/992486879 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7fd4b409c5e0 con 0x7fd4c41073d0 2024-05-22T23:57:41.039 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:41.040 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4a80a2b90 msgr2=0x7fd4a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4a80a2b90 0x7fd4a80a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd4bc005e30 tx=0x7fd4bc005f50 comp rx=0 tx=0).stop 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4c41073d0 msgr2=0x7fd4c419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4c41073d0 0x7fd4c419efd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd4b400e970 tx=0x7fd4b400ed70 comp rx=0 tx=0).stop 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 shutdown_connections 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4a80a2b90 0x7fd4a80a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd4c41073d0 0x7fd4c419efd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4c4104aa0 0x7fd4c4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 --2- 172.21.15.103:0/992486879 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4c4102170 0x7fd4c4197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.037+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 >> 172.21.15.103:0/992486879 conn(0x7fd4c40fbb40 msgr2=0x7fd4c4102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:41.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.038+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 shutdown_connections 2024-05-22T23:57:41.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.038+0000 7fd4c9800640 1 -- 172.21.15.103:0/992486879 wait complete. 2024-05-22T23:57:41.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:40 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1221337743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:41.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:40 smithi103 ceph-mon[31370]: pgmap v326: 129 pgs: 6 creating+peering, 10 unknown, 113 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 7 op/s 2024-05-22T23:57:41.058 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:40 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/512890754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:40 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1221337743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:40 smithi105 ceph-mon[36426]: pgmap v326: 129 pgs: 6 creating+peering, 10 unknown, 113 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 7 op/s 2024-05-22T23:57:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:40 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/512890754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:41.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-22T23:57:32.804111+0000', 'modified': '2024-05-22T23:57:38.243778+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.105:6839/160741513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 160741513}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 160741513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24575, 'qdb_cluster': [24575]} 2024-05-22T23:57:41.061 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-22T23:57:41.061 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-22T23:57:41.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:40 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1221337743' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:41.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:40 smithi174 ceph-mon[36179]: pgmap v326: 129 pgs: 6 creating+peering, 10 unknown, 113 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 7 op/s 2024-05-22T23:57:41.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:40 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/512890754' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.252+0000 7f3585600640 1 -- 172.21.15.103:0/2510082948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3580107db0 msgr2=0x7f35801081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.252+0000 7f3585600640 1 --2- 172.21.15.103:0/2510082948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3580107db0 0x7f35801081d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3570009900 tx=0x7f357002a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.252+0000 7f3585600640 1 -- 172.21.15.103:0/2510082948 shutdown_connections 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.252+0000 7f3585600640 1 --2- 172.21.15.103:0/2510082948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3580109960 0x7f35801101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.252+0000 7f3585600640 1 --2- 172.21.15.103:0/2510082948 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3580108fa0 0x7f3580109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.252+0000 7f3585600640 1 --2- 172.21.15.103:0/2510082948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3580107db0 0x7f35801081d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.252+0000 7f3585600640 1 -- 172.21.15.103:0/2510082948 >> 172.21.15.103:0/2510082948 conn(0x7f3580076d40 msgr2=0x7f3580079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.253+0000 7f3585600640 1 -- 172.21.15.103:0/2510082948 shutdown_connections 2024-05-22T23:57:41.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.253+0000 7f3585600640 1 -- 172.21.15.103:0/2510082948 wait complete. 2024-05-22T23:57:41.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.253+0000 7f3585600640 1 Processor -- start 2024-05-22T23:57:41.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3585600640 1 -- start start 2024-05-22T23:57:41.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3585600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3580107db0 0x7f358019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3585600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3580108fa0 0x7f358019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3585600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3580109960 0x7f35801a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3585600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3580113a80 con 0x7f3580107db0 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3585600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3580113900 con 0x7f3580108fa0 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3585600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3580113c00 con 0x7f3580109960 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3584c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3580109960 0x7f35801a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3584c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3580109960 0x7f35801a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59832/0 (socket says 172.21.15.103:59832) 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3584c00640 1 -- 172.21.15.103:0/2322126304 learned_addr learned my addr 172.21.15.103:0/2322126304 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f3577e00640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3580108fa0 0x7f358019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.254+0000 7f357fe00640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3580107db0 0x7f358019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f3584c00640 1 -- 172.21.15.103:0/2322126304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3580108fa0 msgr2=0x7f358019e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f3584c00640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3580108fa0 0x7f358019e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f3584c00640 1 -- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3580107db0 msgr2=0x7f358019e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f3584c00640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3580107db0 0x7f358019e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f3584c00640 1 -- 172.21.15.103:0/2322126304 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35801a5cb0 con 0x7f3580109960 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f3584c00640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3580109960 0x7f35801a5640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f357800eb10 tx=0x7f357800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f357d600640 1 -- 172.21.15.103:0/2322126304 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578004b20 con 0x7f3580109960 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f357d600640 1 -- 172.21.15.103:0/2322126304 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3578004cc0 con 0x7f3580109960 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f3585600640 1 -- 172.21.15.103:0/2322126304 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35801a5fa0 con 0x7f3580109960 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f357d600640 1 -- 172.21.15.103:0/2322126304 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578010760 con 0x7f3580109960 2024-05-22T23:57:41.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.255+0000 7f3585600640 1 -- 172.21.15.103:0/2322126304 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35801a6460 con 0x7f3580109960 2024-05-22T23:57:41.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.258+0000 7f3589b79740 1 Processor -- start 2024-05-22T23:57:41.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f357d600640 1 -- 172.21.15.103:0/2322126304 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3578005020 con 0x7f3580109960 2024-05-22T23:57:41.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f3589b79740 1 -- start start 2024-05-22T23:57:41.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f3589b79740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f3589b79740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6de40 0x565063d72940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f3589b79740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 0x565063d75440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f3589b79740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565063d34db0 con 0x565063d6d450 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f3589b79740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x565063cd1490 con 0x565063d6de40 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f3589b79740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565063c79100 con 0x565063d72fb0 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.259+0000 7f3577e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48166/0 (socket says 172.21.15.103:48166) 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 -- 172.21.15.103:0/3280649545 learned_addr learned my addr 172.21.15.103:0/3280649545 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f357fe00640 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6de40 0x565063d72940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 0x565063d75440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 -- 172.21.15.103:0/3280649545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 msgr2=0x565063d75440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 0x565063d75440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 -- 172.21.15.103:0/3280649545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6de40 msgr2=0x565063d72940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6de40 0x565063d72940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 -- 172.21.15.103:0/3280649545 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5650639e52d0 con 0x565063d6d450 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f357d600640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f35640a2b90 0x7f35640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f357d600640 1 -- 172.21.15.103:0/2322126304 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f35780cfa30 con 0x7f3580109960 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f3577e00640 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6d870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f35680135a0 tx=0x7f3568009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90224ddd37fb142 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.260+0000 7f357fe00640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f35640a2b90 0x7f35640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.261+0000 7f3575200640 1 -- 172.21.15.103:0/3280649545 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3568004050 con 0x565063d6d450 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.261+0000 7f3575200640 1 -- 172.21.15.103:0/3280649545 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35680041f0 con 0x565063d6d450 2024-05-22T23:57:41.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.261+0000 7f3575200640 1 -- 172.21.15.103:0/3280649545 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356800f7e0 con 0x565063d6d450 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.261+0000 7f3589b79740 1 -- 172.21.15.103:0/3280649545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 msgr2=0x565063d6d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.261+0000 7f357fe00640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f35640a2b90 0x7f35640a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f357002a9e0 tx=0x7f35700315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.261+0000 7f3589b79740 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6d870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f35680135a0 tx=0x7f3568009850 comp rx=0 tx=0).stop 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 -- 172.21.15.103:0/3280649545 shutdown_connections 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 0x565063d75440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6de40 0x565063d72940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 --2- 172.21.15.103:0/3280649545 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6d870 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 -- 172.21.15.103:0/3280649545 >> 172.21.15.103:0/3280649545 conn(0x565063d68ac0 msgr2=0x565063d68ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:41.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 -- 172.21.15.103:0/3280649545 shutdown_connections 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 -- 172.21.15.103:0/3280649545 wait complete. 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 -- start start 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.262+0000 7f3589b79740 1 Processor -- start 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3589b79740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6d450 0x565063d77750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3589b79740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d77c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3589b79740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 0x565063d7ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3589b79740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565063c8c520 con 0x565063d6de40 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3589b79740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x565063ccd450 con 0x565063d6d450 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3589b79740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565063d75f00 con 0x565063d72fb0 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3589b79740 10 client.0 _ll_register_callbacks cb 0x5650639e4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3577e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d77c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f357fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6d450 0x565063d77750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3577e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d77c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48182/0 (socket says 172.21.15.103:48182) 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3577e00640 1 -- 172.21.15.103:0/1066195220 learned_addr learned my addr 172.21.15.103:0/1066195220 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f3584c00640 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 0x565063d7ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f357fe00640 1 -- 172.21.15.103:0/1066195220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 msgr2=0x565063d7ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f357fe00640 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 0x565063d7ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f357fe00640 1 -- 172.21.15.103:0/1066195220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 msgr2=0x565063d77c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.263+0000 7f357fe00640 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d77c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f357fe00640 1 -- 172.21.15.103:0/1066195220 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565063aaf6b0 con 0x565063d6d450 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f3577e00640 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d77c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f357fe00640 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6d450 0x565063d77750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3570009230 tx=0x7f3570009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a729f1e448ea5090 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f356d600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-22T23:57:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f356d600640 1 -- 172.21.15.103:0/1066195220 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357004d070 con 0x565063d6d450 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f356d600640 1 -- 172.21.15.103:0/1066195220 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f35700326c0 con 0x565063d6d450 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f3589b79740 1 -- 172.21.15.103:0/1066195220 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35680041f0 con 0x565063d6d450 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f356d600640 1 -- 172.21.15.103:0/1066195220 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3570059050 con 0x565063d6d450 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f3589b79740 10 client.24683 fetch_fsmap learned FSMap version 17 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.264+0000 7f3589b79740 1 -- 172.21.15.103:0/1066195220 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x565063d82040 con 0x565063d6d450 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.265+0000 7f356d600640 1 -- 172.21.15.103:0/1066195220 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35680041f0 con 0x565063d6d450 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.265+0000 7f356d600640 1 -- 172.21.15.103:0/1066195220 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f3570003b20 con 0x565063d6d450 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.265+0000 7f3589b79740 10 client.24683 fetch_fsmap finished waiting for FSMap version 17 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.265+0000 7f3589b79740 10 client.24683 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.265+0000 7f3589b79740 20 client.24683 populate_metadata read hostname 'smithi103' 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.265+0000 7f3589b79740 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x565063d84610 0x565063d85a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.265+0000 7f3589b79740 4 client.24683 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.265+0000 7f3589b79740 1 -- 172.21.15.103:0/1066195220 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565063cdebc0 con 0x565063d84610 2024-05-22T23:57:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.266+0000 7f3577e00640 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x565063d84610 0x565063d85a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.266+0000 7f3577e00640 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x565063d84610 0x565063d85a80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3dec366cc6224553 server_cookie=d70fdbfd482bbcde in_seq=0 out_seq=0 2024-05-22T23:57:41.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.266+0000 7f356d600640 10 client.24683 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f356d600640 1 -- 172.21.15.103:0/1066195220 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x565063cdebc0 con 0x565063d84610 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f356d600640 10 client.24683 handle_command_reply: tid=0 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 client.24683 shutdown 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 -- 172.21.15.103:0/1066195220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6d450 msgr2=0x565063d77750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6d450 0x565063d77750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3570009230 tx=0x7f3570009260 comp rx=0 tx=0).stop 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 -- 172.21.15.103:0/1066195220 shutdown_connections 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x565063d84610 0x565063d85a80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d72fb0 0x565063d7ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d77c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 --2- 172.21.15.103:0/1066195220 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d6d450 0x565063d77750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f356d600640 0 client.24683 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-22T23:57:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 -- 172.21.15.103:0/1066195220 >> 172.21.15.103:0/1066195220 conn(0x565063d68ac0 msgr2=0x565063d67c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 -- 172.21.15.103:0/1066195220 shutdown_connections 2024-05-22T23:57:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.267+0000 7f3589b79740 1 -- 172.21.15.103:0/1066195220 wait complete. 2024-05-22T23:57:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.268+0000 7f3589b79740 20 client.24683 trim_cache size 0 max 16384 2024-05-22T23:57:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.287+0000 7f3589b79740 1 Processor -- start 2024-05-22T23:57:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.287+0000 7f3589b79740 1 -- start start 2024-05-22T23:57:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.287+0000 7f3589b79740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.287+0000 7f3589b79740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6de40 0x565063d7ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.287+0000 7f3589b79740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d6f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.287+0000 7f3589b79740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565063d83dc0 con 0x565063d6d450 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.287+0000 7f3589b79740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x565063d84490 con 0x565063d72fb0 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.287+0000 7f3589b79740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565063d84b60 con 0x565063d6de40 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48194/0 (socket says 172.21.15.103:48194) 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 -- 172.21.15.103:0/375489915 learned_addr learned my addr 172.21.15.103:0/375489915 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f3577e00640 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6de40 0x565063d7ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f3584c00640 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d6f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 -- 172.21.15.103:0/375489915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6de40 msgr2=0x565063d7ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6de40 0x565063d7ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 -- 172.21.15.103:0/375489915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 msgr2=0x565063d6f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d6f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 -- 172.21.15.103:0/375489915 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565063d725e0 con 0x565063d6d450 2024-05-22T23:57:41.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.288+0000 7f357fe00640 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6f310 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3570003f10 tx=0x7f3570004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ceadd67eb5532c server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f356fe00640 1 -- 172.21.15.103:0/375489915 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357004d050 con 0x565063d6d450 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f356fe00640 1 -- 172.21.15.103:0/375489915 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3570049e20 con 0x565063d6d450 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f356fe00640 1 -- 172.21.15.103:0/375489915 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570035040 con 0x565063d6d450 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f3589b79740 1 -- 172.21.15.103:0/375489915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 msgr2=0x565063d6f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f3589b79740 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6f310 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3570003f10 tx=0x7f3570004290 comp rx=0 tx=0).stop 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f3589b79740 1 -- 172.21.15.103:0/375489915 shutdown_connections 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f3589b79740 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d6f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f3589b79740 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6de40 0x565063d7ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f3589b79740 1 --2- 172.21.15.103:0/375489915 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6d450 0x565063d6f310 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.289+0000 7f3589b79740 1 -- 172.21.15.103:0/375489915 >> 172.21.15.103:0/375489915 conn(0x565063d68ac0 msgr2=0x565063d6e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.290+0000 7f3589b79740 1 -- 172.21.15.103:0/375489915 shutdown_connections 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.290+0000 7f3589b79740 1 -- 172.21.15.103:0/375489915 wait complete. 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.290+0000 7f3589b79740 1 -- start start 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.290+0000 7f3589b79740 1 Processor -- start 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3589b79740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6d450 0x565063d7a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3589b79740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d7aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3589b79740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d7daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3589b79740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565063d85900 con 0x565063d6de40 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3589b79740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x565063d85230 con 0x565063d72fb0 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3589b79740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x565063d85fd0 con 0x565063d6d450 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3589b79740 10 client.0 _ll_register_callbacks cb 0x7f35780093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-22T23:57:41.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6d450 0x565063d7a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3577e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d7aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6d450 0x565063d7a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59882/0 (socket says 172.21.15.103:59882) 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 -- 172.21.15.103:0/1392659086 learned_addr learned my addr 172.21.15.103:0/1392659086 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3577e00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d7aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48200/0 (socket says 172.21.15.103:48200) 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3584c00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d7daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 -- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 msgr2=0x565063d7daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d7daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 -- 172.21.15.103:0/1392659086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 msgr2=0x565063d7aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d7aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 -- 172.21.15.103:0/1392659086 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565063d7e0e0 con 0x565063d6d450 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3584c00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d7daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f3577e00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d7aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:41.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f357fe00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6d450 0x565063d7a590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3570003e90 tx=0x7f357002aaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8d3922c10f77f15 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.291+0000 7f356d600640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.292+0000 7f356d600640 1 -- 172.21.15.103:0/1392659086 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3570069020 con 0x565063d6d450 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.292+0000 7f3589b79740 1 -- 172.21.15.103:0/1392659086 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3570049e20 con 0x565063d6d450 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.292+0000 7f356d600640 1 -- 172.21.15.103:0/1392659086 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f3570049a20 con 0x565063d6d450 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.292+0000 7f356d600640 1 -- 172.21.15.103:0/1392659086 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f3570074050 con 0x565063d6d450 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.292+0000 7f3589b79740 10 client.24712 fetch_fsmap learned FSMap version 17 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.292+0000 7f3589b79740 1 -- 172.21.15.103:0/1392659086 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x565063dea840 con 0x565063d6d450 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.292+0000 7f356d600640 1 -- 172.21.15.103:0/1392659086 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3570049e20 con 0x565063d6d450 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.292+0000 7f356d600640 1 -- 172.21.15.103:0/1392659086 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f3570032bc0 con 0x565063d6d450 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.293+0000 7f3589b79740 10 client.24712 fetch_fsmap finished waiting for FSMap version 17 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.293+0000 7f3589b79740 10 client.24712 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.293+0000 7f3589b79740 20 client.24712 populate_metadata read hostname 'smithi103' 2024-05-22T23:57:41.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.293+0000 7f3589b79740 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x565063def0c0 0x565063deddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.293+0000 7f3589b79740 4 client.24712 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-22T23:57:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.293+0000 7f3589b79740 1 -- 172.21.15.103:0/1392659086 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "status"}) -- 0x565063cd8f10 con 0x565063def0c0 2024-05-22T23:57:41.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.293+0000 7f3577e00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x565063def0c0 0x565063deddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3577e00640 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x565063def0c0 0x565063deddd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e6c23d26197d5ad server_cookie=561f80c30401ea4 in_seq=0 out_seq=0 2024-05-22T23:57:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f356d600640 10 client.24712 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-22T23:57:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f356d600640 1 -- 172.21.15.103:0/1392659086 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+319 (crc 0 0 0) 0x565063cd8f10 con 0x565063def0c0 2024-05-22T23:57:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f356d600640 10 client.24712 handle_command_reply: tid=0 2024-05-22T23:57:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 client.24712 shutdown 2024-05-22T23:57:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 -- 172.21.15.103:0/1392659086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6d450 msgr2=0x565063d7a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6d450 0x565063d7a590 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3570003e90 tx=0x7f357002aaf0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 -- 172.21.15.103:0/1392659086 shutdown_connections 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x565063def0c0 0x565063deddd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x565063d72fb0 0x565063d7daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565063d6de40 0x565063d7aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f356d600640 0 client.24712 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 --2- 172.21.15.103:0/1392659086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x565063d6d450 0x565063d7a590 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 -- 172.21.15.103:0/1392659086 >> 172.21.15.103:0/1392659086 conn(0x565063d68ac0 msgr2=0x565063d6e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 -- 172.21.15.103:0/1392659086 shutdown_connections 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 1 -- 172.21.15.103:0/1392659086 wait complete. 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.294+0000 7f3589b79740 20 client.24712 trim_cache size 0 max 16384 2024-05-22T23:57:41.303 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "whoami": 0, 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 24575, 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "want_state": "up:active", 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "up:active", 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs", 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "rank_uptime": 7.584117826, 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "mdsmap_epoch": 17, 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch": 91, 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch_barrier": 91, 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout: "uptime": 14.352523756 2024-05-22T23:57:41.304 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-22T23:57:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 -- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f35640a2b90 msgr2=0x7f35640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f35640a2b90 0x7f35640a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f357002a9e0 tx=0x7f35700315d0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 -- 172.21.15.103:0/2322126304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3580109960 msgr2=0x7f35801a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3580109960 0x7f35801a5640 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f357800eb10 tx=0x7f357800c2d0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 -- 172.21.15.103:0/2322126304 shutdown_connections 2024-05-22T23:57:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f35640a2b90 0x7f35640a5050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3580109960 0x7f35801a5640 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3580108fa0 0x7f358019e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 --2- 172.21.15.103:0/2322126304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3580107db0 0x7f358019e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 -- 172.21.15.103:0/2322126304 >> 172.21.15.103:0/2322126304 conn(0x7f3580076d40 msgr2=0x7f358010c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 -- 172.21.15.103:0/2322126304 shutdown_connections 2024-05-22T23:57:41.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:41.295+0000 7f3585600640 1 -- 172.21.15.103:0/2322126304 wait complete. 2024-05-22T23:57:41.317 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-22T23:57:41.317 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-22T23:57:41.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph fs ls 2024-05-22T23:57:41.738 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:57:42.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:41 smithi103 ceph-mon[31370]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-22T23:57:42.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:41 smithi103 ceph-mon[31370]: Cluster is now healthy 2024-05-22T23:57:42.057 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:41 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/992486879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:42.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:41 smithi105 ceph-mon[36426]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-22T23:57:42.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:41 smithi105 ceph-mon[36426]: Cluster is now healthy 2024-05-22T23:57:42.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:41 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/992486879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:42.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:41 smithi174 ceph-mon[36179]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-22T23:57:42.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:41 smithi174 ceph-mon[36179]: Cluster is now healthy 2024-05-22T23:57:42.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:41 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/992486879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:43.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:42 smithi103 ceph-mon[31370]: pgmap v327: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s wr, 5 op/s 2024-05-22T23:57:43.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:42 smithi174 ceph-mon[36179]: pgmap v327: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s wr, 5 op/s 2024-05-22T23:57:43.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:42 smithi105 ceph-mon[36426]: pgmap v327: 129 pgs: 1 creating+peering, 128 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s wr, 5 op/s 2024-05-22T23:57:44.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.038+0000 7faef2600700 1 -- 172.21.15.103:0/2222013317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 msgr2=0x7faeec10a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:44.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.038+0000 7faef2600700 1 --2- 172.21.15.103:0/2222013317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 0x7faeec10a920 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faee0009920 tx=0x7faee0009c30 comp rx=0 tx=0).stop 2024-05-22T23:57:44.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.038+0000 7faef2600700 1 -- 172.21.15.103:0/2222013317 shutdown_connections 2024-05-22T23:57:44.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.038+0000 7faef2600700 1 --2- 172.21.15.103:0/2222013317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 0x7faeec10a920 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.038+0000 7faef2600700 1 --2- 172.21.15.103:0/2222013317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeec104e50 0x7faeec111310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.038+0000 7faef2600700 1 --2- 172.21.15.103:0/2222013317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeec104490 0x7faeec104910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.038+0000 7faef2600700 1 -- 172.21.15.103:0/2222013317 >> 172.21.15.103:0/2222013317 conn(0x7faeec0fffa0 msgr2=0x7faeec1023c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:44.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.039+0000 7faef2600700 1 -- 172.21.15.103:0/2222013317 shutdown_connections 2024-05-22T23:57:44.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.039+0000 7faef2600700 1 -- 172.21.15.103:0/2222013317 wait complete. 2024-05-22T23:57:44.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.040+0000 7faef2600700 1 Processor -- start 2024-05-22T23:57:44.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.040+0000 7faef2600700 1 -- start start 2024-05-22T23:57:44.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.040+0000 7faef2600700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeec104490 0x7faeec1a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:44.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.040+0000 7faef2600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeec104e50 0x7faeec1a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:44.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.040+0000 7faef2600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 0x7faeec19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.040+0000 7faef2600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faeec1147f0 con 0x7faeec104e50 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.040+0000 7faef2600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faeec114670 con 0x7faeec10a540 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.040+0000 7faef2600700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faeec114970 con 0x7faeec104490 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faeebe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeec104e50 0x7faeec1a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeec104490 0x7faeec1a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeec104490 0x7faeec1a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:38896/0 (socket says 172.21.15.103:38896) 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1200700 1 -- 172.21.15.103:0/51973938 learned_addr learned my addr 172.21.15.103:0/51973938 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faeebe00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeec104e50 0x7faeec1a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44404/0 (socket says 172.21.15.103:44404) 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1c00700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 0x7faeec19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1c00700 1 -- 172.21.15.103:0/51973938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeec104490 msgr2=0x7faeec1a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1c00700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeec104490 0x7faeec1a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1c00700 1 -- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeec104e50 msgr2=0x7faeec1a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:44.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1c00700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeec104e50 0x7faeec1a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1c00700 1 -- 172.21.15.103:0/51973938 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeec19b1d0 con 0x7faeec10a540 2024-05-22T23:57:44.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.041+0000 7faef1c00700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 0x7faeec19aa70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faee400efc0 tx=0x7faee400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:44.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.042+0000 7faee9600700 1 -- 172.21.15.103:0/51973938 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee40092e0 con 0x7faeec10a540 2024-05-22T23:57:44.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.042+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeec19b460 con 0x7faeec10a540 2024-05-22T23:57:44.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.042+0000 7faee9600700 1 -- 172.21.15.103:0/51973938 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee4004bb0 con 0x7faeec10a540 2024-05-22T23:57:44.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.042+0000 7faee9600700 1 -- 172.21.15.103:0/51973938 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee4004280 con 0x7faeec10a540 2024-05-22T23:57:44.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.042+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeec1a7780 con 0x7faeec10a540 2024-05-22T23:57:44.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.043+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeec10c9d0 con 0x7faeec10a540 2024-05-22T23:57:44.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.045+0000 7faee9600700 1 -- 172.21.15.103:0/51973938 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7faee401b030 con 0x7faeec10a540 2024-05-22T23:57:44.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.046+0000 7faee9600700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faed40a2c20 0x7faed40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:44.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.047+0000 7faee9600700 1 -- 172.21.15.103:0/51973938 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7faee40cf810 con 0x7faeec10a540 2024-05-22T23:57:44.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.047+0000 7faef1200700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faed40a2c20 0x7faed40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:44.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.047+0000 7faef1200700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faed40a2c20 0x7faed40a50e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faee0009680 tx=0x7faee0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:44.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.049+0000 7faee9600700 1 -- 172.21.15.103:0/51973938 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7faee4022020 con 0x7faeec10a540 2024-05-22T23:57:44.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.323+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7faeec069050 con 0x7faeec10a540 2024-05-22T23:57:44.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.324+0000 7faee9600700 1 -- 172.21.15.103:0/51973938 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7faee4010040 con 0x7faeec10a540 2024-05-22T23:57:44.328 INFO:teuthology.orchestra.run.smithi103.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-22T23:57:44.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faed40a2c20 msgr2=0x7faed40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:44.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faed40a2c20 0x7faed40a50e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7faee0009680 tx=0x7faee0019040 comp rx=0 tx=0).stop 2024-05-22T23:57:44.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 msgr2=0x7faeec19aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:44.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 0x7faeec19aa70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faee400efc0 tx=0x7faee400c3a0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 shutdown_connections 2024-05-22T23:57:44.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faeec10a540 0x7faeec19aa70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faed40a2c20 0x7faed40a50e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeec104490 0x7faeec1a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 --2- 172.21.15.103:0/51973938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faeec104e50 0x7faeec1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:44.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 >> 172.21.15.103:0/51973938 conn(0x7faeec0fffa0 msgr2=0x7faeec101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:44.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.327+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 shutdown_connections 2024-05-22T23:57:44.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:44.328+0000 7faef2600700 1 -- 172.21.15.103:0/51973938 wait complete. 2024-05-22T23:57:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:44 smithi105 ceph-mon[36426]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 696 B/s wr, 4 op/s 2024-05-22T23:57:44.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:44 smithi103 ceph-mon[31370]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 696 B/s wr, 4 op/s 2024-05-22T23:57:44.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:44 smithi174 ceph-mon[36179]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 696 B/s wr, 4 op/s 2024-05-22T23:57:45.342 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-22T23:57:45.342 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-22T23:57:45.342 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-05-22T23:57:45.342 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-22T23:57:45.343 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-22T23:57:45.343 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-22T23:57:45.343 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-22T23:57:45.343 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-22T23:57:45.343 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-22T23:57:45.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:45.343 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-05-22T23:57:45.400 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-22T23:57:45.400 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-22T23:57:45.400 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-05-22T23:57:45.400 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-05-22T23:57:45.400 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-05-22T23:57:45.400 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-05-22T23:57:45.400 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-22T23:57:45.401 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 42058sec preferred_lft 42058sec 2024-05-22T23:57:45.402 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-05-22T23:57:45.402 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-05-22T23:57:45.403 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-22T23:57:45.403 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-22T23:57:45.403 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-22T23:57:45.403 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add name ceph-brx type bridge 2024-05-22T23:57:45.403 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr flush dev ceph-brx 2024-05-22T23:57:45.403 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set ceph-brx up 2024-05-22T23:57:45.403 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-22T23:57:45.403 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-22T23:57:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-22T23:57:45.519 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:45 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/51973938' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-22T23:57:45.534 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:45 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/51973938' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-22T23:57:45.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:45 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/51973938' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-22T23:57:45.887 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-22T23:57:45.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:45.901 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-22T23:57:45.964 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-05-22T23:57:45.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:45.968 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-05-22T23:57:46.022 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-05-22T23:57:46.022 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-22T23:57:46.022 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-22T23:57:46.022 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-22T23:57:46.022 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-22T23:57:46.023 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-22T23:57:46.023 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-22T23:57:46.023 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-22T23:57:46.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-22T23:57:46.024 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-22T23:57:46.024 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-22T23:57:46.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-22T23:57:46.424 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:46 smithi103 ceph-mon[31370]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 367 B/s wr, 1 op/s 2024-05-22T23:57:46.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:46 smithi174 ceph-mon[36179]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 367 B/s wr, 1 op/s 2024-05-22T23:57:46.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:46 smithi105 ceph-mon[36426]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 367 B/s wr, 1 op/s 2024-05-22T23:57:46.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-22T23:57:46.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:46.815 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-05-22T23:57:46.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:46.868 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-05-22T23:57:46.921 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-22T23:57:46.956 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-22T23:57:46.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-22T23:57:46.956 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-22T23:57:46.956 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-22T23:57:47.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-22T23:57:47.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-22T23:57:47.144 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-22T23:57:47.144 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-22T23:57:47.144 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-22T23:57:47.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-22T23:57:47.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-22T23:57:47.144 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-22T23:57:47.145 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-22T23:57:47.145 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-22T23:57:47.145 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-22T23:57:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-22T23:57:47.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-22T23:57:47.435 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-22T23:57:47.435 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-22T23:57:47.436 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-05-22T23:57:47.436 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.0 master ceph-brx 2024-05-22T23:57:47.436 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-22T23:57:47.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-22T23:57:47.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-22T23:57:47.643 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-22T23:57:47.643 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-22T23:57:47.643 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-22T23:57:47.687 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-22T23:57:47.688 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-22T23:57:47.688 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-22T23:57:47.779 DEBUG:teuthology.orchestra.run.smithi103:> sudo modprobe fuse 2024-05-22T23:57:47.855 DEBUG:teuthology.orchestra.run.smithi103:> cat /proc/self/mounts | awk '{print $2}' 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/proc 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/sys 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/dev 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/security 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/shm 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/pts 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/run 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/cgroup 2024-05-22T23:57:47.908 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/pstore 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/bpf 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/ 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/selinux 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/proc/sys/fs/binfmt_misc 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/mqueue 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/hugepages 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/debug 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/tracing 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/fuse/connections 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/config 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-sysctl.service 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-22T23:57:47.909 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/nfs/rpc_pipefs 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/run/user/1000 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/ceph 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/df49e73f547dcdc4b3bf4ffb3d49798897278b98e66f874faba555488bbbbef9/merged 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/c16298afa3412001ed74ae97c92f34e455e60843994da2b3bb055af913f0acbe/merged 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/run/user/0 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/proc/sys/fs/binfmt_misc 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/eb0b14552f4162df842230224c9bb075d516aec593dfe48a20a9a8fcc5675f3f/merged 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/57498e415745938ffdc754bae2dd4a6d7199bbf80a5aada7fd586adf7b006141/merged 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/a97f95810007a039a5e2e1e8c1a0d0cd306316a8748ceb697d2ac83def7d34e8/merged 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/94dd9aa425d40df3f500aaa4fe40647c71699fcf33abe1ad2df376f1433fd97c/merged 2024-05-22T23:57:47.910 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/fa63bce37f59009fed33a1c9d9ab131a7716ae70d0dae61aa63d07e6cc4d7541/merged 2024-05-22T23:57:47.911 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/de85e5bff8327540b841416f25e095ca0412b39abcd98306eeddaff9fe70e612/merged 2024-05-22T23:57:47.911 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/da0d052c753aa2fa6d3e2d35ba4b6eb02eee5d23ebb3e5dfca4e883f20fe215d/merged 2024-05-22T23:57:47.911 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/af31f68906530943afa0697fe494e127a5d18abe9b7f4d1dbaa7936606a0dc16/merged 2024-05-22T23:57:47.911 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns 2024-05-22T23:57:47.911 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-22T23:57:47.911 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-22T23:57:47.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:47.912 DEBUG:teuthology.orchestra.run.smithi103:> ls /sys/fs/fuse/connections 2024-05-22T23:57:47.961 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-22T23:57:47.961 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-22T23:57:47.966 DEBUG:teuthology.orchestra.run.smithi103:> sudo modprobe fuse 2024-05-22T23:57:48.066 DEBUG:teuthology.orchestra.run.smithi103:> cat /proc/self/mounts | awk '{print $2}' 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/proc 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/sys 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/dev 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/security 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/shm 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/pts 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/run 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/cgroup 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/pstore 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/bpf 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/ 2024-05-22T23:57:48.112 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/selinux 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/proc/sys/fs/binfmt_misc 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/mqueue 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/dev/hugepages 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/debug 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/tracing 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/fs/fuse/connections 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/sys/kernel/config 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-sysctl.service 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/nfs/rpc_pipefs 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/run/user/1000 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/ceph 2024-05-22T23:57:48.113 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/df49e73f547dcdc4b3bf4ffb3d49798897278b98e66f874faba555488bbbbef9/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/c16298afa3412001ed74ae97c92f34e455e60843994da2b3bb055af913f0acbe/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/run/user/0 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/proc/sys/fs/binfmt_misc 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/eb0b14552f4162df842230224c9bb075d516aec593dfe48a20a9a8fcc5675f3f/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/57498e415745938ffdc754bae2dd4a6d7199bbf80a5aada7fd586adf7b006141/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/a97f95810007a039a5e2e1e8c1a0d0cd306316a8748ceb697d2ac83def7d34e8/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/94dd9aa425d40df3f500aaa4fe40647c71699fcf33abe1ad2df376f1433fd97c/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/fa63bce37f59009fed33a1c9d9ab131a7716ae70d0dae61aa63d07e6cc4d7541/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/de85e5bff8327540b841416f25e095ca0412b39abcd98306eeddaff9fe70e612/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/da0d052c753aa2fa6d3e2d35ba4b6eb02eee5d23ebb3e5dfca4e883f20fe215d/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/var/lib/containers/storage/overlay/af31f68906530943afa0697fe494e127a5d18abe9b7f4d1dbaa7936606a0dc16/merged 2024-05-22T23:57:48.114 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns 2024-05-22T23:57:48.115 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-22T23:57:48.115 INFO:teuthology.orchestra.run.smithi103.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-22T23:57:48.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:48.116 DEBUG:teuthology.orchestra.run.smithi103:> ls /sys/fs/fuse/connections 2024-05-22T23:57:48.185 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi103.stderr:2024-05-22T23:57:48.182+0000 7f3df4271480 -1 init, newargv = 0x562eafe4b630 newargc=15 2024-05-22T23:57:48.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi103.stderr:ceph-fuse[75500]: starting ceph client 2024-05-22T23:57:48.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi103.stderr:ceph-fuse[75500]: starting fuse 2024-05-22T23:57:48.200 INFO:teuthology.orchestra.run.smithi103.stdout:119 2024-05-22T23:57:48.200 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-05-22T23:57:48.200 DEBUG:teuthology.orchestra.run.smithi103:> sudo stdin-killer -- python3 -c ' 2024-05-22T23:57:48.200 DEBUG:teuthology.orchestra.run.smithi103:> import glob 2024-05-22T23:57:48.200 DEBUG:teuthology.orchestra.run.smithi103:> import re 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> import os 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> import subprocess 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> def _find_admin_socket(client_name): 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> files = glob.glob(asok_path) 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> # Given a non-glob path, it better be there 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> if "*" not in asok_path: 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> assert(len(files) == 1) 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> return files[0] 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> 2024-05-22T23:57:48.201 DEBUG:teuthology.orchestra.run.smithi103:> for f in files: 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> contents = proc_f.read() 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> if mountpoint in contents: 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> return f 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> print(_find_admin_socket("client.admin")) 2024-05-22T23:57:48.202 DEBUG:teuthology.orchestra.run.smithi103:> ' 2024-05-22T23:57:48.296 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:48 smithi103 ceph-mon[31370]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-22T23:57:48.390 INFO:teuthology.orchestra.run.smithi103.stdout:/var/run/ceph/ceph-client.admin.75500.asok 2024-05-22T23:57:48.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-22T23:57:48.409 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75500.asok 2024-05-22T23:57:48.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:48.409 DEBUG:teuthology.orchestra.run.smithi103:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75500.asok status 2024-05-22T23:57:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:48 smithi105 ceph-mon[36426]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-22T23:57:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:48 smithi174 ceph-mon[36179]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-22T23:57:48.623 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "metadata": { 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_sha1": "2e1f76222e88afe0f1cda37775b2d45d88289794", 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 19.0.0-2765-g2e1f7622 (2e1f76222e88afe0f1cda37775b2d45d88289794) squid (rc)", 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "entity_id": "admin", 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "hostname": "smithi103", 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "pid": "75500", 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "root": "/" 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-22T23:57:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "dentry_count": 0, 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "dentry_pinned_count": 0, 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 14601, 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "inst": { 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "name": { 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "client", 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "num": 14601 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": { 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "192.168.0.1:0", 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1269728331 2024-05-22T23:57:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": { 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "192.168.0.1:0", 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1269728331 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "inst_str": "client.14601 192.168.0.1:0/1269728331", 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "addr_str": "192.168.0.1:0/1269728331", 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "inode_count": 1, 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_epoch": 17, 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_epoch": 91, 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_epoch_barrier": 0, 2024-05-22T23:57:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "blocklisted": false, 2024-05-22T23:57:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs" 2024-05-22T23:57:48.627 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-22T23:57:48.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:48.642 DEBUG:teuthology.orchestra.run.smithi103:> stat --file-system '--printf=%T 2024-05-22T23:57:48.642 DEBUG:teuthology.orchestra.run.smithi103:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-22T23:57:48.688 INFO:teuthology.orchestra.run.smithi103.stdout:fuseblk 2024-05-22T23:57:48.688 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-22T23:57:48.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:48.688 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-22T23:57:48.796 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-22T23:57:48.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-22T23:57:48.920 INFO:teuthology.orchestra.run.smithi103.stdout:# file: . 2024-05-22T23:57:48.920 INFO:teuthology.orchestra.run.smithi103.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-22T23:57:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:48.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-22T23:57:48.930 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:57:48.930 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-22T23:57:48.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:57:49.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.194+0000 7f3acf800640 1 -- 172.21.15.103:0/739503628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac8100180 msgr2=0x7f3ac81005a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.194+0000 7f3acf800640 1 --2- 172.21.15.103:0/739503628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac8100180 0x7f3ac81005a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009900 tx=0x7f3abc02a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.194+0000 7f3acf800640 1 -- 172.21.15.103:0/739503628 shutdown_connections 2024-05-22T23:57:49.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.194+0000 7f3acf800640 1 --2- 172.21.15.103:0/739503628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ac8101480 0x7f3ac8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.194+0000 7f3acf800640 1 --2- 172.21.15.103:0/739503628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac8100ae0 0x7f3ac8100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.194+0000 7f3acf800640 1 --2- 172.21.15.103:0/739503628 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac8100180 0x7f3ac81005a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.194+0000 7f3acf800640 1 -- 172.21.15.103:0/739503628 >> 172.21.15.103:0/739503628 conn(0x7f3ac80fbd20 msgr2=0x7f3ac80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:49.199 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.194+0000 7f3acf800640 1 -- 172.21.15.103:0/739503628 shutdown_connections 2024-05-22T23:57:49.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.195+0000 7f3acf800640 1 -- 172.21.15.103:0/739503628 wait complete. 2024-05-22T23:57:49.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acf800640 1 Processor -- start 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acf800640 1 -- start start 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acf800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac8100180 0x7f3ac8199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acf800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ac8100ae0 0x7f3ac819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acf800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac8101480 0x7f3ac81a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acf800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac810f710 con 0x7f3ac8100180 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acf800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac810f590 con 0x7f3ac8101480 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acf800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac810f890 con 0x7f3ac8100ae0 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3ac7e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ac8100ae0 0x7f3ac819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3ac7e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ac8100ae0 0x7f3ac819a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:38918/0 (socket says 172.21.15.103:38918) 2024-05-22T23:57:49.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3ac7e00640 1 -- 172.21.15.103:0/1166690873 learned_addr learned my addr 172.21.15.103:0/1166690873 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acce00640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac8100180 0x7f3ac8199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.196+0000 7f3acd800640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac8101480 0x7f3ac81a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac7e00640 1 -- 172.21.15.103:0/1166690873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac8101480 msgr2=0x7f3ac81a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac7e00640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac8101480 0x7f3ac81a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac7e00640 1 -- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac8100180 msgr2=0x7f3ac8199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac7e00640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac8100180 0x7f3ac8199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac7e00640 1 -- 172.21.15.103:0/1166690873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac81a19d0 con 0x7f3ac8100ae0 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac7e00640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ac8100ae0 0x7f3ac819a2f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400e9f0 tx=0x7f3ab400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac5600640 1 -- 172.21.15.103:0/1166690873 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab400cd80 con 0x7f3ac8100ae0 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac81a1cc0 con 0x7f3ac8100ae0 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac5600640 1 -- 172.21.15.103:0/1166690873 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ab4010910 con 0x7f3ac8100ae0 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.197+0000 7f3ac5600640 1 -- 172.21.15.103:0/1166690873 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ab40188f0 con 0x7f3ac8100ae0 2024-05-22T23:57:49.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.198+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac81a21c0 con 0x7f3ac8100ae0 2024-05-22T23:57:49.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.199+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac810b0b0 con 0x7f3ac8100ae0 2024-05-22T23:57:49.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.201+0000 7f3ac5600640 1 -- 172.21.15.103:0/1166690873 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3ab4010ac0 con 0x7f3ac8100ae0 2024-05-22T23:57:49.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.202+0000 7f3ac5600640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3ab00a2b90 0x7f3ab00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:49.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.202+0000 7f3ac5600640 1 -- 172.21.15.103:0/1166690873 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f3ab40cfb80 con 0x7f3ac8100ae0 2024-05-22T23:57:49.206 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.202+0000 7f3acce00640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3ab00a2b90 0x7f3ab00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:49.207 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.203+0000 7f3acce00640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3ab00a2b90 0x7f3ab00a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0314e0 tx=0x7f3abc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:49.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.204+0000 7f3ac5600640 1 -- 172.21.15.103:0/1166690873 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3ab409c1f0 con 0x7f3ac8100ae0 2024-05-22T23:57:49.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.416+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3ac802d400 con 0x7f3ac8100ae0 2024-05-22T23:57:49.422 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.419+0000 7f3ac5600640 1 -- 172.21.15.103:0/1166690873 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7f3ab40a04f0 con 0x7f3ac8100ae0 2024-05-22T23:57:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:49.423 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:57:49.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3ab00a2b90 msgr2=0x7f3ab00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3ab00a2b90 0x7f3ab00a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3abc0314e0 tx=0x7f3abc000bc0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ac8100ae0 msgr2=0x7f3ac819a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ac8100ae0 0x7f3ac819a2f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3ab400e9f0 tx=0x7f3ab400eeb0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 shutdown_connections 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3ab00a2b90 0x7f3ab00a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac8101480 0x7f3ac81a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ac8100ae0 0x7f3ac819a2f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 --2- 172.21.15.103:0/1166690873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3ac8100180 0x7f3ac8199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.421+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 >> 172.21.15.103:0/1166690873 conn(0x7f3ac80fbd20 msgr2=0x7f3ac80fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.422+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 shutdown_connections 2024-05-22T23:57:49.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.422+0000 7f3acf800640 1 -- 172.21.15.103:0/1166690873 wait complete. 2024-05-22T23:57:49.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-22T23:57:49.447 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-22T23:57:49.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 -- 172.21.15.103:0/3612642483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac101440 msgr2=0x7f2cac1018c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 --2- 172.21.15.103:0/3612642483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac101440 0x7f2cac1018c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4009900 tx=0x7f2ca402a410 comp rx=0 tx=0).stop 2024-05-22T23:57:49.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 -- 172.21.15.103:0/3612642483 shutdown_connections 2024-05-22T23:57:49.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 --2- 172.21.15.103:0/3612642483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac101440 0x7f2cac1018c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 --2- 172.21.15.103:0/3612642483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cac100aa0 0x7f2cac100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 --2- 172.21.15.103:0/3612642483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cac100140 0x7f2cac100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 -- 172.21.15.103:0/3612642483 >> 172.21.15.103:0/3612642483 conn(0x7f2cac0fbd40 msgr2=0x7f2cac0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:49.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 -- 172.21.15.103:0/3612642483 shutdown_connections 2024-05-22T23:57:49.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.632+0000 7f2cb4000640 1 -- 172.21.15.103:0/3612642483 wait complete. 2024-05-22T23:57:49.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb4000640 1 Processor -- start 2024-05-22T23:57:49.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb4000640 1 -- start start 2024-05-22T23:57:49.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb4000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cac100140 0x7f2cac199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:49.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb4000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac100aa0 0x7f2cac19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:49.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb4000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cac101440 0x7f2cac1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:49.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb4000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac10f720 con 0x7f2cac100aa0 2024-05-22T23:57:49.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb4000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac10f5a0 con 0x7f2cac100140 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb4000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2cac10f8a0 con 0x7f2cac101440 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.633+0000 7f2cb0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac100aa0 0x7f2cac19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb1600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cac100140 0x7f2cac199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb0c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac100aa0 0x7f2cac19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44448/0 (socket says 172.21.15.103:44448) 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb0c00640 1 -- 172.21.15.103:0/591451328 learned_addr learned my addr 172.21.15.103:0/591451328 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb2000640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cac101440 0x7f2cac1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb0c00640 1 -- 172.21.15.103:0/591451328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cac101440 msgr2=0x7f2cac1a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb0c00640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cac101440 0x7f2cac1a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb0c00640 1 -- 172.21.15.103:0/591451328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cac100140 msgr2=0x7f2cac199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb0c00640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cac100140 0x7f2cac199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb0c00640 1 -- 172.21.15.103:0/591451328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cac1a1990 con 0x7f2cac100aa0 2024-05-22T23:57:49.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb1600640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cac100140 0x7f2cac199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:49.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.634+0000 7f2cb0c00640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac100aa0 0x7f2cac19a2b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800e9f0 tx=0x7f2c9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:49.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.635+0000 7f2caa000640 1 -- 172.21.15.103:0/591451328 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9800cd80 con 0x7f2cac100aa0 2024-05-22T23:57:49.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.635+0000 7f2caa000640 1 -- 172.21.15.103:0/591451328 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c98010910 con 0x7f2cac100aa0 2024-05-22T23:57:49.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.635+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cac1a1c80 con 0x7f2cac100aa0 2024-05-22T23:57:49.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.635+0000 7f2caa000640 1 -- 172.21.15.103:0/591451328 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c98018990 con 0x7f2cac100aa0 2024-05-22T23:57:49.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.635+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cac1a21d0 con 0x7f2cac100aa0 2024-05-22T23:57:49.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.636+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cac10b030 con 0x7f2cac100aa0 2024-05-22T23:57:49.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.639+0000 7f2caa000640 1 -- 172.21.15.103:0/591451328 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f2c98010ac0 con 0x7f2cac100aa0 2024-05-22T23:57:49.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.640+0000 7f2caa000640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2c940a2b90 0x7f2c940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:49.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.641+0000 7f2caa000640 1 -- 172.21.15.103:0/591451328 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f2c980cf5d0 con 0x7f2cac100aa0 2024-05-22T23:57:49.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.641+0000 7f2cb1600640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2c940a2b90 0x7f2c940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:49.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.642+0000 7f2cb1600640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2c940a2b90 0x7f2c940a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0005fd0 tx=0x7f2ca0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:49.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.645+0000 7f2caa000640 1 -- 172.21.15.103:0/591451328 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2c9809bc40 con 0x7f2cac100aa0 2024-05-22T23:57:49.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:49 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1166690873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:49.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:49 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1166690873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.859+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2cac062510 con 0x7f2cac100aa0 2024-05-22T23:57:49.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.860+0000 7f2caa000640 1 -- 172.21.15.103:0/591451328 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f2c9809ff40 con 0x7f2cac100aa0 2024-05-22T23:57:49.864 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2c940a2b90 msgr2=0x7f2c940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2c940a2b90 0x7f2c940a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0005fd0 tx=0x7f2ca0009500 comp rx=0 tx=0).stop 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac100aa0 msgr2=0x7f2cac19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac100aa0 0x7f2cac19a2b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2c9800e9f0 tx=0x7f2c9800eeb0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 shutdown_connections 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2c940a2b90 0x7f2c940a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cac101440 0x7f2cac1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cac100aa0 0x7f2cac19a2b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 --2- 172.21.15.103:0/591451328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cac100140 0x7f2cac199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 >> 172.21.15.103:0/591451328 conn(0x7f2cac0fbd40 msgr2=0x7f2cac0fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.862+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 shutdown_connections 2024-05-22T23:57:49.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:49.863+0000 7f2cb4000640 1 -- 172.21.15.103:0/591451328 wait complete. 2024-05-22T23:57:49.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:49 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1166690873' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:49.885 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-22T23:57:50.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.088+0000 7f780a400640 1 -- 172.21.15.103:0/3852602430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804103240 msgr2=0x7f7804109ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.088+0000 7f780a400640 1 --2- 172.21.15.103:0/3852602430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804103240 0x7f7804109ad0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009ca0 tx=0x7f77fc02a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.088+0000 7f780a400640 1 -- 172.21.15.103:0/3852602430 shutdown_connections 2024-05-22T23:57:50.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.088+0000 7f780a400640 1 --2- 172.21.15.103:0/3852602430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804103240 0x7f7804109ad0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.088+0000 7f780a400640 1 --2- 172.21.15.103:0/3852602430 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7804102880 0x7f7804102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.088+0000 7f780a400640 1 --2- 172.21.15.103:0/3852602430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7804101690 0x7f7804101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.088+0000 7f780a400640 1 -- 172.21.15.103:0/3852602430 >> 172.21.15.103:0/3852602430 conn(0x7f78040fcc00 msgr2=0x7f78040ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:50.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.088+0000 7f780a400640 1 -- 172.21.15.103:0/3852602430 shutdown_connections 2024-05-22T23:57:50.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.089+0000 7f780a400640 1 -- 172.21.15.103:0/3852602430 wait complete. 2024-05-22T23:57:50.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.089+0000 7f780a400640 1 Processor -- start 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f780a400640 1 -- start start 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f780a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804101690 0x7f7804199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f780a400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7804102880 0x7f780419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f780a400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7804103240 0x7f78041a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f780a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f780410d310 con 0x7f7804101690 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f780a400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f780410d190 con 0x7f7804102880 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f780a400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f780410d490 con 0x7f7804103240 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f7809000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804101690 0x7f7804199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f7809000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804101690 0x7f7804199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44468/0 (socket says 172.21.15.103:44468) 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f7809000640 1 -- 172.21.15.103:0/306450176 learned_addr learned my addr 172.21.15.103:0/306450176 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:50.096 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f7803e00640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7804102880 0x7f780419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.090+0000 7f7809a00640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7804103240 0x7f78041a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7809000640 1 -- 172.21.15.103:0/306450176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7804103240 msgr2=0x7f78041a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7809000640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7804103240 0x7f78041a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7809000640 1 -- 172.21.15.103:0/306450176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7804102880 msgr2=0x7f780419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7809000640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7804102880 0x7f780419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7809000640 1 -- 172.21.15.103:0/306450176 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78041a1730 con 0x7f7804101690 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7809000640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804101690 0x7f7804199b10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f77f400e9c0 tx=0x7f77f400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7801600640 1 -- 172.21.15.103:0/306450176 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f400cd80 con 0x7f7804101690 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7801600640 1 -- 172.21.15.103:0/306450176 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77f40035f0 con 0x7f7804101690 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78041a1a20 con 0x7f7804101690 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.091+0000 7f7801600640 1 -- 172.21.15.103:0/306450176 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4010690 con 0x7f7804101690 2024-05-22T23:57:50.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.092+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78041a1f70 con 0x7f7804101690 2024-05-22T23:57:50.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.093+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78041a2200 con 0x7f7804101690 2024-05-22T23:57:50.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.095+0000 7f7801600640 1 -- 172.21.15.103:0/306450176 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f77f40037a0 con 0x7f7804101690 2024-05-22T23:57:50.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.096+0000 7f7801600640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f77e80a2c10 0x7f77e80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.096+0000 7f7801600640 1 -- 172.21.15.103:0/306450176 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f77f40cf5e0 con 0x7f7804101690 2024-05-22T23:57:50.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.096+0000 7f7803e00640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f77e80a2c10 0x7f77e80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.097+0000 7f7803e00640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f77e80a2c10 0x7f77e80a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0095f0 tx=0x7f77ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:50.104 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.100+0000 7f7801600640 1 -- 172.21.15.103:0/306450176 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f77f409c2a0 con 0x7f7804101690 2024-05-22T23:57:50.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.301+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f78041a23d0 con 0x7f7804101690 2024-05-22T23:57:50.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.302+0000 7f7801600640 1 -- 172.21.15.103:0/306450176 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f77f409bba0 con 0x7f7804101690 2024-05-22T23:57:50.306 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.304+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f77e80a2c10 msgr2=0x7f77e80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.304+0000 7f780a400640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f77e80a2c10 0x7f77e80a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f77ec0095f0 tx=0x7f77ec005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.304+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804101690 msgr2=0x7f7804199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.304+0000 7f780a400640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804101690 0x7f7804199b10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f77f400e9c0 tx=0x7f77f400ee80 comp rx=0 tx=0).stop 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.305+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 shutdown_connections 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.305+0000 7f780a400640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f77e80a2c10 0x7f77e80a50d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.305+0000 7f780a400640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7804103240 0x7f78041a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.305+0000 7f780a400640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7804102880 0x7f780419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.305+0000 7f780a400640 1 --2- 172.21.15.103:0/306450176 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7804101690 0x7f7804199b10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.305+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 >> 172.21.15.103:0/306450176 conn(0x7f78040fcc00 msgr2=0x7f7804073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.305+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 shutdown_connections 2024-05-22T23:57:50.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.305+0000 7f780a400640 1 -- 172.21.15.103:0/306450176 wait complete. 2024-05-22T23:57:50.328 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-22T23:57:50.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 -- 172.21.15.103:0/784740289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4103ad0 msgr2=0x7f7bc4103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/784740289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4103ad0 0x7f7bc4103ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009ca0 tx=0x7f7bb402a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 -- 172.21.15.103:0/784740289 shutdown_connections 2024-05-22T23:57:50.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/784740289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7bc4105680 0x7f7bc410bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/784740289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc4104cc0 0x7f7bc4105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/784740289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4103ad0 0x7f7bc4103ef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 -- 172.21.15.103:0/784740289 >> 172.21.15.103:0/784740289 conn(0x7f7bc40ff060 msgr2=0x7f7bc4101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:50.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 -- 172.21.15.103:0/784740289 shutdown_connections 2024-05-22T23:57:50.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.529+0000 7f7bc9e00640 1 -- 172.21.15.103:0/784740289 wait complete. 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.530+0000 7f7bc9e00640 1 Processor -- start 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc9e00640 1 -- start start 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc9e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc4103ad0 0x7f7bc4199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc9e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7bc4104cc0 0x7f7bc419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4105680 0x7f7bc41a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc9e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc410f720 con 0x7f7bc4105680 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc9e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc410f5a0 con 0x7f7bc4104cc0 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc9e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7bc410f8a0 con 0x7f7bc4103ad0 2024-05-22T23:57:50.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4105680 0x7f7bc41a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc2000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7bc4104cc0 0x7f7bc419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc2a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc4103ad0 0x7f7bc4199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc2000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7bc4104cc0 0x7f7bc419a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:39576/0 (socket says 172.21.15.103:39576) 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc2000640 1 -- 172.21.15.103:0/2333024744 learned_addr learned my addr 172.21.15.103:0/2333024744 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.531+0000 7f7bc3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4105680 0x7f7bc41a13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44482/0 (socket says 172.21.15.103:44482) 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc2a00640 1 -- 172.21.15.103:0/2333024744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7bc4104cc0 msgr2=0x7f7bc419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc2a00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7bc4104cc0 0x7f7bc419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc2a00640 1 -- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4105680 msgr2=0x7f7bc41a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc2a00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4105680 0x7f7bc41a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc2a00640 1 -- 172.21.15.103:0/2333024744 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc41a1a20 con 0x7f7bc4103ad0 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc2000640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7bc4104cc0 0x7f7bc419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:50.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc2a00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc4103ad0 0x7f7bc4199e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009bf0 tx=0x7f7bb4031ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:50.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc8a00640 1 -- 172.21.15.103:0/2333024744 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb402fa20 con 0x7f7bc4103ad0 2024-05-22T23:57:50.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc8a00640 1 -- 172.21.15.103:0/2333024744 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bb402fbc0 con 0x7f7bc4103ad0 2024-05-22T23:57:50.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bc41a1cb0 con 0x7f7bc4103ad0 2024-05-22T23:57:50.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.532+0000 7f7bc8a00640 1 -- 172.21.15.103:0/2333024744 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bb4046750 con 0x7f7bc4103ad0 2024-05-22T23:57:50.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.533+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bc41a21a0 con 0x7f7bc4103ad0 2024-05-22T23:57:50.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.534+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bc410ae70 con 0x7f7bc4103ad0 2024-05-22T23:57:50.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.536+0000 7f7bc8a00640 1 -- 172.21.15.103:0/2333024744 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f7bb402fd70 con 0x7f7bc4103ad0 2024-05-22T23:57:50.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.538+0000 7f7bc8a00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ba80a2c10 0x7f7ba80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.538+0000 7f7bc8a00640 1 -- 172.21.15.103:0/2333024744 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f7bb40f0610 con 0x7f7bc4103ad0 2024-05-22T23:57:50.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.538+0000 7f7bc2000640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ba80a2c10 0x7f7ba80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.539+0000 7f7bc2000640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ba80a2c10 0x7f7ba80a50d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0095f0 tx=0x7f7bac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:50.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.540+0000 7f7bc8a00640 1 -- 172.21.15.103:0/2333024744 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7bb40bd2d0 con 0x7f7bc4103ad0 2024-05-22T23:57:50.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:50 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/591451328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:50.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:50 smithi103 ceph-mon[31370]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-22T23:57:50.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:50 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/306450176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:57:50.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.741+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7bc4062510 con 0x7f7bc4103ad0 2024-05-22T23:57:50.746 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.742+0000 7f7bc8a00640 1 -- 172.21.15.103:0/2333024744 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f7bb40bcbd0 con 0x7f7bc4103ad0 2024-05-22T23:57:50.746 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-22T23:57:50.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.744+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ba80a2c10 msgr2=0x7f7ba80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.744+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ba80a2c10 0x7f7ba80a50d0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0095f0 tx=0x7f7bac005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:50.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.744+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc4103ad0 msgr2=0x7f7bc4199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.744+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc4103ad0 0x7f7bc4199e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7bb4009bf0 tx=0x7f7bb4031ee0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.745+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 shutdown_connections 2024-05-22T23:57:50.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.745+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ba80a2c10 0x7f7ba80a50d0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.745+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7bc4105680 0x7f7bc41a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.745+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7bc4104cc0 0x7f7bc419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.745+0000 7f7bc9e00640 1 --2- 172.21.15.103:0/2333024744 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7bc4103ad0 0x7f7bc4199e00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.745+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 >> 172.21.15.103:0/2333024744 conn(0x7f7bc40ff060 msgr2=0x7f7bc4108040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:50.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.745+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 shutdown_connections 2024-05-22T23:57:50.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.745+0000 7f7bc9e00640 1 -- 172.21.15.103:0/2333024744 wait complete. 2024-05-22T23:57:50.768 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-22T23:57:50.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:50 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/591451328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:50.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:50 smithi105 ceph-mon[36426]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-22T23:57:50.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:50 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/306450176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:57:50.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:50 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/591451328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:50.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:50 smithi174 ceph-mon[36179]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-22T23:57:50.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:50 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/306450176' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:57:50.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 -- 172.21.15.103:0/2516632002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72c410ba30 msgr2=0x7f72c410de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 --2- 172.21.15.103:0/2516632002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72c410ba30 0x7f72c410de20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f72ac009860 tx=0x7f72ac02a3d0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 -- 172.21.15.103:0/2516632002 shutdown_connections 2024-05-22T23:57:50.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 --2- 172.21.15.103:0/2516632002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c410e3b0 0x7f72c4110870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 --2- 172.21.15.103:0/2516632002 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72c410ba30 0x7f72c410de20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 --2- 172.21.15.103:0/2516632002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c4109100 0x7f72c410b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 -- 172.21.15.103:0/2516632002 >> 172.21.15.103:0/2516632002 conn(0x7f72c406ea80 msgr2=0x7f72c4070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:50.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 -- 172.21.15.103:0/2516632002 shutdown_connections 2024-05-22T23:57:50.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.958+0000 7f72cb000640 1 -- 172.21.15.103:0/2516632002 wait complete. 2024-05-22T23:57:50.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.959+0000 7f72cb000640 1 Processor -- start 2024-05-22T23:57:50.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.959+0000 7f72cb000640 1 -- start start 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72cb000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72c4109100 0x7f72c41a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72cb000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c410ba30 0x7f72c41a0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72cb000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c410e3b0 0x7f72c41a79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72cb000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72c4115db0 con 0x7f72c410e3b0 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72cb000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f72c4115c30 con 0x7f72c4109100 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72cb000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f72c4115f30 con 0x7f72c410ba30 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72c9000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c410e3b0 0x7f72c41a79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72c3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72c4109100 0x7f72c41a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72c9000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c410e3b0 0x7f72c41a79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44500/0 (socket says 172.21.15.103:44500) 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72c9000640 1 -- 172.21.15.103:0/3115409376 learned_addr learned my addr 172.21.15.103:0/3115409376 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.960+0000 7f72bb400640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c410ba30 0x7f72c41a0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c9000640 1 -- 172.21.15.103:0/3115409376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c410ba30 msgr2=0x7f72c41a0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c9000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c410ba30 0x7f72c41a0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c9000640 1 -- 172.21.15.103:0/3115409376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72c4109100 msgr2=0x7f72c41a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c9000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72c4109100 0x7f72c41a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c9000640 1 -- 172.21.15.103:0/3115409376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c41a8040 con 0x7f72c410e3b0 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c9000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c410e3b0 0x7f72c41a79d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009fd0 tx=0x7f72bc00def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c1600640 1 -- 172.21.15.103:0/3115409376 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc017070 con 0x7f72c410e3b0 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c41a8330 con 0x7f72c410e3b0 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c1600640 1 -- 172.21.15.103:0/3115409376 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72bc003e50 con 0x7f72c410e3b0 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.961+0000 7f72c1600640 1 -- 172.21.15.103:0/3115409376 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc010460 con 0x7f72c410e3b0 2024-05-22T23:57:50.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.962+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c41a8850 con 0x7f72c410e3b0 2024-05-22T23:57:50.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.963+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c4070570 con 0x7f72c410e3b0 2024-05-22T23:57:50.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.965+0000 7f72c1600640 1 -- 172.21.15.103:0/3115409376 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f72bc003970 con 0x7f72c410e3b0 2024-05-22T23:57:50.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.966+0000 7f72c1600640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72a80a2c10 0x7f72a80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:50.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.966+0000 7f72c1600640 1 -- 172.21.15.103:0/3115409376 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f72bc0ceea0 con 0x7f72c410e3b0 2024-05-22T23:57:50.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.966+0000 7f72c3e00640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72a80a2c10 0x7f72a80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:50.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.967+0000 7f72c3e00640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72a80a2c10 0x7f72a80a50d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f72b4005fd0 tx=0x7f72b4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:50.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:50.969+0000 7f72c1600640 1 -- 172.21.15.103:0/3115409376 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f72bc09bbe0 con 0x7f72c410e3b0 2024-05-22T23:57:51.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.170+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f72c402d200 con 0x7f72c410e3b0 2024-05-22T23:57:51.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.171+0000 7f72c1600640 1 -- 172.21.15.103:0/3115409376 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f72bc09b4e0 con 0x7f72c410e3b0 2024-05-22T23:57:51.175 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-22T23:57:51.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72a80a2c10 msgr2=0x7f72a80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72a80a2c10 0x7f72a80a50d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f72b4005fd0 tx=0x7f72b4009260 comp rx=0 tx=0).stop 2024-05-22T23:57:51.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c410e3b0 msgr2=0x7f72c41a79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c410e3b0 0x7f72c41a79d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009fd0 tx=0x7f72bc00def0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 shutdown_connections 2024-05-22T23:57:51.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72a80a2c10 0x7f72a80a50d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c410e3b0 0x7f72c41a79d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f72c410ba30 0x7f72c41a0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 --2- 172.21.15.103:0/3115409376 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72c4109100 0x7f72c41a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 >> 172.21.15.103:0/3115409376 conn(0x7f72c406ea80 msgr2=0x7f72c4109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:51.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 shutdown_connections 2024-05-22T23:57:51.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.174+0000 7f72cb000640 1 -- 172.21.15.103:0/3115409376 wait complete. 2024-05-22T23:57:51.198 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-22T23:57:51.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 -- 172.21.15.103:0/3383504644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa120104b30 msgr2=0x7fa120106f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 --2- 172.21.15.103:0/3383504644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa120104b30 0x7fa120106f20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa108009900 tx=0x7fa10802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 -- 172.21.15.103:0/3383504644 shutdown_connections 2024-05-22T23:57:51.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 --2- 172.21.15.103:0/3383504644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa120107460 0x7fa1201098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 --2- 172.21.15.103:0/3383504644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa120104b30 0x7fa120106f20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 --2- 172.21.15.103:0/3383504644 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa120102200 0x7fa1201045f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 -- 172.21.15.103:0/3383504644 >> 172.21.15.103:0/3383504644 conn(0x7fa1200fbc10 msgr2=0x7fa1200fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:51.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 -- 172.21.15.103:0/3383504644 shutdown_connections 2024-05-22T23:57:51.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.389+0000 7fa124a00640 1 -- 172.21.15.103:0/3383504644 wait complete. 2024-05-22T23:57:51.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.390+0000 7fa124a00640 1 Processor -- start 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.390+0000 7fa124a00640 1 -- start start 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.390+0000 7fa124a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa120102200 0x7fa120199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.390+0000 7fa124a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa120104b30 0x7fa12019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.390+0000 7fa124a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa120107460 0x7fa1201a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.390+0000 7fa124a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa12010d3f0 con 0x7fa120107460 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.390+0000 7fa124a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa12010d270 con 0x7fa120102200 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.390+0000 7fa124a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa12010d570 con 0x7fa120104b30 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa120107460 0x7fa1201a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa120107460 0x7fa1201a1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44516/0 (socket says 172.21.15.103:44516) 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 -- 172.21.15.103:0/3555352260 learned_addr learned my addr 172.21.15.103:0/3555352260 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11f400640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa120102200 0x7fa120199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa117e00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa120104b30 0x7fa12019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 -- 172.21.15.103:0/3555352260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa120104b30 msgr2=0x7fa12019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa120104b30 0x7fa12019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 -- 172.21.15.103:0/3555352260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa120102200 msgr2=0x7fa120199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa120102200 0x7fa120199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 -- 172.21.15.103:0/3555352260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1201a18e0 con 0x7fa120107460 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11f400640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa120102200 0x7fa120199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.391+0000 7fa11fe00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa120107460 0x7fa1201a1240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa11800efd0 tx=0x7fa11800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.392+0000 7fa11cc00640 1 -- 172.21.15.103:0/3555352260 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa118003b70 con 0x7fa120107460 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.392+0000 7fa11cc00640 1 -- 172.21.15.103:0/3555352260 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa118003d10 con 0x7fa120107460 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.392+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1201a1bd0 con 0x7fa120107460 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.392+0000 7fa11cc00640 1 -- 172.21.15.103:0/3555352260 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1180109f0 con 0x7fa120107460 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.392+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1201a20f0 con 0x7fa120107460 2024-05-22T23:57:51.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.393+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa120108fb0 con 0x7fa120107460 2024-05-22T23:57:51.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.395+0000 7fa11cc00640 1 -- 172.21.15.103:0/3555352260 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa118004020 con 0x7fa120107460 2024-05-22T23:57:51.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.396+0000 7fa11cc00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa1040a2c10 0x7fa1040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:51.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.396+0000 7fa11cc00640 1 -- 172.21.15.103:0/3555352260 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa1180cf670 con 0x7fa120107460 2024-05-22T23:57:51.400 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.397+0000 7fa11f400640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa1040a2c10 0x7fa1040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:51.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.397+0000 7fa11f400640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa1040a2c10 0x7fa1040a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa12019ab70 tx=0x7fa110005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:51.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.401+0000 7fa11cc00640 1 -- 172.21.15.103:0/3555352260 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa11809c330 con 0x7fa120107460 2024-05-22T23:57:51.618 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.614+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa12002d200 con 0x7fa120107460 2024-05-22T23:57:51.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.615+0000 7fa11cc00640 1 -- 172.21.15.103:0/3555352260 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+601 (secure 0 0 0) 0x7fa11809bc30 con 0x7fa120107460 2024-05-22T23:57:51.619 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3190009536 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:51.619 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1448560264 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:51.619 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/274656845 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:51.619 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2141778982 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:51.619 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2592202413 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:51.620 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/3762323194 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:51.620 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/3495336128 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:51.620 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/3762323194 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:51.620 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/3495336128 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:51.620 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1425794392 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:51.620 INFO:teuthology.orchestra.run.smithi103.stderr:listed 10 entries 2024-05-22T23:57:51.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.617+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa1040a2c10 msgr2=0x7fa1040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.617+0000 7fa124a00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa1040a2c10 0x7fa1040a50d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa12019ab70 tx=0x7fa110005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:51.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.617+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa120107460 msgr2=0x7fa1201a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.617+0000 7fa124a00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa120107460 0x7fa1201a1240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fa11800efd0 tx=0x7fa11800c6a0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.618+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 shutdown_connections 2024-05-22T23:57:51.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.618+0000 7fa124a00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa1040a2c10 0x7fa1040a50d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.618+0000 7fa124a00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa120107460 0x7fa1201a1240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.618+0000 7fa124a00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa120104b30 0x7fa12019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.618+0000 7fa124a00640 1 --2- 172.21.15.103:0/3555352260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa120102200 0x7fa120199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.618+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 >> 172.21.15.103:0/3555352260 conn(0x7fa1200fbc10 msgr2=0x7fa1201018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:51.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.618+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 shutdown_connections 2024-05-22T23:57:51.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.618+0000 7fa124a00640 1 -- 172.21.15.103:0/3555352260 wait complete. 2024-05-22T23:57:51.639 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:57:51.640 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-22T23:57:51.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:57:51.803 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:51 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2333024744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:57:51.804 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:51 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3115409376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:57:51.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:51 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2333024744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:57:51.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:51 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3115409376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:57:51.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:51 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2333024744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:57:51.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:51 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3115409376' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:57:51.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 -- 172.21.15.103:0/3182116262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a81038f0 msgr2=0x7f85a8103d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 --2- 172.21.15.103:0/3182116262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a81038f0 0x7f85a8103d10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8598009900 tx=0x7f859802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 -- 172.21.15.103:0/3182116262 shutdown_connections 2024-05-22T23:57:51.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 --2- 172.21.15.103:0/3182116262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a81054a0 0x7f85a810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 --2- 172.21.15.103:0/3182116262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85a8104ae0 0x7f85a8104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 --2- 172.21.15.103:0/3182116262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a81038f0 0x7f85a8103d10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 -- 172.21.15.103:0/3182116262 >> 172.21.15.103:0/3182116262 conn(0x7f85a80fee80 msgr2=0x7f85a81012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:51.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 -- 172.21.15.103:0/3182116262 shutdown_connections 2024-05-22T23:57:51.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.932+0000 7f85a7e00640 1 -- 172.21.15.103:0/3182116262 wait complete. 2024-05-22T23:57:51.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a7e00640 1 Processor -- start 2024-05-22T23:57:51.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a7e00640 1 -- start start 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85a81038f0 0x7f85a8199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a8104ae0 0x7f85a819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a81054a0 0x7f85a81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85a810f560 con 0x7f85a8104ae0 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a7e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f85a810f3e0 con 0x7f85a81054a0 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a7e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f85a810f6e0 con 0x7f85a81038f0 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.933+0000 7f85a6000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a8104ae0 0x7f85a819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85a81038f0 0x7f85a8199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85a81038f0 0x7f85a8199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:39044/0 (socket says 172.21.15.103:39044) 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 -- 172.21.15.103:0/308698214 learned_addr learned my addr 172.21.15.103:0/308698214 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a8104ae0 0x7f85a819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44536/0 (socket says 172.21.15.103:44536) 2024-05-22T23:57:51.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a7400640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a81054a0 0x7f85a81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 -- 172.21.15.103:0/308698214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a81054a0 msgr2=0x7f85a81a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a81054a0 0x7f85a81a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 -- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a8104ae0 msgr2=0x7f85a819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a8104ae0 0x7f85a819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 -- 172.21.15.103:0/308698214 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a81a1800 con 0x7f85a81038f0 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.934+0000 7f85a6a00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85a81038f0 0x7f85a8199be0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8598031610 tx=0x7f8598004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.935+0000 7f859f400640 1 -- 172.21.15.103:0/308698214 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f859803f030 con 0x7f85a81038f0 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.935+0000 7f859f400640 1 -- 172.21.15.103:0/308698214 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8598003ad0 con 0x7f85a81038f0 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.935+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a81a1a90 con 0x7f85a81038f0 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.935+0000 7f859f400640 1 -- 172.21.15.103:0/308698214 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8598035070 con 0x7f85a81038f0 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.935+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a81a1f80 con 0x7f85a81038f0 2024-05-22T23:57:51.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.936+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a810ac90 con 0x7f85a81038f0 2024-05-22T23:57:51.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.938+0000 7f859f400640 1 -- 172.21.15.103:0/308698214 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f8598042030 con 0x7f85a81038f0 2024-05-22T23:57:51.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.939+0000 7f859f400640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f858c0a2b90 0x7f858c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:51.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.939+0000 7f859f400640 1 -- 172.21.15.103:0/308698214 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f85980efb40 con 0x7f85a81038f0 2024-05-22T23:57:51.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.939+0000 7f85a6000640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f858c0a2b90 0x7f858c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:51.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.940+0000 7f85a6000640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f858c0a2b90 0x7f858c0a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f85a819b0f0 tx=0x7f8594005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:51.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:51.942+0000 7f859f400640 1 -- 172.21.15.103:0/308698214 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f85980bc1b0 con 0x7f85a81038f0 2024-05-22T23:57:52.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.130+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85a8062470 con 0x7f85a81038f0 2024-05-22T23:57:52.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.132+0000 7f859f400640 1 -- 172.21.15.103:0/308698214 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7f85980c04b0 con 0x7f85a81038f0 2024-05-22T23:57:52.136 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:57:52.137 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:57:52.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f858c0a2b90 msgr2=0x7f858c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f858c0a2b90 0x7f858c0a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f85a819b0f0 tx=0x7f8594005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:52.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85a81038f0 msgr2=0x7f85a8199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85a81038f0 0x7f85a8199be0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8598031610 tx=0x7f8598004270 comp rx=0 tx=0).stop 2024-05-22T23:57:52.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 shutdown_connections 2024-05-22T23:57:52.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f858c0a2b90 0x7f858c0a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85a81054a0 0x7f85a81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85a8104ae0 0x7f85a819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 --2- 172.21.15.103:0/308698214 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85a81038f0 0x7f85a8199be0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 >> 172.21.15.103:0/308698214 conn(0x7f85a80fee80 msgr2=0x7f85a8107e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:52.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 shutdown_connections 2024-05-22T23:57:52.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.135+0000 7f85a7e00640 1 -- 172.21.15.103:0/308698214 wait complete. 2024-05-22T23:57:52.158 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-22T23:57:52.159 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-22T23:57:52.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 -- 172.21.15.103:0/7176407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c103330 msgr2=0x7ff98c109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 --2- 172.21.15.103:0/7176407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c103330 0x7ff98c109bc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff984009be0 tx=0x7ff98402a6d0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 -- 172.21.15.103:0/7176407 shutdown_connections 2024-05-22T23:57:52.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 --2- 172.21.15.103:0/7176407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c103330 0x7ff98c109bc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 --2- 172.21.15.103:0/7176407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff98c102970 0x7ff98c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 --2- 172.21.15.103:0/7176407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff98c101780 0x7ff98c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 -- 172.21.15.103:0/7176407 >> 172.21.15.103:0/7176407 conn(0x7ff98c0fcd10 msgr2=0x7ff98c0ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:52.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 -- 172.21.15.103:0/7176407 shutdown_connections 2024-05-22T23:57:52.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.352+0000 7ff990c00640 1 -- 172.21.15.103:0/7176407 wait complete. 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff990c00640 1 Processor -- start 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff990c00640 1 -- start start 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff990c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c101780 0x7ff98c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff990c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff98c102970 0x7ff98c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff990c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff98c103330 0x7ff98c1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff990c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c10d390 con 0x7ff98c103330 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff990c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c10d210 con 0x7ff98c102970 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff990c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff98c10d510 con 0x7ff98c101780 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.353+0000 7ff98be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff98c103330 0x7ff98c1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98b400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c101780 0x7ff98c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:52.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff98c103330 0x7ff98c1a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44556/0 (socket says 172.21.15.103:44556) 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98be00640 1 -- 172.21.15.103:0/167269372 learned_addr learned my addr 172.21.15.103:0/167269372 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff982a00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff98c102970 0x7ff98c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98b400640 1 -- 172.21.15.103:0/167269372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff98c102970 msgr2=0x7ff98c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98b400640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff98c102970 0x7ff98c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98b400640 1 -- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff98c103330 msgr2=0x7ff98c1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98b400640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff98c103330 0x7ff98c1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98b400640 1 -- 172.21.15.103:0/167269372 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98c1a1850 con 0x7ff98c101780 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.354+0000 7ff98b400640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c101780 0x7ff98c199c40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00d6a0 tx=0x7ff97c00daa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.355+0000 7ff988c00640 1 -- 172.21.15.103:0/167269372 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c017070 con 0x7ff98c101780 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.355+0000 7ff988c00640 1 -- 172.21.15.103:0/167269372 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff97c009c30 con 0x7ff98c101780 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.355+0000 7ff988c00640 1 -- 172.21.15.103:0/167269372 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c003b50 con 0x7ff98c101780 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.355+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98c1a1b40 con 0x7ff98c101780 2024-05-22T23:57:52.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.355+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff98c1a2040 con 0x7ff98c101780 2024-05-22T23:57:52.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.356+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff98c1a2210 con 0x7ff98c101780 2024-05-22T23:57:52.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.358+0000 7ff988c00640 1 -- 172.21.15.103:0/167269372 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff97c0042b0 con 0x7ff98c101780 2024-05-22T23:57:52.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.359+0000 7ff988c00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff9700a2c10 0x7ff9700a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:52.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.359+0000 7ff988c00640 1 -- 172.21.15.103:0/167269372 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff97c0ceff0 con 0x7ff98c101780 2024-05-22T23:57:52.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.359+0000 7ff982a00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff9700a2c10 0x7ff9700a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:52.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.360+0000 7ff982a00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff9700a2c10 0x7ff9700a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff974005fd0 tx=0x7ff974009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:52.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.362+0000 7ff988c00640 1 -- 172.21.15.103:0/167269372 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff97c009de0 con 0x7ff98c101780 2024-05-22T23:57:52.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.564+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff98c1a25a0 con 0x7ff98c101780 2024-05-22T23:57:52.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.565+0000 7ff988c00640 1 -- 172.21.15.103:0/167269372 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7ff97c09bd50 con 0x7ff98c101780 2024-05-22T23:57:52.569 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.567+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff9700a2c10 msgr2=0x7ff9700a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.567+0000 7ff990c00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff9700a2c10 0x7ff9700a50d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff974005fd0 tx=0x7ff974009260 comp rx=0 tx=0).stop 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.567+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c101780 msgr2=0x7ff98c199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.567+0000 7ff990c00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c101780 0x7ff98c199c40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00d6a0 tx=0x7ff97c00daa0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.568+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 shutdown_connections 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.568+0000 7ff990c00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff9700a2c10 0x7ff9700a50d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.568+0000 7ff990c00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff98c103330 0x7ff98c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.568+0000 7ff990c00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff98c102970 0x7ff98c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.568+0000 7ff990c00640 1 --2- 172.21.15.103:0/167269372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff98c101780 0x7ff98c199c40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.568+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 >> 172.21.15.103:0/167269372 conn(0x7ff98c0fcd10 msgr2=0x7ff98c073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:52.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.568+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 shutdown_connections 2024-05-22T23:57:52.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.568+0000 7ff990c00640 1 -- 172.21.15.103:0/167269372 wait complete. 2024-05-22T23:57:52.591 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-22T23:57:52.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 -- 172.21.15.103:0/818085249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c1100f0 msgr2=0x7fa87c1124e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 --2- 172.21.15.103:0/818085249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c1100f0 0x7fa87c1124e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa874009cd0 tx=0x7fa87402a8e0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 -- 172.21.15.103:0/818085249 shutdown_connections 2024-05-22T23:57:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 --2- 172.21.15.103:0/818085249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c1100f0 0x7fa87c1124e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 --2- 172.21.15.103:0/818085249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c1091d0 0x7fa87c10fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 --2- 172.21.15.103:0/818085249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87c108870 0x7fa87c108c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 -- 172.21.15.103:0/818085249 >> 172.21.15.103:0/818085249 conn(0x7fa87c06ea80 msgr2=0x7fa87c070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 -- 172.21.15.103:0/818085249 shutdown_connections 2024-05-22T23:57:52.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.780+0000 7fa882a00640 1 -- 172.21.15.103:0/818085249 wait complete. 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.781+0000 7fa882a00640 1 Processor -- start 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.781+0000 7fa882a00640 1 -- start start 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa882a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c108870 0x7fa87c1a25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa882a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87c1091d0 0x7fa87c1a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa882a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c1100f0 0x7fa87c1a9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa882a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c115d60 con 0x7fa87c1091d0 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa882a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c115be0 con 0x7fa87c108870 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa882a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c115ee0 con 0x7fa87c1100f0 2024-05-22T23:57:52.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c108870 0x7fa87c1a25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c108870 0x7fa87c1a25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:39640/0 (socket says 172.21.15.103:39640) 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87be00640 1 -- 172.21.15.103:0/1235177786 learned_addr learned my addr 172.21.15.103:0/1235177786 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87b400640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87c1091d0 0x7fa87c1a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa880a00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c1100f0 0x7fa87c1a9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87be00640 1 -- 172.21.15.103:0/1235177786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c1100f0 msgr2=0x7fa87c1a9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87be00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c1100f0 0x7fa87c1a9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87be00640 1 -- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87c1091d0 msgr2=0x7fa87c1a2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87be00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87c1091d0 0x7fa87c1a2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.782+0000 7fa87be00640 1 -- 172.21.15.103:0/1235177786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87c1aa1c0 con 0x7fa87c108870 2024-05-22T23:57:52.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.783+0000 7fa87be00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c108870 0x7fa87c1a25a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00e9f0 tx=0x7fa86c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:52.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.783+0000 7fa878c00640 1 -- 172.21.15.103:0/1235177786 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c00cd80 con 0x7fa87c108870 2024-05-22T23:57:52.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.783+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87c1aa4b0 con 0x7fa87c108870 2024-05-22T23:57:52.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.783+0000 7fa878c00640 1 -- 172.21.15.103:0/1235177786 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa86c010910 con 0x7fa87c108870 2024-05-22T23:57:52.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.783+0000 7fa878c00640 1 -- 172.21.15.103:0/1235177786 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0188f0 con 0x7fa87c108870 2024-05-22T23:57:52.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.783+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa87c1aaa50 con 0x7fa87c108870 2024-05-22T23:57:52.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.784+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa87c1aacb0 con 0x7fa87c108870 2024-05-22T23:57:52.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.786+0000 7fa878c00640 1 -- 172.21.15.103:0/1235177786 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa86c010ac0 con 0x7fa87c108870 2024-05-22T23:57:52.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.787+0000 7fa878c00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa8600a2c10 0x7fa8600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:52.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.787+0000 7fa878c00640 1 -- 172.21.15.103:0/1235177786 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa86c0cfac0 con 0x7fa87c108870 2024-05-22T23:57:52.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.788+0000 7fa87b400640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa8600a2c10 0x7fa8600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:52.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.788+0000 7fa87b400640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa8600a2c10 0x7fa8600a50d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa87c1a3ab0 tx=0x7fa868005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:52.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.790+0000 7fa878c00640 1 -- 172.21.15.103:0/1235177786 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa86c09c190 con 0x7fa87c108870 2024-05-22T23:57:52.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:52 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3555352260' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-22T23:57:52.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:52 smithi105 ceph-mon[36426]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:52.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:52 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/308698214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:52.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:52 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3555352260' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-22T23:57:52.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:52 smithi103 ceph-mon[31370]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:52.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:52 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/308698214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:52.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:52 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3555352260' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-22T23:57:52.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:52 smithi174 ceph-mon[36179]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-22T23:57:52.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:52 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/308698214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:57:52.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.990+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa87c1aae80 con 0x7fa87c108870 2024-05-22T23:57:52.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.991+0000 7fa878c00640 1 -- 172.21.15.103:0/1235177786 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7fa86c0a0490 con 0x7fa87c108870 2024-05-22T23:57:52.996 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-22T23:57:52.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa8600a2c10 msgr2=0x7fa8600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa8600a2c10 0x7fa8600a50d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa87c1a3ab0 tx=0x7fa868005f90 comp rx=0 tx=0).stop 2024-05-22T23:57:52.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c108870 msgr2=0x7fa87c1a25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:52.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c108870 0x7fa87c1a25a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00e9f0 tx=0x7fa86c00eeb0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 shutdown_connections 2024-05-22T23:57:52.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa8600a2c10 0x7fa8600a50d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c1100f0 0x7fa87c1a9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:52.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa87c1091d0 0x7fa87c1a2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 --2- 172.21.15.103:0/1235177786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa87c108870 0x7fa87c1a25a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 >> 172.21.15.103:0/1235177786 conn(0x7fa87c06ea80 msgr2=0x7fa87c070e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:53.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 shutdown_connections 2024-05-22T23:57:53.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:52.995+0000 7fa882a00640 1 -- 172.21.15.103:0/1235177786 wait complete. 2024-05-22T23:57:53.018 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-22T23:57:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 -- 172.21.15.103:0/1483312950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c077e10 msgr2=0x7fcb1c0786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1483312950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c077e10 0x7fcb1c0786b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcb14009ca0 tx=0x7fcb1402a8c0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 -- 172.21.15.103:0/1483312950 shutdown_connections 2024-05-22T23:57:53.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1483312950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c077e10 0x7fcb1c0786b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1483312950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb1c077470 0x7fcb1c0778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1483312950 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb1c076230 0x7fcb1c076650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 -- 172.21.15.103:0/1483312950 >> 172.21.15.103:0/1483312950 conn(0x7fcb1c0ff020 msgr2=0x7fcb1c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 -- 172.21.15.103:0/1483312950 shutdown_connections 2024-05-22T23:57:53.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.220+0000 7fcb22c00640 1 -- 172.21.15.103:0/1483312950 wait complete. 2024-05-22T23:57:53.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.221+0000 7fcb22c00640 1 Processor -- start 2024-05-22T23:57:53.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.221+0000 7fcb22c00640 1 -- start start 2024-05-22T23:57:53.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.221+0000 7fcb22c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb1c076230 0x7fcb1c19e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:53.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.221+0000 7fcb22c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb1c077470 0x7fcb1c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:53.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb22c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c077e10 0x7fcb1c1a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:53.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb22c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c113b60 con 0x7fcb1c077e10 2024-05-22T23:57:53.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb22c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1139e0 con 0x7fcb1c076230 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb22c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c113ce0 con 0x7fcb1c077470 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb1c076230 0x7fcb1c19e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb20c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c077e10 0x7fcb1c1a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb1c076230 0x7fcb1c19e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:39662/0 (socket says 172.21.15.103:39662) 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1be00640 1 -- 172.21.15.103:0/1552176714 learned_addr learned my addr 172.21.15.103:0/1552176714 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1b400640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb1c077470 0x7fcb1c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1be00640 1 -- 172.21.15.103:0/1552176714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb1c077470 msgr2=0x7fcb1c19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1be00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb1c077470 0x7fcb1c19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1be00640 1 -- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c077e10 msgr2=0x7fcb1c1a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1be00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c077e10 0x7fcb1c1a5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.222+0000 7fcb1be00640 1 -- 172.21.15.103:0/1552176714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c1a5db0 con 0x7fcb1c076230 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.223+0000 7fcb1be00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb1c076230 0x7fcb1c19e190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00e9f0 tx=0x7fcb0c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:53.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.223+0000 7fcb18c00640 1 -- 172.21.15.103:0/1552176714 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c00cd80 con 0x7fcb1c076230 2024-05-22T23:57:53.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.223+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1c1a60a0 con 0x7fcb1c076230 2024-05-22T23:57:53.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.223+0000 7fcb18c00640 1 -- 172.21.15.103:0/1552176714 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb0c010910 con 0x7fcb1c076230 2024-05-22T23:57:53.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.223+0000 7fcb18c00640 1 -- 172.21.15.103:0/1552176714 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c018980 con 0x7fcb1c076230 2024-05-22T23:57:53.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.223+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb1c1a6640 con 0x7fcb1c076230 2024-05-22T23:57:53.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.224+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb1c110570 con 0x7fcb1c076230 2024-05-22T23:57:53.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.226+0000 7fcb18c00640 1 -- 172.21.15.103:0/1552176714 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fcb0c010ac0 con 0x7fcb1c076230 2024-05-22T23:57:53.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.228+0000 7fcb18c00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcb000a2c10 0x7fcb000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:53.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.228+0000 7fcb18c00640 1 -- 172.21.15.103:0/1552176714 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fcb0c0cf950 con 0x7fcb1c076230 2024-05-22T23:57:53.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.228+0000 7fcb1b400640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcb000a2c10 0x7fcb000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:53.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.229+0000 7fcb1b400640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcb000a2c10 0x7fcb000a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c19f6a0 tx=0x7fcb0800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:53.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.230+0000 7fcb18c00640 1 -- 172.21.15.103:0/1552176714 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcb0c09bfc0 con 0x7fcb1c076230 2024-05-22T23:57:53.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.433+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcb1c062470 con 0x7fcb1c076230 2024-05-22T23:57:53.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.434+0000 7fcb18c00640 1 -- 172.21.15.103:0/1552176714 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fcb0c0a02c0 con 0x7fcb1c076230 2024-05-22T23:57:53.438 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-22T23:57:53.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcb000a2c10 msgr2=0x7fcb000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcb000a2c10 0x7fcb000a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c19f6a0 tx=0x7fcb0800b410 comp rx=0 tx=0).stop 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb1c076230 msgr2=0x7fcb1c19e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb1c076230 0x7fcb1c19e190 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c00e9f0 tx=0x7fcb0c00eeb0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 shutdown_connections 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcb000a2c10 0x7fcb000a50d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcb1c077e10 0x7fcb1c1a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb1c077470 0x7fcb1c19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 --2- 172.21.15.103:0/1552176714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb1c076230 0x7fcb1c19e190 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.436+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 >> 172.21.15.103:0/1552176714 conn(0x7fcb1c0ff020 msgr2=0x7fcb1c100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.437+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 shutdown_connections 2024-05-22T23:57:53.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.437+0000 7fcb22c00640 1 -- 172.21.15.103:0/1552176714 wait complete. 2024-05-22T23:57:53.460 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.663+0000 7fa2f7600640 1 -- 172.21.15.103:0/1260831806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0076230 msgr2=0x7fa2f0076650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.663+0000 7fa2f7600640 1 --2- 172.21.15.103:0/1260831806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0076230 0x7fa2f0076650 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009900 tx=0x7fa2e402a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.664+0000 7fa2f7600640 1 -- 172.21.15.103:0/1260831806 shutdown_connections 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.664+0000 7fa2f7600640 1 --2- 172.21.15.103:0/1260831806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f0077e10 0x7fa2f00786b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.664+0000 7fa2f7600640 1 --2- 172.21.15.103:0/1260831806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2f0077470 0x7fa2f00778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.664+0000 7fa2f7600640 1 --2- 172.21.15.103:0/1260831806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0076230 0x7fa2f0076650 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.664+0000 7fa2f7600640 1 -- 172.21.15.103:0/1260831806 >> 172.21.15.103:0/1260831806 conn(0x7fa2f00ff060 msgr2=0x7fa2f0101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.664+0000 7fa2f7600640 1 -- 172.21.15.103:0/1260831806 shutdown_connections 2024-05-22T23:57:53.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.664+0000 7fa2f7600640 1 -- 172.21.15.103:0/1260831806 wait complete. 2024-05-22T23:57:53.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.665+0000 7fa2f7600640 1 Processor -- start 2024-05-22T23:57:53.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.665+0000 7fa2f7600640 1 -- start start 2024-05-22T23:57:53.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.665+0000 7fa2f7600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f0076230 0x7fa2f019e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:53.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.665+0000 7fa2f7600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2f0077470 0x7fa2f019e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:53.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.665+0000 7fa2f7600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0077e10 0x7fa2f01a5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:53.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.665+0000 7fa2f7600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f0113b20 con 0x7fa2f0076230 2024-05-22T23:57:53.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.665+0000 7fa2f7600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f01139a0 con 0x7fa2f0077470 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.665+0000 7fa2f7600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f0113ca0 con 0x7fa2f0077e10 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2f4c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f0076230 0x7fa2f019e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2efe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2f0077470 0x7fa2f019e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2f4c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f0076230 0x7fa2f019e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44616/0 (socket says 172.21.15.103:44616) 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2f4c00640 1 -- 172.21.15.103:0/3210842359 learned_addr learned my addr 172.21.15.103:0/3210842359 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2f5600640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0077e10 0x7fa2f01a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2efe00640 1 -- 172.21.15.103:0/3210842359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0077e10 msgr2=0x7fa2f01a5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2efe00640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0077e10 0x7fa2f01a5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2efe00640 1 -- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f0076230 msgr2=0x7fa2f019e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2efe00640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f0076230 0x7fa2f019e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2efe00640 1 -- 172.21.15.103:0/3210842359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f01a5db0 con 0x7fa2f0077470 2024-05-22T23:57:53.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.666+0000 7fa2efe00640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2f0077470 0x7fa2f019e6d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00e9f0 tx=0x7fa2dc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:53.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.667+0000 7fa2ed600640 1 -- 172.21.15.103:0/3210842359 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00ccc0 con 0x7fa2f0077470 2024-05-22T23:57:53.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.667+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f01a60a0 con 0x7fa2f0077470 2024-05-22T23:57:53.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.667+0000 7fa2ed600640 1 -- 172.21.15.103:0/3210842359 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2dc010910 con 0x7fa2f0077470 2024-05-22T23:57:53.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.667+0000 7fa2ed600640 1 -- 172.21.15.103:0/3210842359 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc018960 con 0x7fa2f0077470 2024-05-22T23:57:53.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.667+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f01a6640 con 0x7fa2f0077470 2024-05-22T23:57:53.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.668+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f0110580 con 0x7fa2f0077470 2024-05-22T23:57:53.675 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.671+0000 7fa2ed600640 1 -- 172.21.15.103:0/3210842359 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fa2dc010ac0 con 0x7fa2f0077470 2024-05-22T23:57:53.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.673+0000 7fa2ed600640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2d80a2b90 0x7fa2d80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:53.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.673+0000 7fa2ed600640 1 -- 172.21.15.103:0/3210842359 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa2dc0cf670 con 0x7fa2f0077470 2024-05-22T23:57:53.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.673+0000 7fa2f4c00640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2d80a2b90 0x7fa2d80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:53.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.674+0000 7fa2f4c00640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2d80a2b90 0x7fa2d80a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4031440 tx=0x7fa2e40315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:53.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.677+0000 7fa2ed600640 1 -- 172.21.15.103:0/3210842359 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa2dc09bce0 con 0x7fa2f0077470 2024-05-22T23:57:53.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:53 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/167269372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:53 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1235177786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:53 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1552176714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:53 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/167269372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:53 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1235177786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:53 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1552176714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:53 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/167269372' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:53 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1235177786' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:53 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1552176714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:57:53.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.885+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa2f0062470 con 0x7fa2f0077470 2024-05-22T23:57:53.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.887+0000 7fa2ed600640 1 -- 172.21.15.103:0/3210842359 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7fa2dc09ffe0 con 0x7fa2f0077470 2024-05-22T23:57:53.891 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.889+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2d80a2b90 msgr2=0x7fa2d80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.889+0000 7fa2f7600640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2d80a2b90 0x7fa2d80a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4031440 tx=0x7fa2e40315b0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.889+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2f0077470 msgr2=0x7fa2f019e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.889+0000 7fa2f7600640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2f0077470 0x7fa2f019e6d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00e9f0 tx=0x7fa2dc00eeb0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.890+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 shutdown_connections 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.890+0000 7fa2f7600640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2d80a2b90 0x7fa2d80a5050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.890+0000 7fa2f7600640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa2f0077e10 0x7fa2f01a5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.890+0000 7fa2f7600640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa2f0077470 0x7fa2f019e6d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.890+0000 7fa2f7600640 1 --2- 172.21.15.103:0/3210842359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2f0076230 0x7fa2f019e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:53.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.890+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 >> 172.21.15.103:0/3210842359 conn(0x7fa2f00ff060 msgr2=0x7fa2f01007a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:53.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.890+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 shutdown_connections 2024-05-22T23:57:53.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:53.890+0000 7fa2f7600640 1 -- 172.21.15.103:0/3210842359 wait complete. 2024-05-22T23:57:53.912 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-22T23:57:54.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.115+0000 7ff04ba00640 1 -- 172.21.15.103:0/4083966204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff044107e80 msgr2=0x7ff0441082a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.115+0000 7ff04ba00640 1 --2- 172.21.15.103:0/4083966204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff044107e80 0x7ff0441082a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff038009900 tx=0x7ff03802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.115+0000 7ff04ba00640 1 -- 172.21.15.103:0/4083966204 shutdown_connections 2024-05-22T23:57:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.115+0000 7ff04ba00640 1 --2- 172.21.15.103:0/4083966204 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff044109a20 0x7ff044110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.115+0000 7ff04ba00640 1 --2- 172.21.15.103:0/4083966204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff044109060 0x7ff0441094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.115+0000 7ff04ba00640 1 --2- 172.21.15.103:0/4083966204 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff044107e80 0x7ff0441082a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.115+0000 7ff04ba00640 1 -- 172.21.15.103:0/4083966204 >> 172.21.15.103:0/4083966204 conn(0x7ff044076e70 msgr2=0x7ff0440792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.116+0000 7ff04ba00640 1 -- 172.21.15.103:0/4083966204 shutdown_connections 2024-05-22T23:57:54.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.116+0000 7ff04ba00640 1 -- 172.21.15.103:0/4083966204 wait complete. 2024-05-22T23:57:54.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.117+0000 7ff04ba00640 1 Processor -- start 2024-05-22T23:57:54.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.117+0000 7ff04ba00640 1 -- start start 2024-05-22T23:57:54.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.117+0000 7ff04ba00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff044107e80 0x7ff04419e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:54.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.117+0000 7ff04ba00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff044109060 0x7ff04419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff04ba00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff044109a20 0x7ff0441a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff04ba00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff044113b50 con 0x7ff044109060 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff04ba00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff0441139d0 con 0x7ff044107e80 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff04ba00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff044113cd0 con 0x7ff044109a20 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff043e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff044109060 0x7ff04419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff043e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff044109060 0x7ff04419e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48610/0 (socket says 172.21.15.103:48610) 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff043e00640 1 -- 172.21.15.103:0/2973460649 learned_addr learned my addr 172.21.15.103:0/2973460649 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff049000640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff044107e80 0x7ff04419e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff049a00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff044109a20 0x7ff0441a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff043e00640 1 -- 172.21.15.103:0/2973460649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff044109a20 msgr2=0x7ff0441a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff043e00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff044109a20 0x7ff0441a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff043e00640 1 -- 172.21.15.103:0/2973460649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff044107e80 msgr2=0x7ff04419e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff043e00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff044107e80 0x7ff04419e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.123 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.118+0000 7ff043e00640 1 -- 172.21.15.103:0/2973460649 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0441a5e50 con 0x7ff044109060 2024-05-22T23:57:54.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.119+0000 7ff043e00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff044109060 0x7ff04419e770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff03000ea60 tx=0x7ff03000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:54.124 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.119+0000 7ff041600640 1 -- 172.21.15.103:0/2973460649 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03000cd20 con 0x7ff044109060 2024-05-22T23:57:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.119+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0441a6140 con 0x7ff044109060 2024-05-22T23:57:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.119+0000 7ff041600640 1 -- 172.21.15.103:0/2973460649 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0300035f0 con 0x7ff044109060 2024-05-22T23:57:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.119+0000 7ff041600640 1 -- 172.21.15.103:0/2973460649 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff030010690 con 0x7ff044109060 2024-05-22T23:57:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.119+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0441a6690 con 0x7ff044109060 2024-05-22T23:57:54.125 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.121+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff04410f2a0 con 0x7ff044109060 2024-05-22T23:57:54.127 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.123+0000 7ff041600640 1 -- 172.21.15.103:0/2973460649 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7ff0300037a0 con 0x7ff044109060 2024-05-22T23:57:54.128 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.124+0000 7ff041600640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff02c0a2b90 0x7ff02c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:57:54.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.125+0000 7ff041600640 1 -- 172.21.15.103:0/2973460649 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff0300cf5c0 con 0x7ff044109060 2024-05-22T23:57:54.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.125+0000 7ff049000640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff02c0a2b90 0x7ff02c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:57:54.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.126+0000 7ff049000640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff02c0a2b90 0x7ff02c0a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff038005ab0 tx=0x7ff038000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:57:54.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.129+0000 7ff041600640 1 -- 172.21.15.103:0/2973460649 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff03009c280 con 0x7ff044109060 2024-05-22T23:57:54.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.344+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff04402d200 con 0x7ff044109060 2024-05-22T23:57:54.349 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.345+0000 7ff041600640 1 -- 172.21.15.103:0/2973460649 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v91) ==== 81+0+601 (secure 0 0 0) 0x7ff03009bb80 con 0x7ff044109060 2024-05-22T23:57:54.349 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3190009536 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:54.349 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1448560264 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/274656845 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2141778982 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2592202413 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/3762323194 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/3495336128 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/3762323194 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/3495336128 2024-05-23T23:48:54.926142+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1425794392 2024-05-23T23:49:29.953234+0000 2024-05-22T23:57:54.350 INFO:teuthology.orchestra.run.smithi103.stderr:listed 10 entries 2024-05-22T23:57:54.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff02c0a2b90 msgr2=0x7ff02c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:54.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff02c0a2b90 0x7ff02c0a5050 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff038005ab0 tx=0x7ff038000bc0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff044109060 msgr2=0x7ff04419e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff044109060 0x7ff04419e770 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff03000ea60 tx=0x7ff03000ee60 comp rx=0 tx=0).stop 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 shutdown_connections 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff02c0a2b90 0x7ff02c0a5050 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff044109a20 0x7ff0441a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff044109060 0x7ff04419e770 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 --2- 172.21.15.103:0/2973460649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff044107e80 0x7ff04419e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.348+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 >> 172.21.15.103:0/2973460649 conn(0x7ff044076e70 msgr2=0x7ff04410c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.349+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 shutdown_connections 2024-05-22T23:57:54.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:57:54.349+0000 7ff04ba00640 1 -- 172.21.15.103:0/2973460649 wait complete. 2024-05-22T23:57:54.371 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi103.front.sepia.ceph.com... 2024-05-22T23:57:54.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:57:54.372 DEBUG:teuthology.orchestra.run.smithi103:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-22T23:57:54.452 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi103.stderr:ceph-fuse[75500]: fuse finished with error 0 and tester_r 0 2024-05-22T23:57:54.458 INFO:teuthology.orchestra.run:waiting for 300 2024-05-22T23:57:54.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:54 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3210842359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:57:54.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:54 smithi105 ceph-mon[36426]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-22T23:57:54.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:54 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2973460649' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-22T23:57:54.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:54 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3210842359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:57:54.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:54 smithi103 ceph-mon[31370]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-22T23:57:54.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:54 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2973460649' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-22T23:57:54.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:54 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3210842359' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:57:54.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:54 smithi174 ceph-mon[36179]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-22T23:57:54.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:54 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2973460649' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-22T23:57:56.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:56 smithi105 ceph-mon[36426]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-22T23:57:56.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:56 smithi103 ceph-mon[31370]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-22T23:57:56.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:56 smithi174 ceph-mon[36179]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-05-22T23:57:58.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:57:58 smithi105 ceph-mon[36426]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:57:58.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:57:58 smithi103 ceph-mon[31370]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:57:58.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:57:58 smithi174 ceph-mon[36179]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:00 smithi105 ceph-mon[36426]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:00.561 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-05-22T23:58:00.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-22T23:58:00.561 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-22T23:58:00.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:00 smithi103 ceph-mon[31370]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:00.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:00 smithi174 ceph-mon[36179]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:00.643 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:58:00.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 -- 172.21.15.103:0/1669945811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9224105680 msgr2=0x7f922410bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:00.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 --2- 172.21.15.103:0/1669945811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9224105680 0x7f922410bf10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f921c009be0 tx=0x7f921c02a6d0 comp rx=0 tx=0).stop 2024-05-22T23:58:00.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 -- 172.21.15.103:0/1669945811 shutdown_connections 2024-05-22T23:58:00.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 --2- 172.21.15.103:0/1669945811 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9224105680 0x7f922410bf10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:00.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 --2- 172.21.15.103:0/1669945811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9224104cc0 0x7f9224105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:00.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 --2- 172.21.15.103:0/1669945811 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9224103ad0 0x7f9224103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:00.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 -- 172.21.15.103:0/1669945811 >> 172.21.15.103:0/1669945811 conn(0x7f92240ff060 msgr2=0x7f9224101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:00.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 -- 172.21.15.103:0/1669945811 shutdown_connections 2024-05-22T23:58:00.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.891+0000 7f9228c00640 1 -- 172.21.15.103:0/1669945811 wait complete. 2024-05-22T23:58:00.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.892+0000 7f9228c00640 1 Processor -- start 2024-05-22T23:58:00.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.892+0000 7f9228c00640 1 -- start start 2024-05-22T23:58:00.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9228c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9224103ad0 0x7f9224199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:00.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9228c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9224104cc0 0x7f922419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:00.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9228c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9224105680 0x7f92241a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:00.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9228c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f922410f700 con 0x7f9224103ad0 2024-05-22T23:58:00.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9228c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f922410f580 con 0x7f9224104cc0 2024-05-22T23:58:00.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9228c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f922410f880 con 0x7f9224105680 2024-05-22T23:58:00.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9221e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9224103ad0 0x7f9224199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:00.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9222800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9224105680 0x7f92241a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:00.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9221400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9224104cc0 0x7f922419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:00.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9221e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9224103ad0 0x7f9224199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48636/0 (socket says 172.21.15.103:48636) 2024-05-22T23:58:00.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.893+0000 7f9221e00640 1 -- 172.21.15.103:0/75739888 learned_addr learned my addr 172.21.15.103:0/75739888 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:00.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.894+0000 7f9221e00640 1 -- 172.21.15.103:0/75739888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9224105680 msgr2=0x7f92241a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:00.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.894+0000 7f9221e00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9224105680 0x7f92241a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:00.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.894+0000 7f9221e00640 1 -- 172.21.15.103:0/75739888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9224104cc0 msgr2=0x7f922419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:00.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.894+0000 7f9221e00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9224104cc0 0x7f922419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:00.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.894+0000 7f9221e00640 1 -- 172.21.15.103:0/75739888 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92241a19b0 con 0x7f9224103ad0 2024-05-22T23:58:00.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.894+0000 7f9221e00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9224103ad0 0x7f9224199da0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f921400e9a0 tx=0x7f921400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:00.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.894+0000 7f921aa00640 1 -- 172.21.15.103:0/75739888 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f921400cd80 con 0x7f9224103ad0 2024-05-22T23:58:00.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.894+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92241a1ca0 con 0x7f9224103ad0 2024-05-22T23:58:00.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.895+0000 7f921aa00640 1 -- 172.21.15.103:0/75739888 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9214010910 con 0x7f9224103ad0 2024-05-22T23:58:00.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.895+0000 7f921aa00640 1 -- 172.21.15.103:0/75739888 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9214018a40 con 0x7f9224103ad0 2024-05-22T23:58:00.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.895+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92241a2190 con 0x7f9224103ad0 2024-05-22T23:58:00.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.896+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f922410ae10 con 0x7f9224103ad0 2024-05-22T23:58:00.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.898+0000 7f921aa00640 1 -- 172.21.15.103:0/75739888 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f9214010ac0 con 0x7f9224103ad0 2024-05-22T23:58:00.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.899+0000 7f921aa00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92080a2b90 0x7f92080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:00.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.899+0000 7f921aa00640 1 -- 172.21.15.103:0/75739888 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f92140cf6c0 con 0x7f9224103ad0 2024-05-22T23:58:00.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.899+0000 7f9221400640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92080a2b90 0x7f92080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:00.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.900+0000 7f9221400640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92080a2b90 0x7f92080a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9210009bd0 tx=0x7f9210005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:00.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:00.903+0000 7f921aa00640 1 -- 172.21.15.103:0/75739888 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9214053850 con 0x7f9224103ad0 2024-05-22T23:58:01.112 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.108+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9224062510 con 0x7f9224103ad0 2024-05-22T23:58:01.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.112+0000 7f921aa00640 1 -- 172.21.15.103:0/75739888 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7f921409c380 con 0x7f9224103ad0 2024-05-22T23:58:01.116 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:01.117 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:58:01.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92080a2b90 msgr2=0x7f92080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92080a2b90 0x7f92080a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9210009bd0 tx=0x7f9210005f90 comp rx=0 tx=0).stop 2024-05-22T23:58:01.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9224103ad0 msgr2=0x7f9224199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9224103ad0 0x7f9224199da0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f921400e9a0 tx=0x7f921400ee60 comp rx=0 tx=0).stop 2024-05-22T23:58:01.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 shutdown_connections 2024-05-22T23:58:01.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f92080a2b90 0x7f92080a5050 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9224105680 0x7f92241a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9224104cc0 0x7f922419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 --2- 172.21.15.103:0/75739888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9224103ad0 0x7f9224199da0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 >> 172.21.15.103:0/75739888 conn(0x7f92240ff060 msgr2=0x7f9224107fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:01.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 shutdown_connections 2024-05-22T23:58:01.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.115+0000 7f9228c00640 1 -- 172.21.15.103:0/75739888 wait complete. 2024-05-22T23:58:01.140 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.343+0000 7f508e600640 1 -- 172.21.15.103:0/1439103895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50881056a0 msgr2=0x7f508810bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.343+0000 7f508e600640 1 --2- 172.21.15.103:0/1439103895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50881056a0 0x7f508810bf30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f5080009be0 tx=0x7f508002a6c0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.343+0000 7f508e600640 1 -- 172.21.15.103:0/1439103895 shutdown_connections 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.343+0000 7f508e600640 1 --2- 172.21.15.103:0/1439103895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50881056a0 0x7f508810bf30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.343+0000 7f508e600640 1 --2- 172.21.15.103:0/1439103895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5088104ce0 0x7f5088105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.343+0000 7f508e600640 1 --2- 172.21.15.103:0/1439103895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5088103af0 0x7f5088103f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.343+0000 7f508e600640 1 -- 172.21.15.103:0/1439103895 >> 172.21.15.103:0/1439103895 conn(0x7f50880ff060 msgr2=0x7f50881014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.344+0000 7f508e600640 1 -- 172.21.15.103:0/1439103895 shutdown_connections 2024-05-22T23:58:01.348 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.344+0000 7f508e600640 1 -- 172.21.15.103:0/1439103895 wait complete. 2024-05-22T23:58:01.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f508e600640 1 Processor -- start 2024-05-22T23:58:01.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f508e600640 1 -- start start 2024-05-22T23:58:01.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f508e600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5088103af0 0x7f5088199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f508e600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5088104ce0 0x7f508819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f508e600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50881056a0 0x7f50881a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f508e600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f508810f770 con 0x7f50881056a0 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f508e600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f508810f5f0 con 0x7f5088104ce0 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f508e600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f508810f8f0 con 0x7f5088103af0 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f5087e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50881056a0 0x7f50881a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f507ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5088104ce0 0x7f508819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f5087400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5088103af0 0x7f5088199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.345+0000 7f5087e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50881056a0 0x7f50881a1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48662/0 (socket says 172.21.15.103:48662) 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f5087e00640 1 -- 172.21.15.103:0/4266591093 learned_addr learned my addr 172.21.15.103:0/4266591093 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f5087400640 1 -- 172.21.15.103:0/4266591093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5088104ce0 msgr2=0x7f508819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f5087400640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5088104ce0 0x7f508819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f5087400640 1 -- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50881056a0 msgr2=0x7f50881a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f5087400640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50881056a0 0x7f50881a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f5087400640 1 -- 172.21.15.103:0/4266591093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50881a19d0 con 0x7f5088103af0 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f507ea00640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5088104ce0 0x7f508819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f5087400640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5088103af0 0x7f5088199db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5088104b40 tx=0x7f507800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.346+0000 7f5084c00640 1 -- 172.21.15.103:0/4266591093 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078004440 con 0x7f5088103af0 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.347+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50881a1cc0 con 0x7f5088103af0 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.347+0000 7f5084c00640 1 -- 172.21.15.103:0/4266591093 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f507800cda0 con 0x7f5088103af0 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.347+0000 7f5084c00640 1 -- 172.21.15.103:0/4266591093 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50780196e0 con 0x7f5088103af0 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.347+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50881a21c0 con 0x7f5088103af0 2024-05-22T23:58:01.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.348+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508810ae10 con 0x7f5088103af0 2024-05-22T23:58:01.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.350+0000 7f5084c00640 1 -- 172.21.15.103:0/4266591093 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f5078003780 con 0x7f5088103af0 2024-05-22T23:58:01.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.351+0000 7f5084c00640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f506c0a2b90 0x7f506c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:01.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.351+0000 7f5084c00640 1 -- 172.21.15.103:0/4266591093 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f50780cf670 con 0x7f5088103af0 2024-05-22T23:58:01.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.351+0000 7f507ea00640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f506c0a2b90 0x7f506c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:01.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.352+0000 7f507ea00640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f506c0a2b90 0x7f506c0a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f508819b2c0 tx=0x7f5070005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:01.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.355+0000 7f5084c00640 1 -- 172.21.15.103:0/4266591093 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5078003e20 con 0x7f5088103af0 2024-05-22T23:58:01.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:01 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/75739888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:01.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.556+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5088062510 con 0x7f5088103af0 2024-05-22T23:58:01.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.559+0000 7f5084c00640 1 -- 172.21.15.103:0/4266591093 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7f5078009510 con 0x7f5088103af0 2024-05-22T23:58:01.563 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:01.563 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:58:01.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.561+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f506c0a2b90 msgr2=0x7f506c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.561+0000 7f508e600640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f506c0a2b90 0x7f506c0a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f508819b2c0 tx=0x7f5070005f90 comp rx=0 tx=0).stop 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.561+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5088103af0 msgr2=0x7f5088199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.561+0000 7f508e600640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5088103af0 0x7f5088199db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5088104b40 tx=0x7f507800ee50 comp rx=0 tx=0).stop 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.562+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 shutdown_connections 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.562+0000 7f508e600640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f506c0a2b90 0x7f506c0a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.562+0000 7f508e600640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50881056a0 0x7f50881a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.562+0000 7f508e600640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5088104ce0 0x7f508819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.562+0000 7f508e600640 1 --2- 172.21.15.103:0/4266591093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5088103af0 0x7f5088199db0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.562+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 >> 172.21.15.103:0/4266591093 conn(0x7f50880ff060 msgr2=0x7f5088107fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.562+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 shutdown_connections 2024-05-22T23:58:01.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.562+0000 7f508e600640 1 -- 172.21.15.103:0/4266591093 wait complete. 2024-05-22T23:58:01.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:01 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/75739888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:01.586 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-22T23:58:01.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:01 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/75739888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:01.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 -- 172.21.15.103:0/842060936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d80758b0 msgr2=0x7fb6d8073d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/842060936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d80758b0 0x7fb6d8073d00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8009900 tx=0x7fb6c802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 -- 172.21.15.103:0/842060936 shutdown_connections 2024-05-22T23:58:01.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/842060936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6d8109640 0x7fb6d810bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/842060936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d80742d0 0x7fb6d8074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/842060936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d80758b0 0x7fb6d8073d00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 -- 172.21.15.103:0/842060936 >> 172.21.15.103:0/842060936 conn(0x7fb6d80fcd10 msgr2=0x7fb6d80ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:01.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 -- 172.21.15.103:0/842060936 shutdown_connections 2024-05-22T23:58:01.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.778+0000 7fb6d7e00640 1 -- 172.21.15.103:0/842060936 wait complete. 2024-05-22T23:58:01.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.779+0000 7fb6d7e00640 1 Processor -- start 2024-05-22T23:58:01.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.779+0000 7fb6d7e00640 1 -- start start 2024-05-22T23:58:01.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.779+0000 7fb6d7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d80742d0 0x7fb6d8199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:01.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.779+0000 7fb6d7e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6d80758b0 0x7fb6d819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.779+0000 7fb6d7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d8109640 0x7fb6d81a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.779+0000 7fb6d7e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810f610 con 0x7fb6d8109640 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d7e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810f490 con 0x7fb6d80742d0 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d7e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d810f790 con 0x7fb6d80758b0 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d80742d0 0x7fb6d8199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d80742d0 0x7fb6d8199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:50828/0 (socket says 172.21.15.103:50828) 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6d80758b0 0x7fb6d819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6a00640 1 -- 172.21.15.103:0/688267056 learned_addr learned my addr 172.21.15.103:0/688267056 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d7400640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d8109640 0x7fb6d81a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6a00640 1 -- 172.21.15.103:0/688267056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6d80758b0 msgr2=0x7fb6d819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6a00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6d80758b0 0x7fb6d819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6a00640 1 -- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d8109640 msgr2=0x7fb6d81a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:01.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6a00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d8109640 0x7fb6d81a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:01.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d6a00640 1 -- 172.21.15.103:0/688267056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d81a18c0 con 0x7fb6d80742d0 2024-05-22T23:58:01.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.780+0000 7fb6d7400640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d8109640 0x7fb6d81a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:58:01.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.781+0000 7fb6d6a00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d80742d0 0x7fb6d8199c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8031440 tx=0x7fb6c8032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:01.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.781+0000 7fb6cf400640 1 -- 172.21.15.103:0/688267056 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c803f030 con 0x7fb6d80742d0 2024-05-22T23:58:01.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.781+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d81a1b50 con 0x7fb6d80742d0 2024-05-22T23:58:01.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.781+0000 7fb6cf400640 1 -- 172.21.15.103:0/688267056 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6c8003ce0 con 0x7fb6d80742d0 2024-05-22T23:58:01.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.781+0000 7fb6cf400640 1 -- 172.21.15.103:0/688267056 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8035070 con 0x7fb6d80742d0 2024-05-22T23:58:01.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.781+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d81a2090 con 0x7fb6d80742d0 2024-05-22T23:58:01.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.782+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6d8108370 con 0x7fb6d80742d0 2024-05-22T23:58:01.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.785+0000 7fb6cf400640 1 -- 172.21.15.103:0/688267056 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fb6c8042030 con 0x7fb6d80742d0 2024-05-22T23:58:01.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.786+0000 7fb6cf400640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb6bc0a2b90 0x7fb6bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:01.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.786+0000 7fb6cf400640 1 -- 172.21.15.103:0/688267056 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fb6c80ef840 con 0x7fb6d80742d0 2024-05-22T23:58:01.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.787+0000 7fb6d6000640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb6bc0a2b90 0x7fb6bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:01.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.788+0000 7fb6d6000640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb6bc0a2b90 0x7fb6bc0a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb6c40095f0 tx=0x7fb6c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:01.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:01.791+0000 7fb6cf400640 1 -- 172.21.15.103:0/688267056 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb6c80bc500 con 0x7fb6d80742d0 2024-05-22T23:58:02.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.027+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fb6d8062470 con 0x7fb6d80742d0 2024-05-22T23:58:02.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.207+0000 7fb6cf400640 1 -- 172.21.15.103:0/688267056 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fb6c80bbe00 con 0x7fb6d80742d0 2024-05-22T23:58:02.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb6bc0a2b90 msgr2=0x7fb6bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:02.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb6bc0a2b90 0x7fb6bc0a5050 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb6c40095f0 tx=0x7fb6c4005f90 comp rx=0 tx=0).stop 2024-05-22T23:58:02.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d80742d0 msgr2=0x7fb6d8199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:02.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d80742d0 0x7fb6d8199c80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8031440 tx=0x7fb6c8032ce0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 shutdown_connections 2024-05-22T23:58:02.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb6bc0a2b90 0x7fb6bc0a5050 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6d8109640 0x7fb6d81a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb6d80758b0 0x7fb6d819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 --2- 172.21.15.103:0/688267056 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d80742d0 0x7fb6d8199c80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 >> 172.21.15.103:0/688267056 conn(0x7fb6d80fcd10 msgr2=0x7fb6d8105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:02.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 shutdown_connections 2024-05-22T23:58:02.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.210+0000 7fb6d7e00640 1 -- 172.21.15.103:0/688267056 wait complete. 2024-05-22T23:58:02.237 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-05-22T23:58:02.238 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-05-22T23:58:02.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.438+0000 7fd1b3e00640 1 -- 172.21.15.103:0/410839113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b410dd50 msgr2=0x7fd1b4110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:02.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.438+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/410839113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b410dd50 0x7fd1b4110140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac009cd0 tx=0x7fd1ac02a8f0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.439+0000 7fd1b3e00640 1 -- 172.21.15.103:0/410839113 shutdown_connections 2024-05-22T23:58:02.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.439+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/410839113 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b410dd50 0x7fd1b4110140 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.439+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/410839113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b40742d0 0x7fd1b4074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.439+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/410839113 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1b40758b0 0x7fd1b4073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.439+0000 7fd1b3e00640 1 -- 172.21.15.103:0/410839113 >> 172.21.15.103:0/410839113 conn(0x7fd1b40fcd10 msgr2=0x7fd1b40ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:02.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.439+0000 7fd1b3e00640 1 -- 172.21.15.103:0/410839113 shutdown_connections 2024-05-22T23:58:02.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.439+0000 7fd1b3e00640 1 -- 172.21.15.103:0/410839113 wait complete. 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3e00640 1 Processor -- start 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3e00640 1 -- start start 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b40742d0 0x7fd1b419e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1b40758b0 0x7fd1b419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b410dd50 0x7fd1b41a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4113a40 con 0x7fd1b410dd50 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b41138c0 con 0x7fd1b40758b0 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b4113bc0 con 0x7fd1b40742d0 2024-05-22T23:58:02.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b2a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b40742d0 0x7fd1b419e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b410dd50 0x7fd1b41a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b2a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b40742d0 0x7fd1b419e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:54924/0 (socket says 172.21.15.103:54924) 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b2000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1b40758b0 0x7fd1b419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.440+0000 7fd1b2a00640 1 -- 172.21.15.103:0/1273632489 learned_addr learned my addr 172.21.15.103:0/1273632489 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1b2a00640 1 -- 172.21.15.103:0/1273632489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1b40758b0 msgr2=0x7fd1b419e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1b2a00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1b40758b0 0x7fd1b419e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1b2a00640 1 -- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b410dd50 msgr2=0x7fd1b41a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1b2a00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b410dd50 0x7fd1b41a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1b2a00640 1 -- 172.21.15.103:0/1273632489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b41a5c70 con 0x7fd1b40742d0 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1b2a00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b40742d0 0x7fd1b419e050 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400e9f0 tx=0x7fd1a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1ab400640 1 -- 172.21.15.103:0/1273632489 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a400cd80 con 0x7fd1b40742d0 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1ab400640 1 -- 172.21.15.103:0/1273632489 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1a4010910 con 0x7fd1b40742d0 2024-05-22T23:58:02.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b41a5f60 con 0x7fd1b40742d0 2024-05-22T23:58:02.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1ab400640 1 -- 172.21.15.103:0/1273632489 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a40188f0 con 0x7fd1b40742d0 2024-05-22T23:58:02.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.441+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b41a63d0 con 0x7fd1b40742d0 2024-05-22T23:58:02.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.442+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b4103f20 con 0x7fd1b40742d0 2024-05-22T23:58:02.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.444+0000 7fd1ab400640 1 -- 172.21.15.103:0/1273632489 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fd1a4010ac0 con 0x7fd1b40742d0 2024-05-22T23:58:02.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.446+0000 7fd1ab400640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd1980a2b90 0x7fd1980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:02.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.446+0000 7fd1ab400640 1 -- 172.21.15.103:0/1273632489 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fd1a40cfbc0 con 0x7fd1b40742d0 2024-05-22T23:58:02.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.446+0000 7fd1b2000640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd1980a2b90 0x7fd1980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:02.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.447+0000 7fd1b2000640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd1980a2b90 0x7fd1980a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd1b419f560 tx=0x7fd19c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:02.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.448+0000 7fd1ab400640 1 -- 172.21.15.103:0/1273632489 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd1a409c230 con 0x7fd1b40742d0 2024-05-22T23:58:02.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:02 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/4266591093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:02.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:02 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/688267056' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-22T23:58:02.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:02 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-22T23:58:02.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:02 smithi105 ceph-mon[36426]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:02.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:02 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/4266591093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:02.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:02 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/688267056' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-22T23:58:02.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:02 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-22T23:58:02.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:02 smithi103 ceph-mon[31370]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:02 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/4266591093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:02 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/688267056' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-22T23:58:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:02 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-22T23:58:02.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:02 smithi174 ceph-mon[36179]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:02.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:02.691+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7fd1b402d470 con 0x7fd1b40742d0 2024-05-22T23:58:03.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.219+0000 7fd1ab400640 1 -- 172.21.15.103:0/1273632489 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7fd1a40a0530 con 0x7fd1b40742d0 2024-05-22T23:58:03.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd1980a2b90 msgr2=0x7fd1980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:03.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd1980a2b90 0x7fd1980a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd1b419f560 tx=0x7fd19c005f90 comp rx=0 tx=0).stop 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b40742d0 msgr2=0x7fd1b419e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b40742d0 0x7fd1b419e050 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400e9f0 tx=0x7fd1a400eeb0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 shutdown_connections 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd1980a2b90 0x7fd1980a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b410dd50 0x7fd1b41a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd1b40758b0 0x7fd1b419e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 --2- 172.21.15.103:0/1273632489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd1b40742d0 0x7fd1b419e050 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 >> 172.21.15.103:0/1273632489 conn(0x7fd1b40fcd10 msgr2=0x7fd1b41010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:03.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 shutdown_connections 2024-05-22T23:58:03.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.224+0000 7fd1b3e00640 1 -- 172.21.15.103:0/1273632489 wait complete. 2024-05-22T23:58:03.255 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.437+0000 7fc5c4400640 1 -- 172.21.15.103:0/1587853048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5bc10de90 msgr2=0x7fc5bc110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.437+0000 7fc5c4400640 1 --2- 172.21.15.103:0/1587853048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5bc10de90 0x7fc5bc110280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc5b4009ca0 tx=0x7fc5b402a8c0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.438+0000 7fc5c4400640 1 -- 172.21.15.103:0/1587853048 shutdown_connections 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.438+0000 7fc5c4400640 1 --2- 172.21.15.103:0/1587853048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5bc10de90 0x7fc5bc110280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.438+0000 7fc5c4400640 1 --2- 172.21.15.103:0/1587853048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5bc0742d0 0x7fc5bc074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.438+0000 7fc5c4400640 1 --2- 172.21.15.103:0/1587853048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc0758b0 0x7fc5bc073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.438+0000 7fc5c4400640 1 -- 172.21.15.103:0/1587853048 >> 172.21.15.103:0/1587853048 conn(0x7fc5bc0fce50 msgr2=0x7fc5bc0ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.438+0000 7fc5c4400640 1 -- 172.21.15.103:0/1587853048 shutdown_connections 2024-05-22T23:58:03.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.438+0000 7fc5c4400640 1 -- 172.21.15.103:0/1587853048 wait complete. 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c4400640 1 Processor -- start 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c4400640 1 -- start start 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c4400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5bc0742d0 0x7fc5bc19e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c4400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc0758b0 0x7fc5bc19e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c4400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5bc10de90 0x7fc5bc1a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c4400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc113b30 con 0x7fc5bc0758b0 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c4400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc1139b0 con 0x7fc5bc0742d0 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c4400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc113cb0 con 0x7fc5bc10de90 2024-05-22T23:58:03.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c1a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5bc0742d0 0x7fc5bc19e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.439+0000 7fc5c1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc0758b0 0x7fc5bc19e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5bc0742d0 0x7fc5bc19e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:50854/0 (socket says 172.21.15.103:50854) 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1a00640 1 -- 172.21.15.103:0/3576826419 learned_addr learned my addr 172.21.15.103:0/3576826419 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c2400640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5bc10de90 0x7fc5bc1a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1a00640 1 -- 172.21.15.103:0/3576826419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5bc10de90 msgr2=0x7fc5bc1a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1a00640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5bc10de90 0x7fc5bc1a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1a00640 1 -- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc0758b0 msgr2=0x7fc5bc19e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1a00640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc0758b0 0x7fc5bc19e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1a00640 1 -- 172.21.15.103:0/3576826419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc1a5da0 con 0x7fc5bc0742d0 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1000640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc0758b0 0x7fc5bc19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:03.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.440+0000 7fc5c1a00640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5bc0742d0 0x7fc5bc19e180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000e9f0 tx=0x7fc5b000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:03.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.441+0000 7fc5ba000640 1 -- 172.21.15.103:0/3576826419 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b000cd80 con 0x7fc5bc0742d0 2024-05-22T23:58:03.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.441+0000 7fc5ba000640 1 -- 172.21.15.103:0/3576826419 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc5b0010910 con 0x7fc5bc0742d0 2024-05-22T23:58:03.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.441+0000 7fc5ba000640 1 -- 172.21.15.103:0/3576826419 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0018990 con 0x7fc5bc0742d0 2024-05-22T23:58:03.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.441+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5bc1a6090 con 0x7fc5bc0742d0 2024-05-22T23:58:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.441+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5bc1a6590 con 0x7fc5bc0742d0 2024-05-22T23:58:03.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.442+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5bc104060 con 0x7fc5bc0742d0 2024-05-22T23:58:03.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.444+0000 7fc5ba000640 1 -- 172.21.15.103:0/3576826419 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fc5b0010ac0 con 0x7fc5bc0742d0 2024-05-22T23:58:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.445+0000 7fc5ba000640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc5a40a2c10 0x7fc5a40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.446+0000 7fc5ba000640 1 -- 172.21.15.103:0/3576826419 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fc5b00cfb00 con 0x7fc5bc0742d0 2024-05-22T23:58:03.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.446+0000 7fc5c1000640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc5a40a2c10 0x7fc5a40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:03.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.446+0000 7fc5c1000640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc5a40a2c10 0x7fc5a40a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc19f690 tx=0x7fc5a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:03.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.448+0000 7fc5ba000640 1 -- 172.21.15.103:0/3576826419 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc5b009c170 con 0x7fc5bc0742d0 2024-05-22T23:58:03.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:03 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-22T23:58:03.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:03 smithi105 ceph-mon[36426]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:58:03.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:03 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1273632489' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-22T23:58:03.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:03 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-22T23:58:03.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:03 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-22T23:58:03.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:03 smithi103 ceph-mon[31370]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:58:03.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:03 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1273632489' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-22T23:58:03.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:03 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-22T23:58:03.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:03 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-22T23:58:03.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:03 smithi174 ceph-mon[36179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:58:03.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:03 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1273632489' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-22T23:58:03.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:03 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-22T23:58:03.682 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:03.678+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fc5bc02d200 con 0x7fc5bc0742d0 2024-05-22T23:58:04.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.233+0000 7fc5ba000640 1 -- 172.21.15.103:0/3576826419 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7fc5b00a0470 con 0x7fc5bc0742d0 2024-05-22T23:58:04.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc5a40a2c10 msgr2=0x7fc5a40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:04.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc5a40a2c10 0x7fc5a40a50d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc19f690 tx=0x7fc5a8005f90 comp rx=0 tx=0).stop 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5bc0742d0 msgr2=0x7fc5bc19e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5bc0742d0 0x7fc5bc19e180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc5b000e9f0 tx=0x7fc5b000eeb0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 shutdown_connections 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc5a40a2c10 0x7fc5a40a50d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5bc10de90 0x7fc5bc1a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc5bc0758b0 0x7fc5bc19e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 --2- 172.21.15.103:0/3576826419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc5bc0742d0 0x7fc5bc19e180 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.237+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 >> 172.21.15.103:0/3576826419 conn(0x7fc5bc0fce50 msgr2=0x7fc5bc101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.238+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 shutdown_connections 2024-05-22T23:58:04.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.238+0000 7fc5c4400640 1 -- 172.21.15.103:0/3576826419 wait complete. 2024-05-22T23:58:04.266 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-22T23:58:04.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.444+0000 7fdad5000640 1 -- 172.21.15.103:0/3784760872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0077e10 msgr2=0x7fdad00786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.444+0000 7fdad5000640 1 --2- 172.21.15.103:0/3784760872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0077e10 0x7fdad00786b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009ca0 tx=0x7fdac802a8c0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.444+0000 7fdad5000640 1 -- 172.21.15.103:0/3784760872 shutdown_connections 2024-05-22T23:58:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.444+0000 7fdad5000640 1 --2- 172.21.15.103:0/3784760872 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0077e10 0x7fdad00786b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.444+0000 7fdad5000640 1 --2- 172.21.15.103:0/3784760872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad0077470 0x7fdad00778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.444+0000 7fdad5000640 1 --2- 172.21.15.103:0/3784760872 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad0076230 0x7fdad0076650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.444+0000 7fdad5000640 1 -- 172.21.15.103:0/3784760872 >> 172.21.15.103:0/3784760872 conn(0x7fdad00fef20 msgr2=0x7fdad0101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.445+0000 7fdad5000640 1 -- 172.21.15.103:0/3784760872 shutdown_connections 2024-05-22T23:58:04.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.445+0000 7fdad5000640 1 -- 172.21.15.103:0/3784760872 wait complete. 2024-05-22T23:58:04.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.446+0000 7fdad5000640 1 Processor -- start 2024-05-22T23:58:04.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.446+0000 7fdad5000640 1 -- start start 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.446+0000 7fdad5000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0076230 0x7fdad019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.446+0000 7fdad5000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad0077470 0x7fdad019e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.446+0000 7fdad5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad0077e10 0x7fdad01a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.446+0000 7fdad5000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdad0113a40 con 0x7fdad0077e10 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.446+0000 7fdad5000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdad01138c0 con 0x7fdad0076230 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.446+0000 7fdad5000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdad0113bc0 con 0x7fdad0077470 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacf400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0076230 0x7fdad019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacf400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0076230 0x7fdad019e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:33676/0 (socket says 172.21.15.103:33676) 2024-05-22T23:58:04.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacf400640 1 -- 172.21.15.103:0/387992300 learned_addr learned my addr 172.21.15.103:0/387992300 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacea00640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad0077470 0x7fdad019e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacfe00640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad0077e10 0x7fdad01a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacf400640 1 -- 172.21.15.103:0/387992300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad0077470 msgr2=0x7fdad019e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacf400640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad0077470 0x7fdad019e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacf400640 1 -- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad0077e10 msgr2=0x7fdad01a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacf400640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad0077e10 0x7fdad01a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacf400640 1 -- 172.21.15.103:0/387992300 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad01a5c50 con 0x7fdad0076230 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.447+0000 7fdacfe00640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad0077e10 0x7fdad01a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.448+0000 7fdacf400640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0076230 0x7fdad019e030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00d770 tx=0x7fdabc00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.448+0000 7fdac7e00640 1 -- 172.21.15.103:0/387992300 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc017070 con 0x7fdad0076230 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.448+0000 7fdac7e00640 1 -- 172.21.15.103:0/387992300 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdabc005500 con 0x7fdad0076230 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.448+0000 7fdac7e00640 1 -- 172.21.15.103:0/387992300 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdabc004e00 con 0x7fdad0076230 2024-05-22T23:58:04.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.448+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad01a5f40 con 0x7fdad0076230 2024-05-22T23:58:04.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.448+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad01a62f0 con 0x7fdad0076230 2024-05-22T23:58:04.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.449+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad0110320 con 0x7fdad0076230 2024-05-22T23:58:04.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.452+0000 7fdac7e00640 1 -- 172.21.15.103:0/387992300 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7fdabc004fa0 con 0x7fdad0076230 2024-05-22T23:58:04.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.453+0000 7fdac7e00640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdab40a2c10 0x7fdab40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:04.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.453+0000 7fdac7e00640 1 -- 172.21.15.103:0/387992300 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fdabc0cec20 con 0x7fdad0076230 2024-05-22T23:58:04.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.453+0000 7fdacea00640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdab40a2c10 0x7fdab40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:04.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.454+0000 7fdacea00640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdab40a2c10 0x7fdab40a50d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdad019f540 tx=0x7fdac0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:04.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.455+0000 7fdac7e00640 1 -- 172.21.15.103:0/387992300 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdabc09b9c0 con 0x7fdad0076230 2024-05-22T23:58:04.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-22T23:58:04.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-22T23:58:04.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-22T23:58:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:58:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-05-22T23:58:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-05-22T23:58:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3576826419' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-22T23:58:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-22T23:58:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:04 smithi105 ceph-mon[36426]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:04.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-22T23:58:04.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-22T23:58:04.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-22T23:58:04.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:58:04.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-05-22T23:58:04.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-05-22T23:58:04.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3576826419' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-22T23:58:04.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-22T23:58:04.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:04 smithi103 ceph-mon[31370]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:04.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-22T23:58:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-22T23:58:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-22T23:58:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-22T23:58:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-05-22T23:58:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-05-22T23:58:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3576826419' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-22T23:58:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-22T23:58:04.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:04 smithi174 ceph-mon[36179]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-05-22T23:58:04.684 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:04.680+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fdad0068a00 con 0x7fdad0076230 2024-05-22T23:58:05.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.256+0000 7fdac7e00640 1 -- 172.21.15.103:0/387992300 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7fdabc09b2c0 con 0x7fdad0076230 2024-05-22T23:58:05.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdab40a2c10 msgr2=0x7fdab40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:05.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdab40a2c10 0x7fdab40a50d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdad019f540 tx=0x7fdac0005f90 comp rx=0 tx=0).stop 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0076230 msgr2=0x7fdad019e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0076230 0x7fdad019e030 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdabc00d770 tx=0x7fdabc00dc30 comp rx=0 tx=0).stop 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 shutdown_connections 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdab40a2c10 0x7fdab40a50d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad0077e10 0x7fdad01a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad0077470 0x7fdad019e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 --2- 172.21.15.103:0/387992300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad0076230 0x7fdad019e030 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.259+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 >> 172.21.15.103:0/387992300 conn(0x7fdad00fef20 msgr2=0x7fdad0100a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.260+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 shutdown_connections 2024-05-22T23:58:05.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.260+0000 7fdad5000640 1 -- 172.21.15.103:0/387992300 wait complete. 2024-05-22T23:58:05.286 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-05-22T23:58:05.286 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-05-22T23:58:05.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.488+0000 7f00b9600640 1 -- 172.21.15.103:0/2840978047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b41075a0 msgr2=0x7f00b41099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:05.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.488+0000 7f00b9600640 1 --2- 172.21.15.103:0/2840978047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b41075a0 0x7f00b41099f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f00a4009ca0 tx=0x7f00a402a8b0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.488+0000 7f00b9600640 1 -- 172.21.15.103:0/2840978047 shutdown_connections 2024-05-22T23:58:05.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.488+0000 7f00b9600640 1 --2- 172.21.15.103:0/2840978047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b41075a0 0x7f00b41099f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.488+0000 7f00b9600640 1 --2- 172.21.15.103:0/2840978047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00b4104c70 0x7f00b4107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.488+0000 7f00b9600640 1 --2- 172.21.15.103:0/2840978047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b4102340 0x7f00b4104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.488+0000 7f00b9600640 1 -- 172.21.15.103:0/2840978047 >> 172.21.15.103:0/2840978047 conn(0x7f00b40fbd30 msgr2=0x7f00b40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:05.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.489+0000 7f00b9600640 1 -- 172.21.15.103:0/2840978047 shutdown_connections 2024-05-22T23:58:05.493 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.489+0000 7f00b9600640 1 -- 172.21.15.103:0/2840978047 wait complete. 2024-05-22T23:58:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.489+0000 7f00b9600640 1 Processor -- start 2024-05-22T23:58:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b9600640 1 -- start start 2024-05-22T23:58:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b9600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b4102340 0x7f00b4199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:05.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b9600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b4104c70 0x7f00b419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b9600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00b41075a0 0x7f00b41a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b9600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f00b410d5b0 con 0x7f00b41075a0 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b9600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f00b410d430 con 0x7f00b4102340 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b9600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f00b410d730 con 0x7f00b4104c70 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b4102340 0x7f00b4199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b2a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b4102340 0x7f00b4199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:33690/0 (socket says 172.21.15.103:33690) 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b2a00640 1 -- 172.21.15.103:0/2837121133 learned_addr learned my addr 172.21.15.103:0/2837121133 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b3400640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00b41075a0 0x7f00b41a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.490+0000 7f00b2000640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b4104c70 0x7f00b419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b2a00640 1 -- 172.21.15.103:0/2837121133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b4104c70 msgr2=0x7f00b419a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b2a00640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b4104c70 0x7f00b419a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b2a00640 1 -- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00b41075a0 msgr2=0x7f00b41a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:05.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b2a00640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00b41075a0 0x7f00b41a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b2a00640 1 -- 172.21.15.103:0/2837121133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b41a1a20 con 0x7f00b4102340 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b3400640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00b41075a0 0x7f00b41a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b2a00640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b4102340 0x7f00b4199e00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00d770 tx=0x7f00ac00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00ab400640 1 -- 172.21.15.103:0/2837121133 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac017070 con 0x7f00b4102340 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00b41a1d10 con 0x7f00b4102340 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00ab400640 1 -- 172.21.15.103:0/2837121133 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00ac005500 con 0x7f00b4102340 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00ab400640 1 -- 172.21.15.103:0/2837121133 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac004e00 con 0x7f00b4102340 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.491+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00b41a21b0 con 0x7f00b4102340 2024-05-22T23:58:05.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.492+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00b4109090 con 0x7f00b4102340 2024-05-22T23:58:05.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.494+0000 7f00ab400640 1 -- 172.21.15.103:0/2837121133 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f00ac004fa0 con 0x7f00b4102340 2024-05-22T23:58:05.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.495+0000 7f00ab400640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f00980a2b90 0x7f00980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:05.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.496+0000 7f00ab400640 1 -- 172.21.15.103:0/2837121133 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f00ac0ced60 con 0x7f00b4102340 2024-05-22T23:58:05.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.496+0000 7f00b2000640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f00980a2b90 0x7f00980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:05.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.496+0000 7f00b2000640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f00980a2b90 0x7f00980a5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f00b419b310 tx=0x7f009c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:05.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.501+0000 7f00ab400640 1 -- 172.21.15.103:0/2837121133 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f00ac09ba20 con 0x7f00b4102340 2024-05-22T23:58:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] up:active 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: Cluster is now healthy 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/387992300' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-22T23:58:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:05 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-22T23:58:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-22T23:58:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] up:active 2024-05-22T23:58:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-22T23:58:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-22T23:58:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: Cluster is now healthy 2024-05-22T23:58:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-05-22T23:58:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-05-22T23:58:05.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-22T23:58:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/387992300' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-22T23:58:05.581 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:05 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] up:active 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: Cluster is now healthy 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/387992300' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-22T23:58:05.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:05 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-22T23:58:05.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.727+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f00b4061cb0 con 0x7f00980a2b90 2024-05-22T23:58:05.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.767+0000 7f00ab400640 1 -- 172.21.15.103:0/2837121133 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f00b4061cb0 con 0x7f00980a2b90 2024-05-22T23:58:05.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f00980a2b90 msgr2=0x7f00980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:05.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f00980a2b90 0x7f00980a5050 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f00b419b310 tx=0x7f009c00b390 comp rx=0 tx=0).stop 2024-05-22T23:58:05.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b4102340 msgr2=0x7f00b4199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:05.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b4102340 0x7f00b4199e00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f00ac00d770 tx=0x7f00ac00dc30 comp rx=0 tx=0).stop 2024-05-22T23:58:05.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 shutdown_connections 2024-05-22T23:58:05.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f00980a2b90 0x7f00980a5050 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f00b41075a0 0x7f00b41a13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b4104c70 0x7f00b419a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 --2- 172.21.15.103:0/2837121133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f00b4102340 0x7f00b4199e00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:05.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 >> 172.21.15.103:0/2837121133 conn(0x7f00b40fbd30 msgr2=0x7f00b4101a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:05.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 shutdown_connections 2024-05-22T23:58:05.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.770+0000 7f00b9600640 1 -- 172.21.15.103:0/2837121133 wait complete. 2024-05-22T23:58:05.792 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-22T23:58:06.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.997+0000 7f31a9400640 1 -- 172.21.15.103:0/494903240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31a40742d0 msgr2=0x7f31a4074750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.997+0000 7f31a9400640 1 --2- 172.21.15.103:0/494903240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31a40742d0 0x7f31a4074750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f318c009900 tx=0x7f318c02a400 comp rx=0 tx=0).stop 2024-05-22T23:58:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.997+0000 7f31a9400640 1 -- 172.21.15.103:0/494903240 shutdown_connections 2024-05-22T23:58:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.997+0000 7f31a9400640 1 --2- 172.21.15.103:0/494903240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a4109770 0x7f31a410bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.997+0000 7f31a9400640 1 --2- 172.21.15.103:0/494903240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31a40742d0 0x7f31a4074750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.997+0000 7f31a9400640 1 --2- 172.21.15.103:0/494903240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a40758b0 0x7f31a4073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.997+0000 7f31a9400640 1 -- 172.21.15.103:0/494903240 >> 172.21.15.103:0/494903240 conn(0x7f31a40fce20 msgr2=0x7f31a40ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.997+0000 7f31a9400640 1 -- 172.21.15.103:0/494903240 shutdown_connections 2024-05-22T23:58:06.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:05.999+0000 7f31a9400640 1 -- 172.21.15.103:0/494903240 wait complete. 2024-05-22T23:58:06.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.000+0000 7f31a9400640 1 Processor -- start 2024-05-22T23:58:06.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.000+0000 7f31a9400640 1 -- start start 2024-05-22T23:58:06.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.000+0000 7f31a9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a40742d0 0x7f31a4199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:06.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.000+0000 7f31a9400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31a40758b0 0x7f31a419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:06.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.000+0000 7f31a9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a4109770 0x7f31a41a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:06.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.000+0000 7f31a9400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31a410f6f0 con 0x7f31a4109770 2024-05-22T23:58:06.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.000+0000 7f31a9400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f31a410f570 con 0x7f31a40742d0 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.000+0000 7f31a9400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f31a410f870 con 0x7f31a40758b0 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a40742d0 0x7f31a4199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a4109770 0x7f31a41a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a40742d0 0x7f31a4199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:33726/0 (socket says 172.21.15.103:33726) 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 -- 172.21.15.103:0/1153394891 learned_addr learned my addr 172.21.15.103:0/1153394891 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f3199e00640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31a40758b0 0x7f31a419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 -- 172.21.15.103:0/1153394891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31a40758b0 msgr2=0x7f31a419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31a40758b0 0x7f31a419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 -- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a4109770 msgr2=0x7f31a41a13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a4109770 0x7f31a41a13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 -- 172.21.15.103:0/1153394891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a41a1a60 con 0x7f31a40742d0 2024-05-22T23:58:06.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a2800640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a40742d0 0x7f31a4199e40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f319400e9f0 tx=0x7f319400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f319be00640 1 -- 172.21.15.103:0/1153394891 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319400ccc0 con 0x7f31a40742d0 2024-05-22T23:58:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f319be00640 1 -- 172.21.15.103:0/1153394891 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3194010910 con 0x7f31a40742d0 2024-05-22T23:58:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a41a1d50 con 0x7f31a40742d0 2024-05-22T23:58:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f319be00640 1 -- 172.21.15.103:0/1153394891 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31940188a0 con 0x7f31a40742d0 2024-05-22T23:58:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.001+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a41a21c0 con 0x7f31a40742d0 2024-05-22T23:58:06.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.003+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a41084a0 con 0x7f31a40742d0 2024-05-22T23:58:06.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.004+0000 7f319be00640 1 -- 172.21.15.103:0/1153394891 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) ==== 137834+0+0 (secure 0 0 0) 0x7f3194010ac0 con 0x7f31a40742d0 2024-05-22T23:58:06.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.006+0000 7f319be00640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f31880a2b90 0x7f31880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:06.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.007+0000 7f319be00640 1 -- 172.21.15.103:0/1153394891 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f31940cf6d0 con 0x7f31a40742d0 2024-05-22T23:58:06.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.007+0000 7f3199e00640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f31880a2b90 0x7f31880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:06.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.008+0000 7f3199e00640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f31880a2b90 0x7f31880a5050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f318c0314e0 tx=0x7f318c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:06.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.010+0000 7f319be00640 1 -- 172.21.15.103:0/1153394891 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f319409c390 con 0x7f31a40742d0 2024-05-22T23:58:06.080 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:58:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:58:05.741+0000 7fb3eb200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-22T23:58:06.080 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:58:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:58:05.741+0000 7fb3eb200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-22T23:58:06.080 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:58:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:58:05.741+0000 7fb3eb200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-22T23:58:06.080 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:58:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:58:05.741+0000 7fb3eb200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-22T23:58:06.080 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 22 23:58:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-22T23:58:05.741+0000 7fb3eb200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-22T23:58:06.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.243+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f31a4061d40 con 0x7f31880a2b90 2024-05-22T23:58:06.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.287+0000 7f319be00640 1 -- 172.21.15.103:0/1153394891 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f3194098300 con 0x7f31a40742d0 2024-05-22T23:58:06.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.296+0000 7f319be00640 1 -- 172.21.15.103:0/1153394891 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f31a4061d40 con 0x7f31880a2b90 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.298+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f31880a2b90 msgr2=0x7f31880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.298+0000 7f31a9400640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f31880a2b90 0x7f31880a5050 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f318c0314e0 tx=0x7f318c000bc0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.298+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a40742d0 msgr2=0x7f31a4199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.298+0000 7f31a9400640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a40742d0 0x7f31a4199e40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f319400e9f0 tx=0x7f319400edf0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.299+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 shutdown_connections 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.299+0000 7f31a9400640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f31880a2b90 0x7f31880a5050 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.299+0000 7f31a9400640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31a4109770 0x7f31a41a13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.299+0000 7f31a9400640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31a40758b0 0x7f31a419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.299+0000 7f31a9400640 1 --2- 172.21.15.103:0/1153394891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31a40742d0 0x7f31a4199e40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.299+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 >> 172.21.15.103:0/1153394891 conn(0x7f31a40fce20 msgr2=0x7f31a4105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.299+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 shutdown_connections 2024-05-22T23:58:06.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.299+0000 7f31a9400640 1 -- 172.21.15.103:0/1153394891 wait complete. 2024-05-22T23:58:06.319 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.514+0000 7ff65b600640 1 -- 172.21.15.103:0/770370375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff654102ab0 msgr2=0x7ff654102f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.514+0000 7ff65b600640 1 --2- 172.21.15.103:0/770370375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff654102ab0 0x7ff654102f30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009900 tx=0x7ff63c02a3f0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.515+0000 7ff65b600640 1 -- 172.21.15.103:0/770370375 shutdown_connections 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.515+0000 7ff65b600640 1 --2- 172.21.15.103:0/770370375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff654103470 0x7ff654109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.515+0000 7ff65b600640 1 --2- 172.21.15.103:0/770370375 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff654102ab0 0x7ff654102f30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.515+0000 7ff65b600640 1 --2- 172.21.15.103:0/770370375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6541018c0 0x7ff654101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.515+0000 7ff65b600640 1 -- 172.21.15.103:0/770370375 >> 172.21.15.103:0/770370375 conn(0x7ff6540fce50 msgr2=0x7ff6540ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.515+0000 7ff65b600640 1 -- 172.21.15.103:0/770370375 shutdown_connections 2024-05-22T23:58:06.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.515+0000 7ff65b600640 1 -- 172.21.15.103:0/770370375 wait complete. 2024-05-22T23:58:06.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff65b600640 1 Processor -- start 2024-05-22T23:58:06.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff65b600640 1 -- start start 2024-05-22T23:58:06.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff65b600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6541018c0 0x7ff654197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:06.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff65b600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff654102ab0 0x7ff654198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:06.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff65b600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff654103470 0x7ff65419f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:06.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff65b600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff65410d570 con 0x7ff654102ab0 2024-05-22T23:58:06.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff65b600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff65410d3f0 con 0x7ff6541018c0 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff65b600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff65410d6f0 con 0x7ff654103470 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.516+0000 7ff658c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6541018c0 0x7ff654197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff658c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6541018c0 0x7ff654197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:33744/0 (socket says 172.21.15.103:33744) 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff658c00640 1 -- 172.21.15.103:0/3507607364 learned_addr learned my addr 172.21.15.103:0/3507607364 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff653e00640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff654102ab0 0x7ff654198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff659600640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff654103470 0x7ff65419f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff658c00640 1 -- 172.21.15.103:0/3507607364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff654103470 msgr2=0x7ff65419f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff658c00640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff654103470 0x7ff65419f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff658c00640 1 -- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff654102ab0 msgr2=0x7ff654198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff658c00640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff654102ab0 0x7ff654198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff658c00640 1 -- 172.21.15.103:0/3507607364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff65419f810 con 0x7ff6541018c0 2024-05-22T23:58:06.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.517+0000 7ff658c00640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6541018c0 0x7ff654197bf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff644009d20 tx=0x7ff64400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:06.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.518+0000 7ff651600640 1 -- 172.21.15.103:0/3507607364 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644010630 con 0x7ff6541018c0 2024-05-22T23:58:06.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.518+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff65419fb00 con 0x7ff6541018c0 2024-05-22T23:58:06.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.518+0000 7ff651600640 1 -- 172.21.15.103:0/3507607364 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff644010cb0 con 0x7ff6541018c0 2024-05-22T23:58:06.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.518+0000 7ff651600640 1 -- 172.21.15.103:0/3507607364 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff644018ce0 con 0x7ff6541018c0 2024-05-22T23:58:06.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.518+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6541a0070 con 0x7ff6541018c0 2024-05-22T23:58:06.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.519+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff654108c60 con 0x7ff6541018c0 2024-05-22T23:58:06.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.521+0000 7ff651600640 1 -- 172.21.15.103:0/3507607364 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff64401f020 con 0x7ff6541018c0 2024-05-22T23:58:06.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.522+0000 7ff651600640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff6380a2d20 0x7ff6380a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:06.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.522+0000 7ff651600640 1 -- 172.21.15.103:0/3507607364 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff6440cf770 con 0x7ff6541018c0 2024-05-22T23:58:06.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.523+0000 7ff653e00640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff6380a2d20 0x7ff6380a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:06.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.523+0000 7ff653e00640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff6380a2d20 0x7ff6380a51e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff63c031440 tx=0x7ff63c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:06.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.526+0000 7ff651600640 1 -- 172.21.15.103:0/3507607364 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff64409c2d0 con 0x7ff6541018c0 2024-05-22T23:58:06.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:06 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-22T23:58:06.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:06 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:active 2024-05-22T23:58:06.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:06 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-05-22T23:58:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:06 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-22T23:58:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:06 smithi105 ceph-mon[36426]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:06 smithi105 ceph-mon[36426]: pgmap v339: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-05-22T23:58:06.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:06 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-22T23:58:06.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:06 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:active 2024-05-22T23:58:06.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:06 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-05-22T23:58:06.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:06 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-22T23:58:06.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:06 smithi103 ceph-mon[31370]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:06.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:06 smithi103 ceph-mon[31370]: pgmap v339: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-05-22T23:58:06.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:06 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-22T23:58:06.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:06 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:active 2024-05-22T23:58:06.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:06 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-05-22T23:58:06.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:06 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-22T23:58:06.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:06 smithi174 ceph-mon[36179]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:06.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:06 smithi174 ceph-mon[36179]: pgmap v339: 129 pgs: 129 active+clean; 583 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-05-22T23:58:06.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.760+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7ff654061d40 con 0x7ff6380a2d20 2024-05-22T23:58:06.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.768+0000 7ff651600640 1 -- 172.21.15.103:0/3507607364 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7ff654061d40 con 0x7ff6380a2d20 2024-05-22T23:58:06.771 INFO:teuthology.orchestra.run.smithi103.stdout:/volumes/qa/sv_0/b6e9ed96-7fee-4f77-973f-c5730669227c 2024-05-22T23:58:06.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff6380a2d20 msgr2=0x7ff6380a51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff6380a2d20 0x7ff6380a51e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff63c031440 tx=0x7ff63c0315b0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6541018c0 msgr2=0x7ff654197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:06.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6541018c0 0x7ff654197bf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff644009d20 tx=0x7ff64400c6a0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 shutdown_connections 2024-05-22T23:58:06.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff6380a2d20 0x7ff6380a51e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff654103470 0x7ff65419f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff654102ab0 0x7ff654198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 --2- 172.21.15.103:0/3507607364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6541018c0 0x7ff654197bf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:06.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 >> 172.21.15.103:0/3507607364 conn(0x7ff6540fce50 msgr2=0x7ff654105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:06.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.770+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 shutdown_connections 2024-05-22T23:58:06.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.771+0000 7ff65b600640 1 -- 172.21.15.103:0/3507607364 wait complete. 2024-05-22T23:58:06.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-22T23:58:07.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 -- 172.21.15.103:0/205905146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480103990 msgr2=0x7f8480103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 --2- 172.21.15.103:0/205905146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480103990 0x7f8480103db0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8470009900 tx=0x7f847002a3f0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 -- 172.21.15.103:0/205905146 shutdown_connections 2024-05-22T23:58:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 --2- 172.21.15.103:0/205905146 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8480105540 0x7f848010bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 --2- 172.21.15.103:0/205905146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8480104b80 0x7f8480105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 --2- 172.21.15.103:0/205905146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480103990 0x7f8480103db0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 -- 172.21.15.103:0/205905146 >> 172.21.15.103:0/205905146 conn(0x7f84800fef20 msgr2=0x7f8480101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 -- 172.21.15.103:0/205905146 shutdown_connections 2024-05-22T23:58:07.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:06.999+0000 7f8484c00640 1 -- 172.21.15.103:0/205905146 wait complete. 2024-05-22T23:58:07.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.000+0000 7f8484c00640 1 Processor -- start 2024-05-22T23:58:07.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.000+0000 7f8484c00640 1 -- start start 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f8484c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8480103990 0x7f8480199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f8484c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8480104b80 0x7f848019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f8484c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480105540 0x7f84801a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f8484c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f848010f650 con 0x7f8480105540 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f8484c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f848010f4d0 con 0x7f8480104b80 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f8484c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f848010f7d0 con 0x7f8480103990 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480105540 0x7f84801a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480105540 0x7f84801a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58464/0 (socket says 172.21.15.103:58464) 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847fe00640 1 -- 172.21.15.103:0/1398684938 learned_addr learned my addr 172.21.15.103:0/1398684938 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847f400640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8480103990 0x7f8480199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f8476a00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8480104b80 0x7f848019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847fe00640 1 -- 172.21.15.103:0/1398684938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8480103990 msgr2=0x7f8480199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847fe00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8480103990 0x7f8480199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847fe00640 1 -- 172.21.15.103:0/1398684938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8480104b80 msgr2=0x7f848019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847fe00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8480104b80 0x7f848019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.001+0000 7f847fe00640 1 -- 172.21.15.103:0/1398684938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84801a1960 con 0x7f8480105540 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.002+0000 7f847fe00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480105540 0x7f84801a12c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f847800eb10 tx=0x7f847800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.002+0000 7f847cc00640 1 -- 172.21.15.103:0/1398684938 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8478004b20 con 0x7f8480105540 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.002+0000 7f847cc00640 1 -- 172.21.15.103:0/1398684938 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8478004cc0 con 0x7f8480105540 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.002+0000 7f847cc00640 1 -- 172.21.15.103:0/1398684938 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8478010760 con 0x7f8480105540 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.002+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84801a1c50 con 0x7f8480105540 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.002+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84801a2120 con 0x7f8480105540 2024-05-22T23:58:07.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.004+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f848010ad20 con 0x7f8480105540 2024-05-22T23:58:07.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.005+0000 7f847cc00640 1 -- 172.21.15.103:0/1398684938 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f8478005020 con 0x7f8480105540 2024-05-22T23:58:07.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.007+0000 7f847cc00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f84640a2c20 0x7f84640a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.007+0000 7f847cc00640 1 -- 172.21.15.103:0/1398684938 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f84780cfab0 con 0x7f8480105540 2024-05-22T23:58:07.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.007+0000 7f847f400640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f84640a2c20 0x7f84640a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.008+0000 7f847f400640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f84640a2c20 0x7f84640a50e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f847002a9e0 tx=0x7f84700315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:07.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.012+0000 7f847cc00640 1 -- 172.21.15.103:0/1398684938 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f847809c610 con 0x7f8480105540 2024-05-22T23:58:07.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.245+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8480061cb0 con 0x7f84640a2c20 2024-05-22T23:58:07.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.284+0000 7f847cc00640 1 -- 172.21.15.103:0/1398684938 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8480061cb0 con 0x7f84640a2c20 2024-05-22T23:58:07.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f84640a2c20 msgr2=0x7f84640a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f84640a2c20 0x7f84640a50e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f847002a9e0 tx=0x7f84700315b0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480105540 msgr2=0x7f84801a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480105540 0x7f84801a12c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f847800eb10 tx=0x7f847800c2d0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 shutdown_connections 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f84640a2c20 0x7f84640a50e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8480105540 0x7f84801a12c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8480104b80 0x7f848019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 --2- 172.21.15.103:0/1398684938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8480103990 0x7f8480199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.287+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 >> 172.21.15.103:0/1398684938 conn(0x7f84800fef20 msgr2=0x7f8480107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.288+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 shutdown_connections 2024-05-22T23:58:07.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.288+0000 7f8484c00640 1 -- 172.21.15.103:0/1398684938 wait complete. 2024-05-22T23:58:07.310 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-22T23:58:07.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 -- 172.21.15.103:0/3823018657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f8077e10 msgr2=0x7fa9f80786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 --2- 172.21.15.103:0/3823018657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f8077e10 0x7fa9f80786b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009ca0 tx=0x7fa9f002a8c0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 -- 172.21.15.103:0/3823018657 shutdown_connections 2024-05-22T23:58:07.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 --2- 172.21.15.103:0/3823018657 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f8077e10 0x7fa9f80786b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 --2- 172.21.15.103:0/3823018657 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9f8077470 0x7fa9f80778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 --2- 172.21.15.103:0/3823018657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8076230 0x7fa9f8076650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 -- 172.21.15.103:0/3823018657 >> 172.21.15.103:0/3823018657 conn(0x7fa9f80ff060 msgr2=0x7fa9f8101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:07.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 -- 172.21.15.103:0/3823018657 shutdown_connections 2024-05-22T23:58:07.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.514+0000 7fa9fca00640 1 -- 172.21.15.103:0/3823018657 wait complete. 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.515+0000 7fa9fca00640 1 Processor -- start 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.515+0000 7fa9fca00640 1 -- start start 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.515+0000 7fa9fca00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9f8076230 0x7fa9f819e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9fca00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f8077470 0x7fa9f819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9fca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8077e10 0x7fa9f81a5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9fca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f8113b60 con 0x7fa9f8077e10 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9fca00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81139e0 con 0x7fa9f8076230 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9fca00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f8113ce0 con 0x7fa9f8077470 2024-05-22T23:58:07.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9f8076230 0x7fa9f819e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9f8076230 0x7fa9f819e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:33794/0 (socket says 172.21.15.103:33794) 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f5e00640 1 -- 172.21.15.103:0/1698382478 learned_addr learned my addr 172.21.15.103:0/1698382478 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f6800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8077e10 0x7fa9f81a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9efe00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f8077470 0x7fa9f819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f5e00640 1 -- 172.21.15.103:0/1698382478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f8077470 msgr2=0x7fa9f819e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f5e00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f8077470 0x7fa9f819e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f5e00640 1 -- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8077e10 msgr2=0x7fa9f81a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f5e00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8077e10 0x7fa9f81a5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.516+0000 7fa9f5e00640 1 -- 172.21.15.103:0/1698382478 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f81a5dd0 con 0x7fa9f8076230 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.517+0000 7fa9f5e00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9f8076230 0x7fa9f819e1b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800e9f0 tx=0x7fa9e800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.517+0000 7fa9eea00640 1 -- 172.21.15.103:0/1698382478 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e800cd80 con 0x7fa9f8076230 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.517+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f81a60c0 con 0x7fa9f8076230 2024-05-22T23:58:07.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.517+0000 7fa9eea00640 1 -- 172.21.15.103:0/1698382478 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e8010910 con 0x7fa9f8076230 2024-05-22T23:58:07.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.517+0000 7fa9eea00640 1 -- 172.21.15.103:0/1698382478 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80188f0 con 0x7fa9f8076230 2024-05-22T23:58:07.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.517+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f81a6660 con 0x7fa9f8076230 2024-05-22T23:58:07.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.518+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f8110440 con 0x7fa9f8076230 2024-05-22T23:58:07.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.520+0000 7fa9eea00640 1 -- 172.21.15.103:0/1698382478 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fa9e8010ac0 con 0x7fa9f8076230 2024-05-22T23:58:07.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.522+0000 7fa9eea00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9dc0a2ca0 0x7fa9dc0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.522+0000 7fa9eea00640 1 -- 172.21.15.103:0/1698382478 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa9e80cf7b0 con 0x7fa9f8076230 2024-05-22T23:58:07.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.522+0000 7fa9efe00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9dc0a2ca0 0x7fa9dc0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.523+0000 7fa9efe00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9dc0a2ca0 0x7fa9dc0a5160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa9f819f6c0 tx=0x7fa9e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:07.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.525+0000 7fa9eea00640 1 -- 172.21.15.103:0/1698382478 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa9e809bcc0 con 0x7fa9f8076230 2024-05-22T23:58:07.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:07 smithi105 ceph-mon[36426]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:07.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:07 smithi105 ceph-mon[36426]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-22T23:58:07.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:07 smithi105 ceph-mon[36426]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:07.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:07 smithi103 ceph-mon[31370]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:07.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:07 smithi103 ceph-mon[31370]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-22T23:58:07.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:07 smithi103 ceph-mon[31370]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:07.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:07 smithi174 ceph-mon[36179]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:07.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:07 smithi174 ceph-mon[36179]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-05-22T23:58:07.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:07 smithi174 ceph-mon[36179]: from='client.24833 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:07.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.760+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fa9f8061d40 con 0x7fa9dc0a2ca0 2024-05-22T23:58:07.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.767+0000 7fa9eea00640 1 -- 172.21.15.103:0/1698382478 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fa9f8061d40 con 0x7fa9dc0a2ca0 2024-05-22T23:58:07.771 INFO:teuthology.orchestra.run.smithi103.stdout:/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4 2024-05-22T23:58:07.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9dc0a2ca0 msgr2=0x7fa9dc0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9dc0a2ca0 0x7fa9dc0a5160 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa9f819f6c0 tx=0x7fa9e0005f90 comp rx=0 tx=0).stop 2024-05-22T23:58:07.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9f8076230 msgr2=0x7fa9f819e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9f8076230 0x7fa9f819e1b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800e9f0 tx=0x7fa9e800eeb0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 shutdown_connections 2024-05-22T23:58:07.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9dc0a2ca0 0x7fa9dc0a5160 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8077e10 0x7fa9f81a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f8077470 0x7fa9f819e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 --2- 172.21.15.103:0/1698382478 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9f8076230 0x7fa9f819e1b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 >> 172.21.15.103:0/1698382478 conn(0x7fa9f80ff060 msgr2=0x7fa9f8100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:07.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 shutdown_connections 2024-05-22T23:58:07.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.770+0000 7fa9fca00640 1 -- 172.21.15.103:0/1698382478 wait complete. 2024-05-22T23:58:07.791 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:58:07.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 -- 172.21.15.103:0/692199326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8102900 msgr2=0x7f93d8102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 --2- 172.21.15.103:0/692199326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8102900 0x7f93d8102d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f93c0009900 tx=0x7f93c002a3f0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 -- 172.21.15.103:0/692199326 shutdown_connections 2024-05-22T23:58:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 --2- 172.21.15.103:0/692199326 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93d81032c0 0x7f93d8109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 --2- 172.21.15.103:0/692199326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8102900 0x7f93d8102d80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 --2- 172.21.15.103:0/692199326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93d8101710 0x7f93d8101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 -- 172.21.15.103:0/692199326 >> 172.21.15.103:0/692199326 conn(0x7f93d80fcc80 msgr2=0x7f93d80ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 -- 172.21.15.103:0/692199326 shutdown_connections 2024-05-22T23:58:07.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.984+0000 7f93d7200640 1 -- 172.21.15.103:0/692199326 wait complete. 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.985+0000 7f93d7200640 1 Processor -- start 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d7200640 1 -- start start 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8101710 0x7f93d8075620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d7200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93d8102900 0x7f93d8073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d7200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93d81032c0 0x7f93d80741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93d810d3e0 con 0x7f93d8101710 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d7200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f93d810d260 con 0x7f93d8102900 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d7200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93d810d560 con 0x7f93d81032c0 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8101710 0x7f93d8075620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8101710 0x7f93d8075620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58504/0 (socket says 172.21.15.103:58504) 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d5e00640 1 -- 172.21.15.103:0/2125599761 learned_addr learned my addr 172.21.15.103:0/2125599761 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93cd400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93d8102900 0x7f93d8073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.986+0000 7f93d6800640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93d81032c0 0x7f93d80741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93d5e00640 1 -- 172.21.15.103:0/2125599761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93d81032c0 msgr2=0x7f93d80741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93d5e00640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93d81032c0 0x7f93d80741a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93d5e00640 1 -- 172.21.15.103:0/2125599761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93d8102900 msgr2=0x7f93d8073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93d5e00640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93d8102900 0x7f93d8073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93d5e00640 1 -- 172.21.15.103:0/2125599761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d8074a30 con 0x7f93d8101710 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93cd400640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93d8102900 0x7f93d8073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:07.991 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93d5e00640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8101710 0x7f93d8075620 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93c800eb70 tx=0x7f93c800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:07.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93cf400640 1 -- 172.21.15.103:0/2125599761 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8003b70 con 0x7f93d8101710 2024-05-22T23:58:07.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93cf400640 1 -- 172.21.15.103:0/2125599761 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93c800ee70 con 0x7f93d8101710 2024-05-22T23:58:07.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.987+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d81a1b50 con 0x7f93d8101710 2024-05-22T23:58:07.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.988+0000 7f93cf400640 1 -- 172.21.15.103:0/2125599761 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c8010830 con 0x7f93d8101710 2024-05-22T23:58:07.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.988+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d81a2020 con 0x7f93d8101710 2024-05-22T23:58:07.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.989+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d80685b0 con 0x7f93d8101710 2024-05-22T23:58:07.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.991+0000 7f93cf400640 1 -- 172.21.15.103:0/2125599761 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f93c8004020 con 0x7f93d8101710 2024-05-22T23:58:07.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.992+0000 7f93cf400640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f93bc0a2c20 0x7f93bc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:07.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.993+0000 7f93cf400640 1 -- 172.21.15.103:0/2125599761 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f93c80cfb60 con 0x7f93d8101710 2024-05-22T23:58:07.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.993+0000 7f93cd400640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f93bc0a2c20 0x7f93bc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:07.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.994+0000 7f93cd400640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f93bc0a2c20 0x7f93bc0a50e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f93c0031440 tx=0x7f93c00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:08.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:07.997+0000 7f93cf400640 1 -- 172.21.15.103:0/2125599761 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f93c809c6c0 con 0x7f93d8101710 2024-05-22T23:58:08.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.236+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93d8108ab0 con 0x7f93d8101710 2024-05-22T23:58:08.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.238+0000 7f93cf400640 1 -- 172.21.15.103:0/2125599761 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11896 (secure 0 0 0) 0x7f93c809bfc0 con 0x7f93d8101710 2024-05-22T23:58:08.241 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:08.242 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-22T23:58:05.260333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24559,24563,24575]},"id":1}]} 2024-05-22T23:58:08.242 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 24 2024-05-22T23:58:08.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f93bc0a2c20 msgr2=0x7f93bc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f93bc0a2c20 0x7f93bc0a50e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f93c0031440 tx=0x7f93c00315b0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8101710 msgr2=0x7f93d8075620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8101710 0x7f93d8075620 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f93c800eb70 tx=0x7f93c800c6a0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 shutdown_connections 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f93bc0a2c20 0x7f93bc0a50e0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93d81032c0 0x7f93d80741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93d8102900 0x7f93d8073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 --2- 172.21.15.103:0/2125599761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93d8101710 0x7f93d8075620 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 >> 172.21.15.103:0/2125599761 conn(0x7f93d80fcc80 msgr2=0x7f93d8105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.241+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 shutdown_connections 2024-05-22T23:58:08.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.242+0000 7f93d7200640 1 -- 172.21.15.103:0/2125599761 wait complete. 2024-05-22T23:58:08.264 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:58:08.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 -- 172.21.15.103:0/287104143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30109780 msgr2=0x7faa3010bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 --2- 172.21.15.103:0/287104143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30109780 0x7faa3010bc10 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faa28009900 tx=0x7faa2802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 -- 172.21.15.103:0/287104143 shutdown_connections 2024-05-22T23:58:08.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 --2- 172.21.15.103:0/287104143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30109780 0x7faa3010bc10 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 --2- 172.21.15.103:0/287104143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa300742d0 0x7faa30074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 --2- 172.21.15.103:0/287104143 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa300758b0 0x7faa30073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 -- 172.21.15.103:0/287104143 >> 172.21.15.103:0/287104143 conn(0x7faa300fce50 msgr2=0x7faa300ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:08.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 -- 172.21.15.103:0/287104143 shutdown_connections 2024-05-22T23:58:08.474 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.470+0000 7faa37c00640 1 -- 172.21.15.103:0/287104143 wait complete. 2024-05-22T23:58:08.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.471+0000 7faa37c00640 1 Processor -- start 2024-05-22T23:58:08.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.471+0000 7faa37c00640 1 -- start start 2024-05-22T23:58:08.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.471+0000 7faa37c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa300742d0 0x7faa30199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:08.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.471+0000 7faa37c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa300758b0 0x7faa3019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:08.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.471+0000 7faa37c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30109780 0x7faa301a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:08.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.471+0000 7faa37c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010f770 con 0x7faa30109780 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.471+0000 7faa37c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010f5f0 con 0x7faa300758b0 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.471+0000 7faa37c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faa3010f8f0 con 0x7faa300742d0 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa300742d0 0x7faa30199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa2fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa300758b0 0x7faa3019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa300742d0 0x7faa30199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:35742/0 (socket says 172.21.15.103:35742) 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 -- 172.21.15.103:0/4169150445 learned_addr learned my addr 172.21.15.103:0/4169150445 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35c00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30109780 0x7faa301a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 -- 172.21.15.103:0/4169150445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa300758b0 msgr2=0x7faa3019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa300758b0 0x7faa3019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 -- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30109780 msgr2=0x7faa301a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30109780 0x7faa301a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 -- 172.21.15.103:0/4169150445 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa301a19d0 con 0x7faa300742d0 2024-05-22T23:58:08.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa2fe00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa300758b0 0x7faa3019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:08.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.472+0000 7faa35200640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa300742d0 0x7faa30199dc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa20009d50 tx=0x7faa2000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:08.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.473+0000 7faa2d600640 1 -- 172.21.15.103:0/4169150445 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20003b70 con 0x7faa300742d0 2024-05-22T23:58:08.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.473+0000 7faa2d600640 1 -- 172.21.15.103:0/4169150445 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa20003d10 con 0x7faa300742d0 2024-05-22T23:58:08.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.473+0000 7faa2d600640 1 -- 172.21.15.103:0/4169150445 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa20010a20 con 0x7faa300742d0 2024-05-22T23:58:08.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.473+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa301a1cc0 con 0x7faa300742d0 2024-05-22T23:58:08.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.473+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa301a21c0 con 0x7faa300742d0 2024-05-22T23:58:08.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.474+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa301084b0 con 0x7faa300742d0 2024-05-22T23:58:08.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.476+0000 7faa2d600640 1 -- 172.21.15.103:0/4169150445 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7faa20003ec0 con 0x7faa300742d0 2024-05-22T23:58:08.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.477+0000 7faa2d600640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faa140a2ca0 0x7faa140a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:08.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.477+0000 7faa2d600640 1 -- 172.21.15.103:0/4169150445 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7faa200cfab0 con 0x7faa300742d0 2024-05-22T23:58:08.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.478+0000 7faa2fe00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faa140a2ca0 0x7faa140a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:08.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.478+0000 7faa2fe00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faa140a2ca0 0x7faa140a5160 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009fc0 tx=0x7faa1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:08.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.482+0000 7faa2d600640 1 -- 172.21.15.103:0/4169150445 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7faa2009c670 con 0x7faa300742d0 2024-05-22T23:58:08.559 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 22 23:58:08 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b[66976]: 2024-05-22T23:58:08.163+0000 7f633d485700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-05-22T23:58:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:08 smithi105 ceph-mon[36426]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:08 smithi105 ceph-mon[36426]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:08 smithi105 ceph-mon[36426]: pgmap v340: 129 pgs: 129 active+clean; 620 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-05-22T23:58:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:08 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2125599761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:58:08.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:08 smithi103 ceph-mon[31370]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:08.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:08 smithi103 ceph-mon[31370]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:08.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:08 smithi103 ceph-mon[31370]: pgmap v340: 129 pgs: 129 active+clean; 620 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-05-22T23:58:08.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:08 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2125599761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:58:08.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:08 smithi174 ceph-mon[36179]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:08.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:08 smithi174 ceph-mon[36179]: from='client.24845 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:58:08.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:08 smithi174 ceph-mon[36179]: pgmap v340: 129 pgs: 129 active+clean; 620 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5.7 KiB/s wr, 9 op/s 2024-05-22T23:58:08.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:08 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2125599761' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:58:08.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.684+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa3002d490 con 0x7faa300742d0 2024-05-22T23:58:08.688 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.686+0000 7faa2d600640 1 -- 172.21.15.103:0/4169150445 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7faa2009bf70 con 0x7faa300742d0 2024-05-22T23:58:08.689 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:08.689 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.688+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faa140a2ca0 msgr2=0x7faa140a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.688+0000 7faa37c00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faa140a2ca0 0x7faa140a5160 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faa1c009fc0 tx=0x7faa1c005f90 comp rx=0 tx=0).stop 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.688+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa300742d0 msgr2=0x7faa30199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.688+0000 7faa37c00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa300742d0 0x7faa30199dc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faa20009d50 tx=0x7faa2000c6a0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.689+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 shutdown_connections 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.689+0000 7faa37c00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faa140a2ca0 0x7faa140a5160 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.689+0000 7faa37c00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa30109780 0x7faa301a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.689+0000 7faa37c00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa300758b0 0x7faa3019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.689+0000 7faa37c00640 1 --2- 172.21.15.103:0/4169150445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa300742d0 0x7faa30199dc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.689+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 >> 172.21.15.103:0/4169150445 conn(0x7faa300fce50 msgr2=0x7faa30105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.689+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 shutdown_connections 2024-05-22T23:58:08.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.689+0000 7faa37c00640 1 -- 172.21.15.103:0/4169150445 wait complete. 2024-05-22T23:58:08.710 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-22T23:58:08.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-22T23:58:08.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 -- 172.21.15.103:0/633432278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 msgr2=0x7fce4c103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 --2- 172.21.15.103:0/633432278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 0x7fce4c103f10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009900 tx=0x7fce3c02a3f0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 -- 172.21.15.103:0/633432278 shutdown_connections 2024-05-22T23:58:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 --2- 172.21.15.103:0/633432278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce4c1056a0 0x7fce4c10bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 --2- 172.21.15.103:0/633432278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce4c104ce0 0x7fce4c105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 --2- 172.21.15.103:0/633432278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 0x7fce4c103f10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 -- 172.21.15.103:0/633432278 >> 172.21.15.103:0/633432278 conn(0x7fce4c0ff060 msgr2=0x7fce4c1014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 -- 172.21.15.103:0/633432278 shutdown_connections 2024-05-22T23:58:08.909 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.905+0000 7fce51400640 1 -- 172.21.15.103:0/633432278 wait complete. 2024-05-22T23:58:08.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.906+0000 7fce51400640 1 Processor -- start 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce51400640 1 -- start start 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce51400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 0x7fce4c199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce51400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce4c104ce0 0x7fce4c19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce51400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce4c1056a0 0x7fce4c1a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce51400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c10f790 con 0x7fce4c103af0 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce51400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c10f610 con 0x7fce4c104ce0 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce51400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c10f910 con 0x7fce4c1056a0 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce4a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 0x7fce4c199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce4b200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce4c1056a0 0x7fce4c1a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce4a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 0x7fce4c199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58548/0 (socket says 172.21.15.103:58548) 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce4a800640 1 -- 172.21.15.103:0/1613444274 learned_addr learned my addr 172.21.15.103:0/1613444274 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:08.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.907+0000 7fce49e00640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce4c104ce0 0x7fce4c19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce4b200640 1 -- 172.21.15.103:0/1613444274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce4c104ce0 msgr2=0x7fce4c19a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce4b200640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce4c104ce0 0x7fce4c19a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce4b200640 1 -- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 msgr2=0x7fce4c199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce4b200640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 0x7fce4c199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce4b200640 1 -- 172.21.15.103:0/1613444274 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c1a1a70 con 0x7fce4c1056a0 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce4b200640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce4c1056a0 0x7fce4c1a1400 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fce4400ec50 tx=0x7fce4400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce43400640 1 -- 172.21.15.103:0/1613444274 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44003b70 con 0x7fce4c1056a0 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce43400640 1 -- 172.21.15.103:0/1613444274 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce44003d10 con 0x7fce4c1056a0 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4c1a1d60 con 0x7fce4c1056a0 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce43400640 1 -- 172.21.15.103:0/1613444274 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44010890 con 0x7fce4c1056a0 2024-05-22T23:58:08.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.908+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4c1a2200 con 0x7fce4c1056a0 2024-05-22T23:58:08.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.909+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce4c10ae10 con 0x7fce4c1056a0 2024-05-22T23:58:08.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.911+0000 7fce43400640 1 -- 172.21.15.103:0/1613444274 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fce44004b40 con 0x7fce4c1056a0 2024-05-22T23:58:08.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.913+0000 7fce43400640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fce300a2c20 0x7fce300a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:08.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.913+0000 7fce43400640 1 -- 172.21.15.103:0/1613444274 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fce440cf9d0 con 0x7fce4c1056a0 2024-05-22T23:58:08.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.913+0000 7fce4a800640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fce300a2c20 0x7fce300a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:08.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.914+0000 7fce4a800640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fce300a2c20 0x7fce300a50e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fce3c000c00 tx=0x7fce3c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:08.918 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:08.915+0000 7fce43400640 1 -- 172.21.15.103:0/1613444274 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fce4409c530 con 0x7fce4c1056a0 2024-05-22T23:58:09.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.113+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fce4c061cb0 con 0x7fce300a2c20 2024-05-22T23:58:09.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.131+0000 7fce43400640 1 -- 172.21.15.103:0/1613444274 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340624 (secure 0 0 0) 0x7fce4c061cb0 con 0x7fce300a2c20 2024-05-22T23:58:09.140 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:09.142 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fce300a2c20 msgr2=0x7fce300a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fce300a2c20 0x7fce300a50e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fce3c000c00 tx=0x7fce3c0315b0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce4c1056a0 msgr2=0x7fce4c1a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce4c1056a0 0x7fce4c1a1400 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fce4400ec50 tx=0x7fce4400c6a0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 shutdown_connections 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fce300a2c20 0x7fce300a50e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce4c1056a0 0x7fce4c1a1400 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce4c104ce0 0x7fce4c19a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 --2- 172.21.15.103:0/1613444274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce4c103af0 0x7fce4c199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 >> 172.21.15.103:0/1613444274 conn(0x7fce4c0ff060 msgr2=0x7fce4c107fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:09.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 shutdown_connections 2024-05-22T23:58:09.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.139+0000 7fce51400640 1 -- 172.21.15.103:0/1613444274 wait complete. 2024-05-22T23:58:09.163 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-05-22T23:58:08.117340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634474,"num_objects":60,"num_object_clones":0,"num_object_copies":181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":155,"num_read_kb":172,"num_write":319,"num_write_kb":3024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356912,"kb_used_data":11920,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584912,"statfs":{"total":1151940427776,"available":1151574949888,"internally_reserved":0,"allocated":12206080,"data_stored":7655071,"data_compressed":282852,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39608,"num_objects":36,"num_object_clones":0,"num_object_copies":109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":19,"num_read_kb":53,"num_write":90,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006272"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668473+0000","last_change":"2024-05-22T23:57:33.328075+0000","last_active":"2024-05-22T23:57:38.668473+0000","last_peered":"2024-05-22T23:57:38.668473+0000","last_clean":"2024-05-22T23:57:38.668473+0000","last_became_active":"2024-05-22T23:57:33.327704+0000","last_became_peered":"2024-05-22T23:57:33.327704+0000","last_unstale":"2024-05-22T23:57:38.668473+0000","last_undegraded":"2024-05-22T23:57:38.668473+0000","last_fullsized":"2024-05-22T23:57:38.668473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:42:32.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338751+0000","last_change":"2024-05-22T23:57:33.332043+0000","last_active":"2024-05-22T23:57:38.338751+0000","last_peered":"2024-05-22T23:57:38.338751+0000","last_clean":"2024-05-22T23:57:38.338751+0000","last_became_active":"2024-05-22T23:57:33.322269+0000","last_became_peered":"2024-05-22T23:57:33.322269+0000","last_unstale":"2024-05-22T23:57:38.338751+0000","last_undegraded":"2024-05-22T23:57:38.338751+0000","last_fullsized":"2024-05-22T23:57:38.338751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:40:23.542938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338063+0000","last_change":"2024-05-22T23:57:33.334993+0000","last_active":"2024-05-22T23:57:38.338063+0000","last_peered":"2024-05-22T23:57:38.338063+0000","last_clean":"2024-05-22T23:57:38.338063+0000","last_became_active":"2024-05-22T23:57:33.326446+0000","last_became_peered":"2024-05-22T23:57:33.326446+0000","last_unstale":"2024-05-22T23:57:38.338063+0000","last_undegraded":"2024-05-22T23:57:38.338063+0000","last_fullsized":"2024-05-22T23:57:38.338063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:44:45.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336567+0000","last_change":"2024-05-22T23:57:33.320181+0000","last_active":"2024-05-22T23:57:38.336567+0000","last_peered":"2024-05-22T23:57:38.336567+0000","last_clean":"2024-05-22T23:57:38.336567+0000","last_became_active":"2024-05-22T23:57:33.319630+0000","last_became_peered":"2024-05-22T23:57:33.319630+0000","last_unstale":"2024-05-22T23:57:38.336567+0000","last_undegraded":"2024-05-22T23:57:38.336567+0000","last_fullsized":"2024-05-22T23:57:38.336567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:38:51.836241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504159+0000","last_change":"2024-05-22T23:57:33.332168+0000","last_active":"2024-05-22T23:57:38.504159+0000","last_peered":"2024-05-22T23:57:38.504159+0000","last_clean":"2024-05-22T23:57:38.504159+0000","last_became_active":"2024-05-22T23:57:33.331755+0000","last_became_peered":"2024-05-22T23:57:33.331755+0000","last_unstale":"2024-05-22T23:57:38.504159+0000","last_undegraded":"2024-05-22T23:57:38.504159+0000","last_fullsized":"2024-05-22T23:57:38.504159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:28.013365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956794+0000","last_change":"2024-05-22T23:57:33.322449+0000","last_active":"2024-05-22T23:57:39.956794+0000","last_peered":"2024-05-22T23:57:39.956794+0000","last_clean":"2024-05-22T23:57:39.956794+0000","last_became_active":"2024-05-22T23:57:33.322099+0000","last_became_peered":"2024-05-22T23:57:33.322099+0000","last_unstale":"2024-05-22T23:57:39.956794+0000","last_undegraded":"2024-05-22T23:57:39.956794+0000","last_fullsized":"2024-05-22T23:57:39.956794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:38.427434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955072+0000","last_change":"2024-05-22T23:57:33.322391+0000","last_active":"2024-05-22T23:57:39.955072+0000","last_peered":"2024-05-22T23:57:39.955072+0000","last_clean":"2024-05-22T23:57:39.955072+0000","last_became_active":"2024-05-22T23:57:33.322076+0000","last_became_peered":"2024-05-22T23:57:33.322076+0000","last_unstale":"2024-05-22T23:57:39.955072+0000","last_undegraded":"2024-05-22T23:57:39.955072+0000","last_fullsized":"2024-05-22T23:57:39.955072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:45:44.857586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.877939+0000","last_change":"2024-05-22T23:57:33.333005+0000","last_active":"2024-05-22T23:57:39.877939+0000","last_peered":"2024-05-22T23:57:39.877939+0000","last_clean":"2024-05-22T23:57:39.877939+0000","last_became_active":"2024-05-22T23:57:33.332669+0000","last_became_peered":"2024-05-22T23:57:33.332669+0000","last_unstale":"2024-05-22T23:57:39.877939+0000","last_undegraded":"2024-05-22T23:57:39.877939+0000","last_fullsized":"2024-05-22T23:57:39.877939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:47:25.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956357+0000","last_change":"2024-05-22T23:57:33.328428+0000","last_active":"2024-05-22T23:57:39.956357+0000","last_peered":"2024-05-22T23:57:39.956357+0000","last_clean":"2024-05-22T23:57:39.956357+0000","last_became_active":"2024-05-22T23:57:33.327574+0000","last_became_peered":"2024-05-22T23:57:33.327574+0000","last_unstale":"2024-05-22T23:57:39.956357+0000","last_undegraded":"2024-05-22T23:57:39.956357+0000","last_fullsized":"2024-05-22T23:57:39.956357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:35:07.217079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956792+0000","last_change":"2024-05-22T23:57:33.324445+0000","last_active":"2024-05-22T23:57:39.956792+0000","last_peered":"2024-05-22T23:57:39.956792+0000","last_clean":"2024-05-22T23:57:39.956792+0000","last_became_active":"2024-05-22T23:57:33.324085+0000","last_became_peered":"2024-05-22T23:57:33.324085+0000","last_unstale":"2024-05-22T23:57:39.956792+0000","last_undegraded":"2024-05-22T23:57:39.956792+0000","last_fullsized":"2024-05-22T23:57:39.956792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:30:51.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956263+0000","last_change":"2024-05-22T23:57:33.321481+0000","last_active":"2024-05-22T23:57:39.956263+0000","last_peered":"2024-05-22T23:57:39.956263+0000","last_clean":"2024-05-22T23:57:39.956263+0000","last_became_active":"2024-05-22T23:57:33.320911+0000","last_became_peered":"2024-05-22T23:57:33.320911+0000","last_unstale":"2024-05-22T23:57:39.956263+0000","last_undegraded":"2024-05-22T23:57:39.956263+0000","last_fullsized":"2024-05-22T23:57:39.956263+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:24:24.041980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336271+0000","last_change":"2024-05-22T23:57:33.318819+0000","last_active":"2024-05-22T23:57:38.336271+0000","last_peered":"2024-05-22T23:57:38.336271+0000","last_clean":"2024-05-22T23:57:38.336271+0000","last_became_active":"2024-05-22T23:57:33.318284+0000","last_became_peered":"2024-05-22T23:57:33.318284+0000","last_unstale":"2024-05-22T23:57:38.336271+0000","last_undegraded":"2024-05-22T23:57:38.336271+0000","last_fullsized":"2024-05-22T23:57:38.336271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:56.043446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957969+0000","last_change":"2024-05-22T23:57:33.321268+0000","last_active":"2024-05-22T23:57:39.957969+0000","last_peered":"2024-05-22T23:57:39.957969+0000","last_clean":"2024-05-22T23:57:39.957969+0000","last_became_active":"2024-05-22T23:57:33.320902+0000","last_became_peered":"2024-05-22T23:57:33.320902+0000","last_unstale":"2024-05-22T23:57:39.957969+0000","last_undegraded":"2024-05-22T23:57:39.957969+0000","last_fullsized":"2024-05-22T23:57:39.957969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:51:39.716541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.337466+0000","last_change":"2024-05-22T23:57:33.322638+0000","last_active":"2024-05-22T23:57:38.337466+0000","last_peered":"2024-05-22T23:57:38.337466+0000","last_clean":"2024-05-22T23:57:38.337466+0000","last_became_active":"2024-05-22T23:57:33.322318+0000","last_became_peered":"2024-05-22T23:57:33.322318+0000","last_unstale":"2024-05-22T23:57:38.337466+0000","last_undegraded":"2024-05-22T23:57:38.337466+0000","last_fullsized":"2024-05-22T23:57:38.337466+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:44:25.487618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.337446+0000","last_change":"2024-05-22T23:57:33.325832+0000","last_active":"2024-05-22T23:57:38.337446+0000","last_peered":"2024-05-22T23:57:38.337446+0000","last_clean":"2024-05-22T23:57:38.337446+0000","last_became_active":"2024-05-22T23:57:33.325062+0000","last_became_peered":"2024-05-22T23:57:33.325062+0000","last_unstale":"2024-05-22T23:57:38.337446+0000","last_undegraded":"2024-05-22T23:57:38.337446+0000","last_fullsized":"2024-05-22T23:57:38.337446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:33:04.161351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954875+0000","last_change":"2024-05-22T23:57:33.324864+0000","last_active":"2024-05-22T23:57:39.954875+0000","last_peered":"2024-05-22T23:57:39.954875+0000","last_clean":"2024-05-22T23:57:39.954875+0000","last_became_active":"2024-05-22T23:57:33.324558+0000","last_became_peered":"2024-05-22T23:57:33.324558+0000","last_unstale":"2024-05-22T23:57:39.954875+0000","last_undegraded":"2024-05-22T23:57:39.954875+0000","last_fullsized":"2024-05-22T23:57:39.954875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:48:49.722371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668272+0000","last_change":"2024-05-22T23:57:33.337985+0000","last_active":"2024-05-22T23:57:38.668272+0000","last_peered":"2024-05-22T23:57:38.668272+0000","last_clean":"2024-05-22T23:57:38.668272+0000","last_became_active":"2024-05-22T23:57:33.337719+0000","last_became_peered":"2024-05-22T23:57:33.337719+0000","last_unstale":"2024-05-22T23:57:38.668272+0000","last_undegraded":"2024-05-22T23:57:38.668272+0000","last_fullsized":"2024-05-22T23:57:38.668272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:50:44.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.960606+0000","last_change":"2024-05-22T23:57:33.325709+0000","last_active":"2024-05-22T23:57:39.960606+0000","last_peered":"2024-05-22T23:57:39.960606+0000","last_clean":"2024-05-22T23:57:39.960606+0000","last_became_active":"2024-05-22T23:57:33.325063+0000","last_became_peered":"2024-05-22T23:57:33.325063+0000","last_unstale":"2024-05-22T23:57:39.960606+0000","last_undegraded":"2024-05-22T23:57:39.960606+0000","last_fullsized":"2024-05-22T23:57:39.960606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:39.145217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336101+0000","last_change":"2024-05-22T23:57:33.333251+0000","last_active":"2024-05-22T23:57:38.336101+0000","last_peered":"2024-05-22T23:57:38.336101+0000","last_clean":"2024-05-22T23:57:38.336101+0000","last_became_active":"2024-05-22T23:57:33.332894+0000","last_became_peered":"2024-05-22T23:57:33.332894+0000","last_unstale":"2024-05-22T23:57:38.336101+0000","last_undegraded":"2024-05-22T23:57:38.336101+0000","last_fullsized":"2024-05-22T23:57:38.336101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:13:12.082781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335605+0000","last_change":"2024-05-22T23:57:37.788060+0000","last_active":"2024-05-22T23:57:38.335605+0000","last_peered":"2024-05-22T23:57:38.335605+0000","last_clean":"2024-05-22T23:57:38.335605+0000","last_became_active":"2024-05-22T23:57:37.787656+0000","last_became_peered":"2024-05-22T23:57:37.787656+0000","last_unstale":"2024-05-22T23:57:38.335605+0000","last_undegraded":"2024-05-22T23:57:38.335605+0000","last_fullsized":"2024-05-22T23:57:38.335605+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:39:13.718817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.260659+0000","last_change":"2024-05-22T23:57:32.323434+0000","last_active":"2024-05-22T23:58:04.260659+0000","last_peered":"2024-05-22T23:58:04.260659+0000","last_clean":"2024-05-22T23:58:04.260659+0000","last_became_active":"2024-05-22T23:57:32.323033+0000","last_became_peered":"2024-05-22T23:57:32.323033+0000","last_unstale":"2024-05-22T23:58:04.260659+0000","last_undegraded":"2024-05-22T23:58:04.260659+0000","last_fullsized":"2024-05-22T23:58:04.260659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:15:27.676027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878645+0000","last_change":"2024-05-22T23:57:33.323336+0000","last_active":"2024-05-22T23:57:39.878645+0000","last_peered":"2024-05-22T23:57:39.878645+0000","last_clean":"2024-05-22T23:57:39.878645+0000","last_became_active":"2024-05-22T23:57:33.323026+0000","last_became_peered":"2024-05-22T23:57:33.323026+0000","last_unstale":"2024-05-22T23:57:39.878645+0000","last_undegraded":"2024-05-22T23:57:39.878645+0000","last_fullsized":"2024-05-22T23:57:39.878645+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:02:20.142760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335797+0000","last_change":"2024-05-22T23:57:37.785137+0000","last_active":"2024-05-22T23:57:38.335797+0000","last_peered":"2024-05-22T23:57:38.335797+0000","last_clean":"2024-05-22T23:57:38.335797+0000","last_became_active":"2024-05-22T23:57:37.784791+0000","last_became_peered":"2024-05-22T23:57:37.784791+0000","last_unstale":"2024-05-22T23:57:38.335797+0000","last_undegraded":"2024-05-22T23:57:38.335797+0000","last_fullsized":"2024-05-22T23:57:38.335797+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:54:51.036334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878798+0000","last_change":"2024-05-22T23:57:32.332969+0000","last_active":"2024-05-22T23:57:39.878798+0000","last_peered":"2024-05-22T23:57:39.878798+0000","last_clean":"2024-05-22T23:57:39.878798+0000","last_became_active":"2024-05-22T23:57:32.332611+0000","last_became_peered":"2024-05-22T23:57:32.332611+0000","last_unstale":"2024-05-22T23:57:39.878798+0000","last_undegraded":"2024-05-22T23:57:39.878798+0000","last_fullsized":"2024-05-22T23:57:39.878798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:00:22.006355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.339090+0000","last_change":"2024-05-22T23:57:33.332151+0000","last_active":"2024-05-22T23:57:38.339090+0000","last_peered":"2024-05-22T23:57:38.339090+0000","last_clean":"2024-05-22T23:57:38.339090+0000","last_became_active":"2024-05-22T23:57:33.329518+0000","last_became_peered":"2024-05-22T23:57:33.329518+0000","last_unstale":"2024-05-22T23:57:38.339090+0000","last_undegraded":"2024-05-22T23:57:38.339090+0000","last_fullsized":"2024-05-22T23:57:38.339090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:10:12.083766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336083+0000","last_change":"2024-05-22T23:57:37.787524+0000","last_active":"2024-05-22T23:57:38.336083+0000","last_peered":"2024-05-22T23:57:38.336083+0000","last_clean":"2024-05-22T23:57:38.336083+0000","last_became_active":"2024-05-22T23:57:37.787103+0000","last_became_peered":"2024-05-22T23:57:37.787103+0000","last_unstale":"2024-05-22T23:57:38.336083+0000","last_undegraded":"2024-05-22T23:57:38.336083+0000","last_fullsized":"2024-05-22T23:57:38.336083+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:19:16.963159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.255300+0000","last_change":"2024-05-22T23:57:32.318469+0000","last_active":"2024-05-22T23:58:04.255300+0000","last_peered":"2024-05-22T23:58:04.255300+0000","last_clean":"2024-05-22T23:58:04.255300+0000","last_became_active":"2024-05-22T23:57:32.317640+0000","last_became_peered":"2024-05-22T23:57:32.317640+0000","last_unstale":"2024-05-22T23:58:04.255300+0000","last_undegraded":"2024-05-22T23:58:04.255300+0000","last_fullsized":"2024-05-22T23:58:04.255300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:01:55.710019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956539+0000","last_change":"2024-05-22T23:57:33.318604+0000","last_active":"2024-05-22T23:57:39.956539+0000","last_peered":"2024-05-22T23:57:39.956539+0000","last_clean":"2024-05-22T23:57:39.956539+0000","last_became_active":"2024-05-22T23:57:33.318184+0000","last_became_peered":"2024-05-22T23:57:33.318184+0000","last_unstale":"2024-05-22T23:57:39.956539+0000","last_undegraded":"2024-05-22T23:57:39.956539+0000","last_fullsized":"2024-05-22T23:57:39.956539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:16.140799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335942+0000","last_change":"2024-05-22T23:57:37.796085+0000","last_active":"2024-05-22T23:57:38.335942+0000","last_peered":"2024-05-22T23:57:38.335942+0000","last_clean":"2024-05-22T23:57:38.335942+0000","last_became_active":"2024-05-22T23:57:37.795265+0000","last_became_peered":"2024-05-22T23:57:37.795265+0000","last_unstale":"2024-05-22T23:57:38.335942+0000","last_undegraded":"2024-05-22T23:57:38.335942+0000","last_fullsized":"2024-05-22T23:57:38.335942+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:43:55.577072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504490+0000","last_change":"2024-05-22T23:57:32.316346+0000","last_active":"2024-05-22T23:57:38.504490+0000","last_peered":"2024-05-22T23:57:38.504490+0000","last_clean":"2024-05-22T23:57:38.504490+0000","last_became_active":"2024-05-22T23:57:32.316038+0000","last_became_peered":"2024-05-22T23:57:32.316038+0000","last_unstale":"2024-05-22T23:57:38.504490+0000","last_undegraded":"2024-05-22T23:57:38.504490+0000","last_fullsized":"2024-05-22T23:57:38.504490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:57:41.987250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955165+0000","last_change":"2024-05-22T23:57:33.329341+0000","last_active":"2024-05-22T23:57:39.955165+0000","last_peered":"2024-05-22T23:57:39.955165+0000","last_clean":"2024-05-22T23:57:39.955165+0000","last_became_active":"2024-05-22T23:57:33.329009+0000","last_became_peered":"2024-05-22T23:57:33.329009+0000","last_unstale":"2024-05-22T23:57:39.955165+0000","last_undegraded":"2024-05-22T23:57:39.955165+0000","last_fullsized":"2024-05-22T23:57:39.955165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:53:07.591497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336071+0000","last_change":"2024-05-22T23:57:37.779097+0000","last_active":"2024-05-22T23:57:38.336071+0000","last_peered":"2024-05-22T23:57:38.336071+0000","last_clean":"2024-05-22T23:57:38.336071+0000","last_became_active":"2024-05-22T23:57:37.778564+0000","last_became_peered":"2024-05-22T23:57:37.778564+0000","last_unstale":"2024-05-22T23:57:38.336071+0000","last_undegraded":"2024-05-22T23:57:38.336071+0000","last_fullsized":"2024-05-22T23:57:38.336071+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:09:16.351947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.243487+0000","last_change":"2024-05-22T23:57:32.316031+0000","last_active":"2024-05-22T23:58:03.243487+0000","last_peered":"2024-05-22T23:58:03.243487+0000","last_clean":"2024-05-22T23:58:03.243487+0000","last_became_active":"2024-05-22T23:57:32.315673+0000","last_became_peered":"2024-05-22T23:57:32.315673+0000","last_unstale":"2024-05-22T23:58:03.243487+0000","last_undegraded":"2024-05-22T23:58:03.243487+0000","last_fullsized":"2024-05-22T23:58:03.243487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:28.316600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504150+0000","last_change":"2024-05-22T23:57:33.324587+0000","last_active":"2024-05-22T23:57:38.504150+0000","last_peered":"2024-05-22T23:57:38.504150+0000","last_clean":"2024-05-22T23:57:38.504150+0000","last_became_active":"2024-05-22T23:57:33.323972+0000","last_became_peered":"2024-05-22T23:57:33.323972+0000","last_unstale":"2024-05-22T23:57:38.504150+0000","last_undegraded":"2024-05-22T23:57:38.504150+0000","last_fullsized":"2024-05-22T23:57:38.504150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:23:52.551699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335999+0000","last_change":"2024-05-22T23:57:37.791726+0000","last_active":"2024-05-22T23:57:38.335999+0000","last_peered":"2024-05-22T23:57:38.335999+0000","last_clean":"2024-05-22T23:57:38.335999+0000","last_became_active":"2024-05-22T23:57:37.791185+0000","last_became_peered":"2024-05-22T23:57:37.791185+0000","last_unstale":"2024-05-22T23:57:38.335999+0000","last_undegraded":"2024-05-22T23:57:38.335999+0000","last_fullsized":"2024-05-22T23:57:38.335999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:55:30.994272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.259539+0000","last_change":"2024-05-22T23:57:32.328130+0000","last_active":"2024-05-22T23:58:04.259539+0000","last_peered":"2024-05-22T23:58:04.259539+0000","last_clean":"2024-05-22T23:58:04.259539+0000","last_became_active":"2024-05-22T23:57:32.327835+0000","last_became_peered":"2024-05-22T23:57:32.327835+0000","last_unstale":"2024-05-22T23:58:04.259539+0000","last_undegraded":"2024-05-22T23:58:04.259539+0000","last_fullsized":"2024-05-22T23:58:04.259539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:32:40.442824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955119+0000","last_change":"2024-05-22T23:57:33.336525+0000","last_active":"2024-05-22T23:57:39.955119+0000","last_peered":"2024-05-22T23:57:39.955119+0000","last_clean":"2024-05-22T23:57:39.955119+0000","last_became_active":"2024-05-22T23:57:33.336245+0000","last_became_peered":"2024-05-22T23:57:33.336245+0000","last_unstale":"2024-05-22T23:57:39.955119+0000","last_undegraded":"2024-05-22T23:57:39.955119+0000","last_fullsized":"2024-05-22T23:57:39.955119+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:14.329213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668430+0000","last_change":"2024-05-22T23:57:37.787910+0000","last_active":"2024-05-22T23:57:38.668430+0000","last_peered":"2024-05-22T23:57:38.668430+0000","last_clean":"2024-05-22T23:57:38.668430+0000","last_became_active":"2024-05-22T23:57:37.787465+0000","last_became_peered":"2024-05-22T23:57:37.787465+0000","last_unstale":"2024-05-22T23:57:38.668430+0000","last_undegraded":"2024-05-22T23:57:38.668430+0000","last_fullsized":"2024-05-22T23:57:38.668430+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:54:59.202855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957447+0000","last_change":"2024-05-22T23:57:32.329365+0000","last_active":"2024-05-22T23:57:39.957447+0000","last_peered":"2024-05-22T23:57:39.957447+0000","last_clean":"2024-05-22T23:57:39.957447+0000","last_became_active":"2024-05-22T23:57:32.327864+0000","last_became_peered":"2024-05-22T23:57:32.327864+0000","last_unstale":"2024-05-22T23:57:39.957447+0000","last_undegraded":"2024-05-22T23:57:39.957447+0000","last_fullsized":"2024-05-22T23:57:39.957447+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:26.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.877820+0000","last_change":"2024-05-22T23:57:33.313930+0000","last_active":"2024-05-22T23:57:39.877820+0000","last_peered":"2024-05-22T23:57:39.877820+0000","last_clean":"2024-05-22T23:57:39.877820+0000","last_became_active":"2024-05-22T23:57:33.313612+0000","last_became_peered":"2024-05-22T23:57:33.313612+0000","last_unstale":"2024-05-22T23:57:39.877820+0000","last_undegraded":"2024-05-22T23:57:39.877820+0000","last_fullsized":"2024-05-22T23:57:39.877820+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:45:25.800338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338306+0000","last_change":"2024-05-22T23:57:37.791537+0000","last_active":"2024-05-22T23:57:38.338306+0000","last_peered":"2024-05-22T23:57:38.338306+0000","last_clean":"2024-05-22T23:57:38.338306+0000","last_became_active":"2024-05-22T23:57:37.791052+0000","last_became_peered":"2024-05-22T23:57:37.791052+0000","last_unstale":"2024-05-22T23:57:38.338306+0000","last_undegraded":"2024-05-22T23:57:38.338306+0000","last_fullsized":"2024-05-22T23:57:38.338306+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:18:11.759840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954695+0000","last_change":"2024-05-22T23:57:32.315907+0000","last_active":"2024-05-22T23:57:39.954695+0000","last_peered":"2024-05-22T23:57:39.954695+0000","last_clean":"2024-05-22T23:57:39.954695+0000","last_became_active":"2024-05-22T23:57:32.315576+0000","last_became_peered":"2024-05-22T23:57:32.315576+0000","last_unstale":"2024-05-22T23:57:39.954695+0000","last_undegraded":"2024-05-22T23:57:39.954695+0000","last_fullsized":"2024-05-22T23:57:39.954695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:45:48.998377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957104+0000","last_change":"2024-05-22T23:57:33.338681+0000","last_active":"2024-05-22T23:57:39.957104+0000","last_peered":"2024-05-22T23:57:39.957104+0000","last_clean":"2024-05-22T23:57:39.957104+0000","last_became_active":"2024-05-22T23:57:33.334705+0000","last_became_peered":"2024-05-22T23:57:33.334705+0000","last_unstale":"2024-05-22T23:57:39.957104+0000","last_undegraded":"2024-05-22T23:57:39.957104+0000","last_fullsized":"2024-05-22T23:57:39.957104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:22.135137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878483+0000","last_change":"2024-05-22T23:57:37.780470+0000","last_active":"2024-05-22T23:57:39.878483+0000","last_peered":"2024-05-22T23:57:39.878483+0000","last_clean":"2024-05-22T23:57:39.878483+0000","last_became_active":"2024-05-22T23:57:37.779110+0000","last_became_peered":"2024-05-22T23:57:37.779110+0000","last_unstale":"2024-05-22T23:57:39.878483+0000","last_undegraded":"2024-05-22T23:57:39.878483+0000","last_fullsized":"2024-05-22T23:57:39.878483+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:55:45.486862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:06.271408+0000","last_change":"2024-05-22T23:57:32.322489+0000","last_active":"2024-05-22T23:58:06.271408+0000","last_peered":"2024-05-22T23:58:06.271408+0000","last_clean":"2024-05-22T23:58:06.271408+0000","last_became_active":"2024-05-22T23:57:32.322125+0000","last_became_peered":"2024-05-22T23:57:32.322125+0000","last_unstale":"2024-05-22T23:58:06.271408+0000","last_undegraded":"2024-05-22T23:58:06.271408+0000","last_fullsized":"2024-05-22T23:58:06.271408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:11:47.707506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955706+0000","last_change":"2024-05-22T23:57:33.322417+0000","last_active":"2024-05-22T23:57:39.955706+0000","last_peered":"2024-05-22T23:57:39.955706+0000","last_clean":"2024-05-22T23:57:39.955706+0000","last_became_active":"2024-05-22T23:57:33.322102+0000","last_became_peered":"2024-05-22T23:57:33.322102+0000","last_unstale":"2024-05-22T23:57:39.955706+0000","last_undegraded":"2024-05-22T23:57:39.955706+0000","last_fullsized":"2024-05-22T23:57:39.955706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:26:43.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956585+0000","last_change":"2024-05-22T23:57:37.784424+0000","last_active":"2024-05-22T23:57:39.956585+0000","last_peered":"2024-05-22T23:57:39.956585+0000","last_clean":"2024-05-22T23:57:39.956585+0000","last_became_active":"2024-05-22T23:57:37.783949+0000","last_became_peered":"2024-05-22T23:57:37.783949+0000","last_unstale":"2024-05-22T23:57:39.956585+0000","last_undegraded":"2024-05-22T23:57:39.956585+0000","last_fullsized":"2024-05-22T23:57:39.956585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:52:23.763273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.240433+0000","last_change":"2024-05-22T23:57:32.317045+0000","last_active":"2024-05-22T23:58:03.240433+0000","last_peered":"2024-05-22T23:58:03.240433+0000","last_clean":"2024-05-22T23:58:03.240433+0000","last_became_active":"2024-05-22T23:57:32.316698+0000","last_became_peered":"2024-05-22T23:57:32.316698+0000","last_unstale":"2024-05-22T23:58:03.240433+0000","last_undegraded":"2024-05-22T23:58:03.240433+0000","last_fullsized":"2024-05-22T23:58:03.240433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:40.861593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338431+0000","last_change":"2024-05-22T23:57:33.324237+0000","last_active":"2024-05-22T23:57:38.338431+0000","last_peered":"2024-05-22T23:57:38.338431+0000","last_clean":"2024-05-22T23:57:38.338431+0000","last_became_active":"2024-05-22T23:57:33.323870+0000","last_became_peered":"2024-05-22T23:57:33.323870+0000","last_unstale":"2024-05-22T23:57:38.338431+0000","last_undegraded":"2024-05-22T23:57:38.338431+0000","last_fullsized":"2024-05-22T23:57:38.338431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:41:29.328104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336375+0000","last_change":"2024-05-22T23:57:37.787795+0000","last_active":"2024-05-22T23:57:38.336375+0000","last_peered":"2024-05-22T23:57:38.336375+0000","last_clean":"2024-05-22T23:57:38.336375+0000","last_became_active":"2024-05-22T23:57:37.787210+0000","last_became_peered":"2024-05-22T23:57:37.787210+0000","last_unstale":"2024-05-22T23:57:38.336375+0000","last_undegraded":"2024-05-22T23:57:38.336375+0000","last_fullsized":"2024-05-22T23:57:38.336375+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:19:04.616758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955176+0000","last_change":"2024-05-22T23:57:32.321415+0000","last_active":"2024-05-22T23:57:39.955176+0000","last_peered":"2024-05-22T23:57:39.955176+0000","last_clean":"2024-05-22T23:57:39.955176+0000","last_became_active":"2024-05-22T23:57:32.321055+0000","last_became_peered":"2024-05-22T23:57:32.321055+0000","last_unstale":"2024-05-22T23:57:39.955176+0000","last_undegraded":"2024-05-22T23:57:39.955176+0000","last_fullsized":"2024-05-22T23:57:39.955176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:28:58.577548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956417+0000","last_change":"2024-05-22T23:57:33.317043+0000","last_active":"2024-05-22T23:57:39.956417+0000","last_peered":"2024-05-22T23:57:39.956417+0000","last_clean":"2024-05-22T23:57:39.956417+0000","last_became_active":"2024-05-22T23:57:33.314687+0000","last_became_peered":"2024-05-22T23:57:33.314687+0000","last_unstale":"2024-05-22T23:57:39.956417+0000","last_undegraded":"2024-05-22T23:57:39.956417+0000","last_fullsized":"2024-05-22T23:57:39.956417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:18:02.718481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.960634+0000","last_change":"2024-05-22T23:57:37.781367+0000","last_active":"2024-05-22T23:57:39.960634+0000","last_peered":"2024-05-22T23:57:39.960634+0000","last_clean":"2024-05-22T23:57:39.960634+0000","last_became_active":"2024-05-22T23:57:37.780955+0000","last_became_peered":"2024-05-22T23:57:37.780955+0000","last_unstale":"2024-05-22T23:57:39.960634+0000","last_undegraded":"2024-05-22T23:57:39.960634+0000","last_fullsized":"2024-05-22T23:57:39.960634+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:36:18.962940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'25","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:07.276226+0000","last_change":"2024-05-22T23:57:32.313873+0000","last_active":"2024-05-22T23:58:07.276226+0000","last_peered":"2024-05-22T23:58:07.276226+0000","last_clean":"2024-05-22T23:58:07.276226+0000","last_became_active":"2024-05-22T23:57:32.313388+0000","last_became_peered":"2024-05-22T23:57:32.313388+0000","last_unstale":"2024-05-22T23:58:07.276226+0000","last_undegraded":"2024-05-22T23:58:07.276226+0000","last_fullsized":"2024-05-22T23:58:07.276226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:16.383384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40469,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":44,"num_write":26,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504454+0000","last_change":"2024-05-22T23:57:33.318751+0000","last_active":"2024-05-22T23:57:38.504454+0000","last_peered":"2024-05-22T23:57:38.504454+0000","last_clean":"2024-05-22T23:57:38.504454+0000","last_became_active":"2024-05-22T23:57:33.317710+0000","last_became_peered":"2024-05-22T23:57:33.317710+0000","last_unstale":"2024-05-22T23:57:38.504454+0000","last_undegraded":"2024-05-22T23:57:38.504454+0000","last_fullsized":"2024-05-22T23:57:38.504454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:57:50.852615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955499+0000","last_change":"2024-05-22T23:57:37.790073+0000","last_active":"2024-05-22T23:57:39.955499+0000","last_peered":"2024-05-22T23:57:39.955499+0000","last_clean":"2024-05-22T23:57:39.955499+0000","last_became_active":"2024-05-22T23:57:37.789477+0000","last_became_peered":"2024-05-22T23:57:37.789477+0000","last_unstale":"2024-05-22T23:57:39.955499+0000","last_undegraded":"2024-05-22T23:57:39.955499+0000","last_fullsized":"2024-05-22T23:57:39.955499+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:11.003201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.259989+0000","last_change":"2024-05-22T23:57:32.329846+0000","last_active":"2024-05-22T23:58:04.259989+0000","last_peered":"2024-05-22T23:58:04.259989+0000","last_clean":"2024-05-22T23:58:04.259989+0000","last_became_active":"2024-05-22T23:57:32.329414+0000","last_became_peered":"2024-05-22T23:57:32.329414+0000","last_unstale":"2024-05-22T23:58:04.259989+0000","last_undegraded":"2024-05-22T23:58:04.259989+0000","last_fullsized":"2024-05-22T23:58:04.259989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:15:38.176790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958847+0000","last_change":"2024-05-22T23:57:33.333298+0000","last_active":"2024-05-22T23:57:39.958847+0000","last_peered":"2024-05-22T23:57:39.958847+0000","last_clean":"2024-05-22T23:57:39.958847+0000","last_became_active":"2024-05-22T23:57:33.332934+0000","last_became_peered":"2024-05-22T23:57:33.332934+0000","last_unstale":"2024-05-22T23:57:39.958847+0000","last_undegraded":"2024-05-22T23:57:39.958847+0000","last_fullsized":"2024-05-22T23:57:39.958847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:06:00.427627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668248+0000","last_change":"2024-05-22T23:57:37.795628+0000","last_active":"2024-05-22T23:57:38.668248+0000","last_peered":"2024-05-22T23:57:38.668248+0000","last_clean":"2024-05-22T23:57:38.668248+0000","last_became_active":"2024-05-22T23:57:37.795208+0000","last_became_peered":"2024-05-22T23:57:37.795208+0000","last_unstale":"2024-05-22T23:57:38.668248+0000","last_undegraded":"2024-05-22T23:57:38.668248+0000","last_fullsized":"2024-05-22T23:57:38.668248+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:03:12.684474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.245200+0000","last_change":"2024-05-22T23:57:32.334039+0000","last_active":"2024-05-22T23:58:03.245200+0000","last_peered":"2024-05-22T23:58:03.245200+0000","last_clean":"2024-05-22T23:58:03.245200+0000","last_became_active":"2024-05-22T23:57:32.333709+0000","last_became_peered":"2024-05-22T23:57:32.333709+0000","last_unstale":"2024-05-22T23:58:03.245200+0000","last_undegraded":"2024-05-22T23:58:03.245200+0000","last_fullsized":"2024-05-22T23:58:03.245200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:41:43.319094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956045+0000","last_change":"2024-05-22T23:57:33.328056+0000","last_active":"2024-05-22T23:57:39.956045+0000","last_peered":"2024-05-22T23:57:39.956045+0000","last_clean":"2024-05-22T23:57:39.956045+0000","last_became_active":"2024-05-22T23:57:33.326559+0000","last_became_peered":"2024-05-22T23:57:33.326559+0000","last_unstale":"2024-05-22T23:57:39.956045+0000","last_undegraded":"2024-05-22T23:57:39.956045+0000","last_fullsized":"2024-05-22T23:57:39.956045+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:06.463837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.960354+0000","last_change":"2024-05-22T23:57:37.795892+0000","last_active":"2024-05-22T23:57:39.960354+0000","last_peered":"2024-05-22T23:57:39.960354+0000","last_clean":"2024-05-22T23:57:39.960354+0000","last_became_active":"2024-05-22T23:57:37.792246+0000","last_became_peered":"2024-05-22T23:57:37.792246+0000","last_unstale":"2024-05-22T23:57:39.960354+0000","last_undegraded":"2024-05-22T23:57:39.960354+0000","last_fullsized":"2024-05-22T23:57:39.960354+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:18.462267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.253582+0000","last_change":"2024-05-22T23:57:32.319379+0000","last_active":"2024-05-22T23:58:04.253582+0000","last_peered":"2024-05-22T23:58:04.253582+0000","last_clean":"2024-05-22T23:58:04.253582+0000","last_became_active":"2024-05-22T23:57:32.318803+0000","last_became_peered":"2024-05-22T23:57:32.318803+0000","last_unstale":"2024-05-22T23:58:04.253582+0000","last_undegraded":"2024-05-22T23:58:04.253582+0000","last_fullsized":"2024-05-22T23:58:04.253582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:59:15.689119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955097+0000","last_change":"2024-05-22T23:57:33.325294+0000","last_active":"2024-05-22T23:57:39.955097+0000","last_peered":"2024-05-22T23:57:39.955097+0000","last_clean":"2024-05-22T23:57:39.955097+0000","last_became_active":"2024-05-22T23:57:33.324339+0000","last_became_peered":"2024-05-22T23:57:33.324339+0000","last_unstale":"2024-05-22T23:57:39.955097+0000","last_undegraded":"2024-05-22T23:57:39.955097+0000","last_fullsized":"2024-05-22T23:57:39.955097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:29:47.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.959996+0000","last_change":"2024-05-22T23:57:37.797961+0000","last_active":"2024-05-22T23:57:39.959996+0000","last_peered":"2024-05-22T23:57:39.959996+0000","last_clean":"2024-05-22T23:57:39.959996+0000","last_became_active":"2024-05-22T23:57:37.797532+0000","last_became_peered":"2024-05-22T23:57:37.797532+0000","last_unstale":"2024-05-22T23:57:39.959996+0000","last_undegraded":"2024-05-22T23:57:39.959996+0000","last_fullsized":"2024-05-22T23:57:39.959996+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:51:12.997674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.258018+0000","last_change":"2024-05-22T23:57:32.317434+0000","last_active":"2024-05-22T23:58:04.258018+0000","last_peered":"2024-05-22T23:58:04.258018+0000","last_clean":"2024-05-22T23:58:04.258018+0000","last_became_active":"2024-05-22T23:57:32.316987+0000","last_became_peered":"2024-05-22T23:57:32.316987+0000","last_unstale":"2024-05-22T23:58:04.258018+0000","last_undegraded":"2024-05-22T23:58:04.258018+0000","last_fullsized":"2024-05-22T23:58:04.258018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:46:50.306756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954821+0000","last_change":"2024-05-22T23:57:33.328670+0000","last_active":"2024-05-22T23:57:39.954821+0000","last_peered":"2024-05-22T23:57:39.954821+0000","last_clean":"2024-05-22T23:57:39.954821+0000","last_became_active":"2024-05-22T23:57:33.328294+0000","last_became_peered":"2024-05-22T23:57:33.328294+0000","last_unstale":"2024-05-22T23:57:39.954821+0000","last_undegraded":"2024-05-22T23:57:39.954821+0000","last_fullsized":"2024-05-22T23:57:39.954821+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:09:10.807844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954816+0000","last_change":"2024-05-22T23:57:37.788885+0000","last_active":"2024-05-22T23:57:39.954816+0000","last_peered":"2024-05-22T23:57:39.954816+0000","last_clean":"2024-05-22T23:57:39.954816+0000","last_became_active":"2024-05-22T23:57:37.788145+0000","last_became_peered":"2024-05-22T23:57:37.788145+0000","last_unstale":"2024-05-22T23:57:39.954816+0000","last_undegraded":"2024-05-22T23:57:39.954816+0000","last_fullsized":"2024-05-22T23:57:39.954816+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:14:50.027785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:05.268296+0000","last_change":"2024-05-22T23:57:32.317855+0000","last_active":"2024-05-22T23:58:05.268296+0000","last_peered":"2024-05-22T23:58:05.268296+0000","last_clean":"2024-05-22T23:58:05.268296+0000","last_became_active":"2024-05-22T23:57:32.317087+0000","last_became_peered":"2024-05-22T23:57:32.317087+0000","last_unstale":"2024-05-22T23:58:05.268296+0000","last_undegraded":"2024-05-22T23:58:05.268296+0000","last_fullsized":"2024-05-22T23:58:05.268296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:49:40.486222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668239+0000","last_change":"2024-05-22T23:57:33.323897+0000","last_active":"2024-05-22T23:57:38.668239+0000","last_peered":"2024-05-22T23:57:38.668239+0000","last_clean":"2024-05-22T23:57:38.668239+0000","last_became_active":"2024-05-22T23:57:33.323567+0000","last_became_peered":"2024-05-22T23:57:33.323567+0000","last_unstale":"2024-05-22T23:57:38.668239+0000","last_undegraded":"2024-05-22T23:57:38.668239+0000","last_fullsized":"2024-05-22T23:57:38.668239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:17:48.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504206+0000","last_change":"2024-05-22T23:57:38.218313+0000","last_active":"2024-05-22T23:57:38.504206+0000","last_peered":"2024-05-22T23:57:38.504206+0000","last_clean":"2024-05-22T23:57:38.504206+0000","last_became_active":"2024-05-22T23:57:38.217756+0000","last_became_peered":"2024-05-22T23:57:38.217756+0000","last_unstale":"2024-05-22T23:57:38.504206+0000","last_undegraded":"2024-05-22T23:57:38.504206+0000","last_fullsized":"2024-05-22T23:57:38.504206+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:07:55.122260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958715+0000","last_change":"2024-05-22T23:57:32.325729+0000","last_active":"2024-05-22T23:57:39.958715+0000","last_peered":"2024-05-22T23:57:39.958715+0000","last_clean":"2024-05-22T23:57:39.958715+0000","last_became_active":"2024-05-22T23:57:32.325383+0000","last_became_peered":"2024-05-22T23:57:32.325383+0000","last_unstale":"2024-05-22T23:57:39.958715+0000","last_undegraded":"2024-05-22T23:57:39.958715+0000","last_fullsized":"2024-05-22T23:57:39.958715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:57.978806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956927+0000","last_change":"2024-05-22T23:57:33.337499+0000","last_active":"2024-05-22T23:57:39.956927+0000","last_peered":"2024-05-22T23:57:39.956927+0000","last_clean":"2024-05-22T23:57:39.956927+0000","last_became_active":"2024-05-22T23:57:33.337271+0000","last_became_peered":"2024-05-22T23:57:33.337271+0000","last_unstale":"2024-05-22T23:57:39.956927+0000","last_undegraded":"2024-05-22T23:57:39.956927+0000","last_fullsized":"2024-05-22T23:57:39.956927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:32:39.133984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958026+0000","last_change":"2024-05-22T23:57:37.792707+0000","last_active":"2024-05-22T23:57:39.958026+0000","last_peered":"2024-05-22T23:57:39.958026+0000","last_clean":"2024-05-22T23:57:39.958026+0000","last_became_active":"2024-05-22T23:57:37.792209+0000","last_became_peered":"2024-05-22T23:57:37.792209+0000","last_unstale":"2024-05-22T23:57:39.958026+0000","last_undegraded":"2024-05-22T23:57:39.958026+0000","last_fullsized":"2024-05-22T23:57:39.958026+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:36:34.089424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955194+0000","last_change":"2024-05-22T23:57:32.330400+0000","last_active":"2024-05-22T23:57:39.955194+0000","last_peered":"2024-05-22T23:57:39.955194+0000","last_clean":"2024-05-22T23:57:39.955194+0000","last_became_active":"2024-05-22T23:57:32.330051+0000","last_became_peered":"2024-05-22T23:57:32.330051+0000","last_unstale":"2024-05-22T23:57:39.955194+0000","last_undegraded":"2024-05-22T23:57:39.955194+0000","last_fullsized":"2024-05-22T23:57:39.955194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:26:57.966530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956191+0000","last_change":"2024-05-22T23:57:33.328129+0000","last_active":"2024-05-22T23:57:39.956191+0000","last_peered":"2024-05-22T23:57:39.956191+0000","last_clean":"2024-05-22T23:57:39.956191+0000","last_became_active":"2024-05-22T23:57:33.327732+0000","last_became_peered":"2024-05-22T23:57:33.327732+0000","last_unstale":"2024-05-22T23:57:39.956191+0000","last_undegraded":"2024-05-22T23:57:39.956191+0000","last_fullsized":"2024-05-22T23:57:39.956191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:25:14.105235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.959156+0000","last_change":"2024-05-22T23:57:38.220884+0000","last_active":"2024-05-22T23:57:39.959156+0000","last_peered":"2024-05-22T23:57:39.959156+0000","last_clean":"2024-05-22T23:57:39.959156+0000","last_became_active":"2024-05-22T23:57:38.219365+0000","last_became_peered":"2024-05-22T23:57:38.219365+0000","last_unstale":"2024-05-22T23:57:39.959156+0000","last_undegraded":"2024-05-22T23:57:39.959156+0000","last_fullsized":"2024-05-22T23:57:39.959156+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:54:56.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954873+0000","last_change":"2024-05-22T23:57:33.328272+0000","last_active":"2024-05-22T23:57:39.954873+0000","last_peered":"2024-05-22T23:57:39.954873+0000","last_clean":"2024-05-22T23:57:39.954873+0000","last_became_active":"2024-05-22T23:57:33.327915+0000","last_became_peered":"2024-05-22T23:57:33.327915+0000","last_unstale":"2024-05-22T23:57:39.954873+0000","last_undegraded":"2024-05-22T23:57:39.954873+0000","last_fullsized":"2024-05-22T23:57:39.954873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:59:09.085649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957387+0000","last_change":"2024-05-22T23:57:32.331979+0000","last_active":"2024-05-22T23:57:39.957387+0000","last_peered":"2024-05-22T23:57:39.957387+0000","last_clean":"2024-05-22T23:57:39.957387+0000","last_became_active":"2024-05-22T23:57:32.330983+0000","last_became_peered":"2024-05-22T23:57:32.330983+0000","last_unstale":"2024-05-22T23:57:39.957387+0000","last_undegraded":"2024-05-22T23:57:39.957387+0000","last_fullsized":"2024-05-22T23:57:39.957387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:07:19.137652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956118+0000","last_change":"2024-05-22T23:57:37.787935+0000","last_active":"2024-05-22T23:57:39.956118+0000","last_peered":"2024-05-22T23:57:39.956118+0000","last_clean":"2024-05-22T23:57:39.956118+0000","last_became_active":"2024-05-22T23:57:37.787495+0000","last_became_peered":"2024-05-22T23:57:37.787495+0000","last_unstale":"2024-05-22T23:57:39.956118+0000","last_undegraded":"2024-05-22T23:57:39.956118+0000","last_fullsized":"2024-05-22T23:57:39.956118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:37.682711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878181+0000","last_change":"2024-05-22T23:57:33.329451+0000","last_active":"2024-05-22T23:57:39.878181+0000","last_peered":"2024-05-22T23:57:39.878181+0000","last_clean":"2024-05-22T23:57:39.878181+0000","last_became_active":"2024-05-22T23:57:33.329107+0000","last_became_peered":"2024-05-22T23:57:33.329107+0000","last_unstale":"2024-05-22T23:57:39.878181+0000","last_undegraded":"2024-05-22T23:57:39.878181+0000","last_fullsized":"2024-05-22T23:57:39.878181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:44:50.198040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.247682+0000","last_change":"2024-05-22T23:57:32.321873+0000","last_active":"2024-05-22T23:58:03.247682+0000","last_peered":"2024-05-22T23:58:03.247682+0000","last_clean":"2024-05-22T23:58:03.247682+0000","last_became_active":"2024-05-22T23:57:32.321550+0000","last_became_peered":"2024-05-22T23:57:32.321550+0000","last_unstale":"2024-05-22T23:58:03.247682+0000","last_undegraded":"2024-05-22T23:58:03.247682+0000","last_fullsized":"2024-05-22T23:58:03.247682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:30:17.479238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954789+0000","last_change":"2024-05-22T23:56:11.327788+0000","last_active":"2024-05-22T23:57:39.954789+0000","last_peered":"2024-05-22T23:57:39.954789+0000","last_clean":"2024-05-22T23:57:39.954789+0000","last_became_active":"2024-05-22T23:56:11.327232+0000","last_became_peered":"2024-05-22T23:56:11.327232+0000","last_unstale":"2024-05-22T23:57:39.954789+0000","last_undegraded":"2024-05-22T23:57:39.954789+0000","last_fullsized":"2024-05-22T23:57:39.954789+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_clean_scrub_stamp":"2024-05-22T23:52:16.719643+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:26:13.077470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955505+0000","last_change":"2024-05-22T23:57:38.219326+0000","last_active":"2024-05-22T23:57:39.955505+0000","last_peered":"2024-05-22T23:57:39.955505+0000","last_clean":"2024-05-22T23:57:39.955505+0000","last_became_active":"2024-05-22T23:57:38.218835+0000","last_became_peered":"2024-05-22T23:57:38.218835+0000","last_unstale":"2024-05-22T23:57:39.955505+0000","last_undegraded":"2024-05-22T23:57:39.955505+0000","last_fullsized":"2024-05-22T23:57:39.955505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:29.722862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957209+0000","last_change":"2024-05-22T23:57:33.316760+0000","last_active":"2024-05-22T23:57:39.957209+0000","last_peered":"2024-05-22T23:57:39.957209+0000","last_clean":"2024-05-22T23:57:39.957209+0000","last_became_active":"2024-05-22T23:57:33.316328+0000","last_became_peered":"2024-05-22T23:57:33.316328+0000","last_unstale":"2024-05-22T23:57:39.957209+0000","last_undegraded":"2024-05-22T23:57:39.957209+0000","last_fullsized":"2024-05-22T23:57:39.957209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:53:11.592134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.254957+0000","last_change":"2024-05-22T23:57:32.319154+0000","last_active":"2024-05-22T23:58:04.254957+0000","last_peered":"2024-05-22T23:58:04.254957+0000","last_clean":"2024-05-22T23:58:04.254957+0000","last_became_active":"2024-05-22T23:57:32.318808+0000","last_became_peered":"2024-05-22T23:57:32.318808+0000","last_unstale":"2024-05-22T23:58:04.254957+0000","last_undegraded":"2024-05-22T23:58:04.254957+0000","last_fullsized":"2024-05-22T23:58:04.254957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:37:47.271767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668241+0000","last_change":"2024-05-22T23:57:37.791464+0000","last_active":"2024-05-22T23:57:38.668241+0000","last_peered":"2024-05-22T23:57:38.668241+0000","last_clean":"2024-05-22T23:57:38.668241+0000","last_became_active":"2024-05-22T23:57:37.791044+0000","last_became_peered":"2024-05-22T23:57:37.791044+0000","last_unstale":"2024-05-22T23:57:38.668241+0000","last_undegraded":"2024-05-22T23:57:38.668241+0000","last_fullsized":"2024-05-22T23:57:38.668241+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:47:00.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.246405+0000","last_change":"2024-05-22T23:57:32.322740+0000","last_active":"2024-05-22T23:58:03.246405+0000","last_peered":"2024-05-22T23:58:03.246405+0000","last_clean":"2024-05-22T23:58:03.246405+0000","last_became_active":"2024-05-22T23:57:32.322425+0000","last_became_peered":"2024-05-22T23:57:32.322425+0000","last_unstale":"2024-05-22T23:58:03.246405+0000","last_undegraded":"2024-05-22T23:58:03.246405+0000","last_fullsized":"2024-05-22T23:58:03.246405+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:05:10.522462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958552+0000","last_change":"2024-05-22T23:57:33.324667+0000","last_active":"2024-05-22T23:57:39.958552+0000","last_peered":"2024-05-22T23:57:39.958552+0000","last_clean":"2024-05-22T23:57:39.958552+0000","last_became_active":"2024-05-22T23:57:33.324251+0000","last_became_peered":"2024-05-22T23:57:33.324251+0000","last_unstale":"2024-05-22T23:57:39.958552+0000","last_undegraded":"2024-05-22T23:57:39.958552+0000","last_fullsized":"2024-05-22T23:57:39.958552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:49:57.203996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.877970+0000","last_change":"2024-05-22T23:57:38.222346+0000","last_active":"2024-05-22T23:57:39.877970+0000","last_peered":"2024-05-22T23:57:39.877970+0000","last_clean":"2024-05-22T23:57:39.877970+0000","last_became_active":"2024-05-22T23:57:38.220248+0000","last_became_peered":"2024-05-22T23:57:38.220248+0000","last_unstale":"2024-05-22T23:57:39.877970+0000","last_undegraded":"2024-05-22T23:57:39.877970+0000","last_fullsized":"2024-05-22T23:57:39.877970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:26:44.026641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.260409+0000","last_change":"2024-05-22T23:57:32.336308+0000","last_active":"2024-05-22T23:58:04.260409+0000","last_peered":"2024-05-22T23:58:04.260409+0000","last_clean":"2024-05-22T23:58:04.260409+0000","last_became_active":"2024-05-22T23:57:32.334872+0000","last_became_peered":"2024-05-22T23:57:32.334872+0000","last_unstale":"2024-05-22T23:58:04.260409+0000","last_undegraded":"2024-05-22T23:58:04.260409+0000","last_fullsized":"2024-05-22T23:58:04.260409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:44:16.560930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878851+0000","last_change":"2024-05-22T23:57:33.315266+0000","last_active":"2024-05-22T23:57:39.878851+0000","last_peered":"2024-05-22T23:57:39.878851+0000","last_clean":"2024-05-22T23:57:39.878851+0000","last_became_active":"2024-05-22T23:57:33.314212+0000","last_became_peered":"2024-05-22T23:57:33.314212+0000","last_unstale":"2024-05-22T23:57:39.878851+0000","last_undegraded":"2024-05-22T23:57:39.878851+0000","last_fullsized":"2024-05-22T23:57:39.878851+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:22.896162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335426+0000","last_change":"2024-05-22T23:57:37.778524+0000","last_active":"2024-05-22T23:57:38.335426+0000","last_peered":"2024-05-22T23:57:38.335426+0000","last_clean":"2024-05-22T23:57:38.335426+0000","last_became_active":"2024-05-22T23:57:37.777827+0000","last_became_peered":"2024-05-22T23:57:37.777827+0000","last_unstale":"2024-05-22T23:57:38.335426+0000","last_undegraded":"2024-05-22T23:57:38.335426+0000","last_fullsized":"2024-05-22T23:57:38.335426+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:30:59.154921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.243324+0000","last_change":"2024-05-22T23:57:32.317649+0000","last_active":"2024-05-22T23:58:03.243324+0000","last_peered":"2024-05-22T23:58:03.243324+0000","last_clean":"2024-05-22T23:58:03.243324+0000","last_became_active":"2024-05-22T23:57:32.317280+0000","last_became_peered":"2024-05-22T23:57:32.317280+0000","last_unstale":"2024-05-22T23:58:03.243324+0000","last_undegraded":"2024-05-22T23:58:03.243324+0000","last_fullsized":"2024-05-22T23:58:03.243324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:01:16.917079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338452+0000","last_change":"2024-05-22T23:57:33.339296+0000","last_active":"2024-05-22T23:57:38.338452+0000","last_peered":"2024-05-22T23:57:38.338452+0000","last_clean":"2024-05-22T23:57:38.338452+0000","last_became_active":"2024-05-22T23:57:33.338457+0000","last_became_peered":"2024-05-22T23:57:33.338457+0000","last_unstale":"2024-05-22T23:57:38.338452+0000","last_undegraded":"2024-05-22T23:57:38.338452+0000","last_fullsized":"2024-05-22T23:57:38.338452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:13:05.927893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336383+0000","last_change":"2024-05-22T23:57:38.218649+0000","last_active":"2024-05-22T23:57:38.336383+0000","last_peered":"2024-05-22T23:57:38.336383+0000","last_clean":"2024-05-22T23:57:38.336383+0000","last_became_active":"2024-05-22T23:57:38.218189+0000","last_became_peered":"2024-05-22T23:57:38.218189+0000","last_unstale":"2024-05-22T23:57:38.336383+0000","last_undegraded":"2024-05-22T23:57:38.336383+0000","last_fullsized":"2024-05-22T23:57:38.336383+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:30.747293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.246280+0000","last_change":"2024-05-22T23:57:32.309379+0000","last_active":"2024-05-22T23:58:03.246280+0000","last_peered":"2024-05-22T23:58:03.246280+0000","last_clean":"2024-05-22T23:58:03.246280+0000","last_became_active":"2024-05-22T23:57:32.309068+0000","last_became_peered":"2024-05-22T23:57:32.309068+0000","last_unstale":"2024-05-22T23:58:03.246280+0000","last_undegraded":"2024-05-22T23:58:03.246280+0000","last_fullsized":"2024-05-22T23:58:03.246280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:43:54.787996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.503994+0000","last_change":"2024-05-22T23:57:33.320253+0000","last_active":"2024-05-22T23:57:38.503994+0000","last_peered":"2024-05-22T23:57:38.503994+0000","last_clean":"2024-05-22T23:57:38.503994+0000","last_became_active":"2024-05-22T23:57:33.319604+0000","last_became_peered":"2024-05-22T23:57:33.319604+0000","last_unstale":"2024-05-22T23:57:38.503994+0000","last_undegraded":"2024-05-22T23:57:38.503994+0000","last_fullsized":"2024-05-22T23:57:38.503994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:23:51.745040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.960879+0000","last_change":"2024-05-22T23:57:37.795368+0000","last_active":"2024-05-22T23:57:39.960879+0000","last_peered":"2024-05-22T23:57:39.960879+0000","last_clean":"2024-05-22T23:57:39.960879+0000","last_became_active":"2024-05-22T23:57:37.792848+0000","last_became_peered":"2024-05-22T23:57:37.792848+0000","last_unstale":"2024-05-22T23:57:39.960879+0000","last_undegraded":"2024-05-22T23:57:39.960879+0000","last_fullsized":"2024-05-22T23:57:39.960879+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:45.859867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.247516+0000","last_change":"2024-05-22T23:57:32.329147+0000","last_active":"2024-05-22T23:58:03.247516+0000","last_peered":"2024-05-22T23:58:03.247516+0000","last_clean":"2024-05-22T23:58:03.247516+0000","last_became_active":"2024-05-22T23:57:32.327254+0000","last_became_peered":"2024-05-22T23:57:32.327254+0000","last_unstale":"2024-05-22T23:58:03.247516+0000","last_undegraded":"2024-05-22T23:58:03.247516+0000","last_fullsized":"2024-05-22T23:58:03.247516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:55:24.915817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.667879+0000","last_change":"2024-05-22T23:57:33.332843+0000","last_active":"2024-05-22T23:57:38.667879+0000","last_peered":"2024-05-22T23:57:38.667879+0000","last_clean":"2024-05-22T23:57:38.667879+0000","last_became_active":"2024-05-22T23:57:33.332497+0000","last_became_peered":"2024-05-22T23:57:33.332497+0000","last_unstale":"2024-05-22T23:57:38.667879+0000","last_undegraded":"2024-05-22T23:57:38.667879+0000","last_fullsized":"2024-05-22T23:57:38.667879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:55:16.109676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336600+0000","last_change":"2024-05-22T23:57:37.787609+0000","last_active":"2024-05-22T23:57:38.336600+0000","last_peered":"2024-05-22T23:57:38.336600+0000","last_clean":"2024-05-22T23:57:38.336600+0000","last_became_active":"2024-05-22T23:57:37.787153+0000","last_became_peered":"2024-05-22T23:57:37.787153+0000","last_unstale":"2024-05-22T23:57:38.336600+0000","last_undegraded":"2024-05-22T23:57:38.336600+0000","last_fullsized":"2024-05-22T23:57:38.336600+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:57:51.640037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335396+0000","last_change":"2024-05-22T23:57:32.329784+0000","last_active":"2024-05-22T23:57:38.335396+0000","last_peered":"2024-05-22T23:57:38.335396+0000","last_clean":"2024-05-22T23:57:38.335396+0000","last_became_active":"2024-05-22T23:57:32.329393+0000","last_became_peered":"2024-05-22T23:57:32.329393+0000","last_unstale":"2024-05-22T23:57:38.335396+0000","last_undegraded":"2024-05-22T23:57:38.335396+0000","last_fullsized":"2024-05-22T23:57:38.335396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:17:54.811962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878789+0000","last_change":"2024-05-22T23:57:33.318803+0000","last_active":"2024-05-22T23:57:39.878789+0000","last_peered":"2024-05-22T23:57:39.878789+0000","last_clean":"2024-05-22T23:57:39.878789+0000","last_became_active":"2024-05-22T23:57:33.318170+0000","last_became_peered":"2024-05-22T23:57:33.318170+0000","last_unstale":"2024-05-22T23:57:39.878789+0000","last_undegraded":"2024-05-22T23:57:39.878789+0000","last_fullsized":"2024-05-22T23:57:39.878789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:56:56.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336577+0000","last_change":"2024-05-22T23:57:38.218583+0000","last_active":"2024-05-22T23:57:38.336577+0000","last_peered":"2024-05-22T23:57:38.336577+0000","last_clean":"2024-05-22T23:57:38.336577+0000","last_became_active":"2024-05-22T23:57:38.218048+0000","last_became_peered":"2024-05-22T23:57:38.218048+0000","last_unstale":"2024-05-22T23:57:38.336577+0000","last_undegraded":"2024-05-22T23:57:38.336577+0000","last_fullsized":"2024-05-22T23:57:38.336577+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:10:57.579656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.242063+0000","last_change":"2024-05-22T23:57:32.317908+0000","last_active":"2024-05-22T23:58:03.242063+0000","last_peered":"2024-05-22T23:58:03.242063+0000","last_clean":"2024-05-22T23:58:03.242063+0000","last_became_active":"2024-05-22T23:57:32.317580+0000","last_became_peered":"2024-05-22T23:57:32.317580+0000","last_unstale":"2024-05-22T23:58:03.242063+0000","last_undegraded":"2024-05-22T23:58:03.242063+0000","last_fullsized":"2024-05-22T23:58:03.242063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:50:02.044817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955261+0000","last_change":"2024-05-22T23:57:33.321184+0000","last_active":"2024-05-22T23:57:39.955261+0000","last_peered":"2024-05-22T23:57:39.955261+0000","last_clean":"2024-05-22T23:57:39.955261+0000","last_became_active":"2024-05-22T23:57:33.320418+0000","last_became_peered":"2024-05-22T23:57:33.320418+0000","last_unstale":"2024-05-22T23:57:39.955261+0000","last_undegraded":"2024-05-22T23:57:39.955261+0000","last_fullsized":"2024-05-22T23:57:39.955261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:22:43.019809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954795+0000","last_change":"2024-05-22T23:57:37.795718+0000","last_active":"2024-05-22T23:57:39.954795+0000","last_peered":"2024-05-22T23:57:39.954795+0000","last_clean":"2024-05-22T23:57:39.954795+0000","last_became_active":"2024-05-22T23:57:37.795297+0000","last_became_peered":"2024-05-22T23:57:37.795297+0000","last_unstale":"2024-05-22T23:57:39.954795+0000","last_undegraded":"2024-05-22T23:57:39.954795+0000","last_fullsized":"2024-05-22T23:57:39.954795+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:56:47.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.257734+0000","last_change":"2024-05-22T23:57:32.311921+0000","last_active":"2024-05-22T23:58:04.257734+0000","last_peered":"2024-05-22T23:58:04.257734+0000","last_clean":"2024-05-22T23:58:04.257734+0000","last_became_active":"2024-05-22T23:57:32.311600+0000","last_became_peered":"2024-05-22T23:57:32.311600+0000","last_unstale":"2024-05-22T23:58:04.257734+0000","last_undegraded":"2024-05-22T23:58:04.257734+0000","last_fullsized":"2024-05-22T23:58:04.257734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:35:21.882773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956456+0000","last_change":"2024-05-22T23:57:33.321289+0000","last_active":"2024-05-22T23:57:39.956456+0000","last_peered":"2024-05-22T23:57:39.956456+0000","last_clean":"2024-05-22T23:57:39.956456+0000","last_became_active":"2024-05-22T23:57:33.320458+0000","last_became_peered":"2024-05-22T23:57:33.320458+0000","last_unstale":"2024-05-22T23:57:39.956456+0000","last_undegraded":"2024-05-22T23:57:39.956456+0000","last_fullsized":"2024-05-22T23:57:39.956456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:02:05.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338005+0000","last_change":"2024-05-22T23:57:37.786062+0000","last_active":"2024-05-22T23:57:38.338005+0000","last_peered":"2024-05-22T23:57:38.338005+0000","last_clean":"2024-05-22T23:57:38.338005+0000","last_became_active":"2024-05-22T23:57:37.785624+0000","last_became_peered":"2024-05-22T23:57:37.785624+0000","last_unstale":"2024-05-22T23:57:38.338005+0000","last_undegraded":"2024-05-22T23:57:38.338005+0000","last_fullsized":"2024-05-22T23:57:38.338005+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:38:56.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955273+0000","last_change":"2024-05-22T23:57:32.333909+0000","last_active":"2024-05-22T23:57:39.955273+0000","last_peered":"2024-05-22T23:57:39.955273+0000","last_clean":"2024-05-22T23:57:39.955273+0000","last_became_active":"2024-05-22T23:57:32.333531+0000","last_became_peered":"2024-05-22T23:57:32.333531+0000","last_unstale":"2024-05-22T23:57:39.955273+0000","last_undegraded":"2024-05-22T23:57:39.955273+0000","last_fullsized":"2024-05-22T23:57:39.955273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:53:21.843150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955703+0000","last_change":"2024-05-22T23:57:33.334366+0000","last_active":"2024-05-22T23:57:39.955703+0000","last_peered":"2024-05-22T23:57:39.955703+0000","last_clean":"2024-05-22T23:57:39.955703+0000","last_became_active":"2024-05-22T23:57:33.334037+0000","last_became_peered":"2024-05-22T23:57:33.334037+0000","last_unstale":"2024-05-22T23:57:39.955703+0000","last_undegraded":"2024-05-22T23:57:39.955703+0000","last_fullsized":"2024-05-22T23:57:39.955703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:15:10.631087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954652+0000","last_change":"2024-05-22T23:57:37.780241+0000","last_active":"2024-05-22T23:57:39.954652+0000","last_peered":"2024-05-22T23:57:39.954652+0000","last_clean":"2024-05-22T23:57:39.954652+0000","last_became_active":"2024-05-22T23:57:37.779816+0000","last_became_peered":"2024-05-22T23:57:37.779816+0000","last_unstale":"2024-05-22T23:57:39.954652+0000","last_undegraded":"2024-05-22T23:57:39.954652+0000","last_fullsized":"2024-05-22T23:57:39.954652+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:11.397681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.959726+0000","last_change":"2024-05-22T23:57:32.334829+0000","last_active":"2024-05-22T23:57:39.959726+0000","last_peered":"2024-05-22T23:57:39.959726+0000","last_clean":"2024-05-22T23:57:39.959726+0000","last_became_active":"2024-05-22T23:57:32.334368+0000","last_became_peered":"2024-05-22T23:57:32.334368+0000","last_unstale":"2024-05-22T23:57:39.959726+0000","last_undegraded":"2024-05-22T23:57:39.959726+0000","last_fullsized":"2024-05-22T23:57:39.959726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:17:42.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955807+0000","last_change":"2024-05-22T23:57:33.318740+0000","last_active":"2024-05-22T23:57:39.955807+0000","last_peered":"2024-05-22T23:57:39.955807+0000","last_clean":"2024-05-22T23:57:39.955807+0000","last_became_active":"2024-05-22T23:57:33.318363+0000","last_became_peered":"2024-05-22T23:57:33.318363+0000","last_unstale":"2024-05-22T23:57:39.955807+0000","last_undegraded":"2024-05-22T23:57:39.955807+0000","last_fullsized":"2024-05-22T23:57:39.955807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:51:42.319667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668014+0000","last_change":"2024-05-22T23:57:33.325582+0000","last_active":"2024-05-22T23:57:38.668014+0000","last_peered":"2024-05-22T23:57:38.668014+0000","last_clean":"2024-05-22T23:57:38.668014+0000","last_became_active":"2024-05-22T23:57:33.325268+0000","last_became_peered":"2024-05-22T23:57:33.325268+0000","last_unstale":"2024-05-22T23:57:38.668014+0000","last_undegraded":"2024-05-22T23:57:38.668014+0000","last_fullsized":"2024-05-22T23:57:38.668014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:02:18.968634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335721+0000","last_change":"2024-05-22T23:57:33.325875+0000","last_active":"2024-05-22T23:57:38.335721+0000","last_peered":"2024-05-22T23:57:38.335721+0000","last_clean":"2024-05-22T23:57:38.335721+0000","last_became_active":"2024-05-22T23:57:33.325540+0000","last_became_peered":"2024-05-22T23:57:33.325540+0000","last_unstale":"2024-05-22T23:57:38.335721+0000","last_undegraded":"2024-05-22T23:57:38.335721+0000","last_fullsized":"2024-05-22T23:57:38.335721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:59:42.817107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335334+0000","last_change":"2024-05-22T23:57:33.332935+0000","last_active":"2024-05-22T23:57:38.335334+0000","last_peered":"2024-05-22T23:57:38.335334+0000","last_clean":"2024-05-22T23:57:38.335334+0000","last_became_active":"2024-05-22T23:57:33.332607+0000","last_became_peered":"2024-05-22T23:57:33.332607+0000","last_unstale":"2024-05-22T23:57:38.335334+0000","last_undegraded":"2024-05-22T23:57:38.335334+0000","last_fullsized":"2024-05-22T23:57:38.335334+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:09.589896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338514+0000","last_change":"2024-05-22T23:57:33.333269+0000","last_active":"2024-05-22T23:57:38.338514+0000","last_peered":"2024-05-22T23:57:38.338514+0000","last_clean":"2024-05-22T23:57:38.338514+0000","last_became_active":"2024-05-22T23:57:33.329017+0000","last_became_peered":"2024-05-22T23:57:33.329017+0000","last_unstale":"2024-05-22T23:57:38.338514+0000","last_undegraded":"2024-05-22T23:57:38.338514+0000","last_fullsized":"2024-05-22T23:57:38.338514+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:35:42.156434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958815+0000","last_change":"2024-05-22T23:57:33.328721+0000","last_active":"2024-05-22T23:57:39.958815+0000","last_peered":"2024-05-22T23:57:39.958815+0000","last_clean":"2024-05-22T23:57:39.958815+0000","last_became_active":"2024-05-22T23:57:33.328318+0000","last_became_peered":"2024-05-22T23:57:33.328318+0000","last_unstale":"2024-05-22T23:57:39.958815+0000","last_undegraded":"2024-05-22T23:57:39.958815+0000","last_fullsized":"2024-05-22T23:57:39.958815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:14.772688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668343+0000","last_change":"2024-05-22T23:57:33.324777+0000","last_active":"2024-05-22T23:57:38.668343+0000","last_peered":"2024-05-22T23:57:38.668343+0000","last_clean":"2024-05-22T23:57:38.668343+0000","last_became_active":"2024-05-22T23:57:33.324446+0000","last_became_peered":"2024-05-22T23:57:33.324446+0000","last_unstale":"2024-05-22T23:57:38.668343+0000","last_undegraded":"2024-05-22T23:57:38.668343+0000","last_fullsized":"2024-05-22T23:57:38.668343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:20:42.327094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335801+0000","last_change":"2024-05-22T23:57:33.318783+0000","last_active":"2024-05-22T23:57:38.335801+0000","last_peered":"2024-05-22T23:57:38.335801+0000","last_clean":"2024-05-22T23:57:38.335801+0000","last_became_active":"2024-05-22T23:57:33.318481+0000","last_became_peered":"2024-05-22T23:57:33.318481+0000","last_unstale":"2024-05-22T23:57:38.335801+0000","last_undegraded":"2024-05-22T23:57:38.335801+0000","last_fullsized":"2024-05-22T23:57:38.335801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:14:35.745683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878949+0000","last_change":"2024-05-22T23:57:33.319939+0000","last_active":"2024-05-22T23:57:39.878949+0000","last_peered":"2024-05-22T23:57:39.878949+0000","last_clean":"2024-05-22T23:57:39.878949+0000","last_became_active":"2024-05-22T23:57:33.316029+0000","last_became_peered":"2024-05-22T23:57:33.316029+0000","last_unstale":"2024-05-22T23:57:39.878949+0000","last_undegraded":"2024-05-22T23:57:39.878949+0000","last_fullsized":"2024-05-22T23:57:39.878949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:46:34.383953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954758+0000","last_change":"2024-05-22T23:57:33.334625+0000","last_active":"2024-05-22T23:57:39.954758+0000","last_peered":"2024-05-22T23:57:39.954758+0000","last_clean":"2024-05-22T23:57:39.954758+0000","last_became_active":"2024-05-22T23:57:33.334341+0000","last_became_peered":"2024-05-22T23:57:33.334341+0000","last_unstale":"2024-05-22T23:57:39.954758+0000","last_undegraded":"2024-05-22T23:57:39.954758+0000","last_fullsized":"2024-05-22T23:57:39.954758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:38.319721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878077+0000","last_change":"2024-05-22T23:57:33.330958+0000","last_active":"2024-05-22T23:57:39.878077+0000","last_peered":"2024-05-22T23:57:39.878077+0000","last_clean":"2024-05-22T23:57:39.878077+0000","last_became_active":"2024-05-22T23:57:33.317355+0000","last_became_peered":"2024-05-22T23:57:33.317355+0000","last_unstale":"2024-05-22T23:57:39.878077+0000","last_undegraded":"2024-05-22T23:57:39.878077+0000","last_fullsized":"2024-05-22T23:57:39.878077+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:49:43.784780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955348+0000","last_change":"2024-05-22T23:57:33.319530+0000","last_active":"2024-05-22T23:57:39.955348+0000","last_peered":"2024-05-22T23:57:39.955348+0000","last_clean":"2024-05-22T23:57:39.955348+0000","last_became_active":"2024-05-22T23:57:33.319122+0000","last_became_peered":"2024-05-22T23:57:33.319122+0000","last_unstale":"2024-05-22T23:57:39.955348+0000","last_undegraded":"2024-05-22T23:57:39.955348+0000","last_fullsized":"2024-05-22T23:57:39.955348+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:31:26.937804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957776+0000","last_change":"2024-05-22T23:57:33.323722+0000","last_active":"2024-05-22T23:57:39.957776+0000","last_peered":"2024-05-22T23:57:39.957776+0000","last_clean":"2024-05-22T23:57:39.957776+0000","last_became_active":"2024-05-22T23:57:33.323343+0000","last_became_peered":"2024-05-22T23:57:33.323343+0000","last_unstale":"2024-05-22T23:57:39.957776+0000","last_undegraded":"2024-05-22T23:57:39.957776+0000","last_fullsized":"2024-05-22T23:57:39.957776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:24:59.575536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668203+0000","last_change":"2024-05-22T23:57:33.320925+0000","last_active":"2024-05-22T23:57:38.668203+0000","last_peered":"2024-05-22T23:57:38.668203+0000","last_clean":"2024-05-22T23:57:38.668203+0000","last_became_active":"2024-05-22T23:57:33.320570+0000","last_became_peered":"2024-05-22T23:57:33.320570+0000","last_unstale":"2024-05-22T23:57:38.668203+0000","last_undegraded":"2024-05-22T23:57:38.668203+0000","last_fullsized":"2024-05-22T23:57:38.668203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:38:27.659025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244946,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":43989,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":19,"num_read_kb":53,"num_write":137,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":79359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318292,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":485342,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579930,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1114112,"data_stored":1117565,"data_compressed":50343,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":489674,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005093,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":485498,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266732,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":1081796,"data_compressed":50343,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430255,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":484760,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":488601,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":485454,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":489951,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":485044,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888124,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":484782,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691896,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":1076604,"data_compressed":50343,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-22T23:58:09.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-22T23:58:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.404+0000 7f6360600640 1 -- 172.21.15.103:0/3041675250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 msgr2=0x7f6358103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.404+0000 7f6360600640 1 --2- 172.21.15.103:0/3041675250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 0x7f6358103f10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f634c009900 tx=0x7f634c02a3f0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.404+0000 7f6360600640 1 -- 172.21.15.103:0/3041675250 shutdown_connections 2024-05-22T23:58:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.404+0000 7f6360600640 1 --2- 172.21.15.103:0/3041675250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63581056a0 0x7f635810bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.404+0000 7f6360600640 1 --2- 172.21.15.103:0/3041675250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6358104ce0 0x7f6358105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.405+0000 7f6360600640 1 --2- 172.21.15.103:0/3041675250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 0x7f6358103f10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.405+0000 7f6360600640 1 -- 172.21.15.103:0/3041675250 >> 172.21.15.103:0/3041675250 conn(0x7f63580ff060 msgr2=0x7f63581014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:09.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.405+0000 7f6360600640 1 -- 172.21.15.103:0/3041675250 shutdown_connections 2024-05-22T23:58:09.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.405+0000 7f6360600640 1 -- 172.21.15.103:0/3041675250 wait complete. 2024-05-22T23:58:09.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.406+0000 7f6360600640 1 Processor -- start 2024-05-22T23:58:09.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.406+0000 7f6360600640 1 -- start start 2024-05-22T23:58:09.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.406+0000 7f6360600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 0x7f6358199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:09.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.406+0000 7f6360600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6358104ce0 0x7f635819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:09.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.406+0000 7f6360600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63581056a0 0x7f63581a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:09.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.406+0000 7f6360600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f635810f7c0 con 0x7f63581056a0 2024-05-22T23:58:09.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.406+0000 7f6360600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f635810f640 con 0x7f6358103af0 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.406+0000 7f6360600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f635810f940 con 0x7f6358104ce0 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 0x7f6358199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635dc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 0x7f6358199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:33864/0 (socket says 172.21.15.103:33864) 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635dc00640 1 -- 172.21.15.103:0/524127029 learned_addr learned my addr 172.21.15.103:0/524127029 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635d200640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6358104ce0 0x7f635819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635e600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63581056a0 0x7f63581a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635e600640 1 -- 172.21.15.103:0/524127029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6358104ce0 msgr2=0x7f635819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635e600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6358104ce0 0x7f635819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635e600640 1 -- 172.21.15.103:0/524127029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 msgr2=0x7f6358199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635e600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 0x7f6358199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635e600640 1 -- 172.21.15.103:0/524127029 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63581a1a30 con 0x7f63581056a0 2024-05-22T23:58:09.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.407+0000 7f635d200640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6358104ce0 0x7f635819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:09.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.408+0000 7f635e600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63581056a0 0x7f63581a13c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6350009fd0 tx=0x7f635000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:09.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.408+0000 7f6356000640 1 -- 172.21.15.103:0/524127029 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635000ce20 con 0x7f63581056a0 2024-05-22T23:58:09.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.408+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63581a1d20 con 0x7f63581056a0 2024-05-22T23:58:09.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.408+0000 7f6356000640 1 -- 172.21.15.103:0/524127029 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6350005020 con 0x7f63581056a0 2024-05-22T23:58:09.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.408+0000 7f6356000640 1 -- 172.21.15.103:0/524127029 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63500106c0 con 0x7f63581056a0 2024-05-22T23:58:09.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.408+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63581a2210 con 0x7f63581056a0 2024-05-22T23:58:09.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.409+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f635810ae10 con 0x7f63581056a0 2024-05-22T23:58:09.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.411+0000 7f6356000640 1 -- 172.21.15.103:0/524127029 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f6350005540 con 0x7f63581056a0 2024-05-22T23:58:09.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.412+0000 7f6356000640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f63400a2c20 0x7f63400a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:09.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.412+0000 7f6356000640 1 -- 172.21.15.103:0/524127029 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f63500cfac0 con 0x7f63581056a0 2024-05-22T23:58:09.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.413+0000 7f635dc00640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f63400a2c20 0x7f63400a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:09.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.413+0000 7f635dc00640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f63400a2c20 0x7f63400a50e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f634c000c00 tx=0x7f634c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:09.420 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.417+0000 7f6356000640 1 -- 172.21.15.103:0/524127029 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f635009c620 con 0x7f63581056a0 2024-05-22T23:58:09.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:09 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/4169150445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:09.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:09 smithi105 ceph-mon[36426]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:58:09.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:09 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/4169150445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:09.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:09 smithi103 ceph-mon[31370]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:58:09.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.614+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6358061cb0 con 0x7f63400a2c20 2024-05-22T23:58:09.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:09 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/4169150445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:09.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:09 smithi174 ceph-mon[36179]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:58:09.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.630+0000 7f6356000640 1 -- 172.21.15.103:0/524127029 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340624 (secure 0 0 0) 0x7f6358061cb0 con 0x7f63400a2c20 2024-05-22T23:58:09.637 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:09.638 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-05-22T23:58:09.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f63400a2c20 msgr2=0x7f63400a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:09.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f63400a2c20 0x7f63400a50e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f634c000c00 tx=0x7f634c0315b0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63581056a0 msgr2=0x7f63581a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:09.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63581056a0 0x7f63581a13c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6350009fd0 tx=0x7f635000ef10 comp rx=0 tx=0).stop 2024-05-22T23:58:09.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 shutdown_connections 2024-05-22T23:58:09.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f63400a2c20 0x7f63400a50e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63581056a0 0x7f63581a13c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6358104ce0 0x7f635819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 --2- 172.21.15.103:0/524127029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6358103af0 0x7f6358199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:09.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 >> 172.21.15.103:0/524127029 conn(0x7f63580ff060 msgr2=0x7f6358107fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:09.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 shutdown_connections 2024-05-22T23:58:09.641 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:09.636+0000 7f6360600640 1 -- 172.21.15.103:0/524127029 wait complete. 2024-05-22T23:58:09.657 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-05-22T23:58:08.117340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634474,"num_objects":60,"num_object_clones":0,"num_object_copies":181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":155,"num_read_kb":172,"num_write":319,"num_write_kb":3024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356912,"kb_used_data":11920,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584912,"statfs":{"total":1151940427776,"available":1151574949888,"internally_reserved":0,"allocated":12206080,"data_stored":7655071,"data_compressed":282852,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39608,"num_objects":36,"num_object_clones":0,"num_object_copies":109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":19,"num_read_kb":53,"num_write":90,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":24,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006272"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668473+0000","last_change":"2024-05-22T23:57:33.328075+0000","last_active":"2024-05-22T23:57:38.668473+0000","last_peered":"2024-05-22T23:57:38.668473+0000","last_clean":"2024-05-22T23:57:38.668473+0000","last_became_active":"2024-05-22T23:57:33.327704+0000","last_became_peered":"2024-05-22T23:57:33.327704+0000","last_unstale":"2024-05-22T23:57:38.668473+0000","last_undegraded":"2024-05-22T23:57:38.668473+0000","last_fullsized":"2024-05-22T23:57:38.668473+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:42:32.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338751+0000","last_change":"2024-05-22T23:57:33.332043+0000","last_active":"2024-05-22T23:57:38.338751+0000","last_peered":"2024-05-22T23:57:38.338751+0000","last_clean":"2024-05-22T23:57:38.338751+0000","last_became_active":"2024-05-22T23:57:33.322269+0000","last_became_peered":"2024-05-22T23:57:33.322269+0000","last_unstale":"2024-05-22T23:57:38.338751+0000","last_undegraded":"2024-05-22T23:57:38.338751+0000","last_fullsized":"2024-05-22T23:57:38.338751+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:40:23.542938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338063+0000","last_change":"2024-05-22T23:57:33.334993+0000","last_active":"2024-05-22T23:57:38.338063+0000","last_peered":"2024-05-22T23:57:38.338063+0000","last_clean":"2024-05-22T23:57:38.338063+0000","last_became_active":"2024-05-22T23:57:33.326446+0000","last_became_peered":"2024-05-22T23:57:33.326446+0000","last_unstale":"2024-05-22T23:57:38.338063+0000","last_undegraded":"2024-05-22T23:57:38.338063+0000","last_fullsized":"2024-05-22T23:57:38.338063+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:44:45.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336567+0000","last_change":"2024-05-22T23:57:33.320181+0000","last_active":"2024-05-22T23:57:38.336567+0000","last_peered":"2024-05-22T23:57:38.336567+0000","last_clean":"2024-05-22T23:57:38.336567+0000","last_became_active":"2024-05-22T23:57:33.319630+0000","last_became_peered":"2024-05-22T23:57:33.319630+0000","last_unstale":"2024-05-22T23:57:38.336567+0000","last_undegraded":"2024-05-22T23:57:38.336567+0000","last_fullsized":"2024-05-22T23:57:38.336567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:38:51.836241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504159+0000","last_change":"2024-05-22T23:57:33.332168+0000","last_active":"2024-05-22T23:57:38.504159+0000","last_peered":"2024-05-22T23:57:38.504159+0000","last_clean":"2024-05-22T23:57:38.504159+0000","last_became_active":"2024-05-22T23:57:33.331755+0000","last_became_peered":"2024-05-22T23:57:33.331755+0000","last_unstale":"2024-05-22T23:57:38.504159+0000","last_undegraded":"2024-05-22T23:57:38.504159+0000","last_fullsized":"2024-05-22T23:57:38.504159+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:28.013365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956794+0000","last_change":"2024-05-22T23:57:33.322449+0000","last_active":"2024-05-22T23:57:39.956794+0000","last_peered":"2024-05-22T23:57:39.956794+0000","last_clean":"2024-05-22T23:57:39.956794+0000","last_became_active":"2024-05-22T23:57:33.322099+0000","last_became_peered":"2024-05-22T23:57:33.322099+0000","last_unstale":"2024-05-22T23:57:39.956794+0000","last_undegraded":"2024-05-22T23:57:39.956794+0000","last_fullsized":"2024-05-22T23:57:39.956794+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:38.427434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955072+0000","last_change":"2024-05-22T23:57:33.322391+0000","last_active":"2024-05-22T23:57:39.955072+0000","last_peered":"2024-05-22T23:57:39.955072+0000","last_clean":"2024-05-22T23:57:39.955072+0000","last_became_active":"2024-05-22T23:57:33.322076+0000","last_became_peered":"2024-05-22T23:57:33.322076+0000","last_unstale":"2024-05-22T23:57:39.955072+0000","last_undegraded":"2024-05-22T23:57:39.955072+0000","last_fullsized":"2024-05-22T23:57:39.955072+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:45:44.857586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.877939+0000","last_change":"2024-05-22T23:57:33.333005+0000","last_active":"2024-05-22T23:57:39.877939+0000","last_peered":"2024-05-22T23:57:39.877939+0000","last_clean":"2024-05-22T23:57:39.877939+0000","last_became_active":"2024-05-22T23:57:33.332669+0000","last_became_peered":"2024-05-22T23:57:33.332669+0000","last_unstale":"2024-05-22T23:57:39.877939+0000","last_undegraded":"2024-05-22T23:57:39.877939+0000","last_fullsized":"2024-05-22T23:57:39.877939+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:47:25.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956357+0000","last_change":"2024-05-22T23:57:33.328428+0000","last_active":"2024-05-22T23:57:39.956357+0000","last_peered":"2024-05-22T23:57:39.956357+0000","last_clean":"2024-05-22T23:57:39.956357+0000","last_became_active":"2024-05-22T23:57:33.327574+0000","last_became_peered":"2024-05-22T23:57:33.327574+0000","last_unstale":"2024-05-22T23:57:39.956357+0000","last_undegraded":"2024-05-22T23:57:39.956357+0000","last_fullsized":"2024-05-22T23:57:39.956357+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:35:07.217079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956792+0000","last_change":"2024-05-22T23:57:33.324445+0000","last_active":"2024-05-22T23:57:39.956792+0000","last_peered":"2024-05-22T23:57:39.956792+0000","last_clean":"2024-05-22T23:57:39.956792+0000","last_became_active":"2024-05-22T23:57:33.324085+0000","last_became_peered":"2024-05-22T23:57:33.324085+0000","last_unstale":"2024-05-22T23:57:39.956792+0000","last_undegraded":"2024-05-22T23:57:39.956792+0000","last_fullsized":"2024-05-22T23:57:39.956792+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:30:51.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956263+0000","last_change":"2024-05-22T23:57:33.321481+0000","last_active":"2024-05-22T23:57:39.956263+0000","last_peered":"2024-05-22T23:57:39.956263+0000","last_clean":"2024-05-22T23:57:39.956263+0000","last_became_active":"2024-05-22T23:57:33.320911+0000","last_became_peered":"2024-05-22T23:57:33.320911+0000","last_unstale":"2024-05-22T23:57:39.956263+0000","last_undegraded":"2024-05-22T23:57:39.956263+0000","last_fullsized":"2024-05-22T23:57:39.956263+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:24:24.041980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336271+0000","last_change":"2024-05-22T23:57:33.318819+0000","last_active":"2024-05-22T23:57:38.336271+0000","last_peered":"2024-05-22T23:57:38.336271+0000","last_clean":"2024-05-22T23:57:38.336271+0000","last_became_active":"2024-05-22T23:57:33.318284+0000","last_became_peered":"2024-05-22T23:57:33.318284+0000","last_unstale":"2024-05-22T23:57:38.336271+0000","last_undegraded":"2024-05-22T23:57:38.336271+0000","last_fullsized":"2024-05-22T23:57:38.336271+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:56.043446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957969+0000","last_change":"2024-05-22T23:57:33.321268+0000","last_active":"2024-05-22T23:57:39.957969+0000","last_peered":"2024-05-22T23:57:39.957969+0000","last_clean":"2024-05-22T23:57:39.957969+0000","last_became_active":"2024-05-22T23:57:33.320902+0000","last_became_peered":"2024-05-22T23:57:33.320902+0000","last_unstale":"2024-05-22T23:57:39.957969+0000","last_undegraded":"2024-05-22T23:57:39.957969+0000","last_fullsized":"2024-05-22T23:57:39.957969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:51:39.716541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.337466+0000","last_change":"2024-05-22T23:57:33.322638+0000","last_active":"2024-05-22T23:57:38.337466+0000","last_peered":"2024-05-22T23:57:38.337466+0000","last_clean":"2024-05-22T23:57:38.337466+0000","last_became_active":"2024-05-22T23:57:33.322318+0000","last_became_peered":"2024-05-22T23:57:33.322318+0000","last_unstale":"2024-05-22T23:57:38.337466+0000","last_undegraded":"2024-05-22T23:57:38.337466+0000","last_fullsized":"2024-05-22T23:57:38.337466+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:44:25.487618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.337446+0000","last_change":"2024-05-22T23:57:33.325832+0000","last_active":"2024-05-22T23:57:38.337446+0000","last_peered":"2024-05-22T23:57:38.337446+0000","last_clean":"2024-05-22T23:57:38.337446+0000","last_became_active":"2024-05-22T23:57:33.325062+0000","last_became_peered":"2024-05-22T23:57:33.325062+0000","last_unstale":"2024-05-22T23:57:38.337446+0000","last_undegraded":"2024-05-22T23:57:38.337446+0000","last_fullsized":"2024-05-22T23:57:38.337446+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:33:04.161351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954875+0000","last_change":"2024-05-22T23:57:33.324864+0000","last_active":"2024-05-22T23:57:39.954875+0000","last_peered":"2024-05-22T23:57:39.954875+0000","last_clean":"2024-05-22T23:57:39.954875+0000","last_became_active":"2024-05-22T23:57:33.324558+0000","last_became_peered":"2024-05-22T23:57:33.324558+0000","last_unstale":"2024-05-22T23:57:39.954875+0000","last_undegraded":"2024-05-22T23:57:39.954875+0000","last_fullsized":"2024-05-22T23:57:39.954875+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:48:49.722371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668272+0000","last_change":"2024-05-22T23:57:33.337985+0000","last_active":"2024-05-22T23:57:38.668272+0000","last_peered":"2024-05-22T23:57:38.668272+0000","last_clean":"2024-05-22T23:57:38.668272+0000","last_became_active":"2024-05-22T23:57:33.337719+0000","last_became_peered":"2024-05-22T23:57:33.337719+0000","last_unstale":"2024-05-22T23:57:38.668272+0000","last_undegraded":"2024-05-22T23:57:38.668272+0000","last_fullsized":"2024-05-22T23:57:38.668272+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:50:44.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.960606+0000","last_change":"2024-05-22T23:57:33.325709+0000","last_active":"2024-05-22T23:57:39.960606+0000","last_peered":"2024-05-22T23:57:39.960606+0000","last_clean":"2024-05-22T23:57:39.960606+0000","last_became_active":"2024-05-22T23:57:33.325063+0000","last_became_peered":"2024-05-22T23:57:33.325063+0000","last_unstale":"2024-05-22T23:57:39.960606+0000","last_undegraded":"2024-05-22T23:57:39.960606+0000","last_fullsized":"2024-05-22T23:57:39.960606+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:39.145217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336101+0000","last_change":"2024-05-22T23:57:33.333251+0000","last_active":"2024-05-22T23:57:38.336101+0000","last_peered":"2024-05-22T23:57:38.336101+0000","last_clean":"2024-05-22T23:57:38.336101+0000","last_became_active":"2024-05-22T23:57:33.332894+0000","last_became_peered":"2024-05-22T23:57:33.332894+0000","last_unstale":"2024-05-22T23:57:38.336101+0000","last_undegraded":"2024-05-22T23:57:38.336101+0000","last_fullsized":"2024-05-22T23:57:38.336101+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:13:12.082781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335605+0000","last_change":"2024-05-22T23:57:37.788060+0000","last_active":"2024-05-22T23:57:38.335605+0000","last_peered":"2024-05-22T23:57:38.335605+0000","last_clean":"2024-05-22T23:57:38.335605+0000","last_became_active":"2024-05-22T23:57:37.787656+0000","last_became_peered":"2024-05-22T23:57:37.787656+0000","last_unstale":"2024-05-22T23:57:38.335605+0000","last_undegraded":"2024-05-22T23:57:38.335605+0000","last_fullsized":"2024-05-22T23:57:38.335605+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:39:13.718817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.260659+0000","last_change":"2024-05-22T23:57:32.323434+0000","last_active":"2024-05-22T23:58:04.260659+0000","last_peered":"2024-05-22T23:58:04.260659+0000","last_clean":"2024-05-22T23:58:04.260659+0000","last_became_active":"2024-05-22T23:57:32.323033+0000","last_became_peered":"2024-05-22T23:57:32.323033+0000","last_unstale":"2024-05-22T23:58:04.260659+0000","last_undegraded":"2024-05-22T23:58:04.260659+0000","last_fullsized":"2024-05-22T23:58:04.260659+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:15:27.676027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878645+0000","last_change":"2024-05-22T23:57:33.323336+0000","last_active":"2024-05-22T23:57:39.878645+0000","last_peered":"2024-05-22T23:57:39.878645+0000","last_clean":"2024-05-22T23:57:39.878645+0000","last_became_active":"2024-05-22T23:57:33.323026+0000","last_became_peered":"2024-05-22T23:57:33.323026+0000","last_unstale":"2024-05-22T23:57:39.878645+0000","last_undegraded":"2024-05-22T23:57:39.878645+0000","last_fullsized":"2024-05-22T23:57:39.878645+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:02:20.142760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335797+0000","last_change":"2024-05-22T23:57:37.785137+0000","last_active":"2024-05-22T23:57:38.335797+0000","last_peered":"2024-05-22T23:57:38.335797+0000","last_clean":"2024-05-22T23:57:38.335797+0000","last_became_active":"2024-05-22T23:57:37.784791+0000","last_became_peered":"2024-05-22T23:57:37.784791+0000","last_unstale":"2024-05-22T23:57:38.335797+0000","last_undegraded":"2024-05-22T23:57:38.335797+0000","last_fullsized":"2024-05-22T23:57:38.335797+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:54:51.036334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878798+0000","last_change":"2024-05-22T23:57:32.332969+0000","last_active":"2024-05-22T23:57:39.878798+0000","last_peered":"2024-05-22T23:57:39.878798+0000","last_clean":"2024-05-22T23:57:39.878798+0000","last_became_active":"2024-05-22T23:57:32.332611+0000","last_became_peered":"2024-05-22T23:57:32.332611+0000","last_unstale":"2024-05-22T23:57:39.878798+0000","last_undegraded":"2024-05-22T23:57:39.878798+0000","last_fullsized":"2024-05-22T23:57:39.878798+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:00:22.006355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.339090+0000","last_change":"2024-05-22T23:57:33.332151+0000","last_active":"2024-05-22T23:57:38.339090+0000","last_peered":"2024-05-22T23:57:38.339090+0000","last_clean":"2024-05-22T23:57:38.339090+0000","last_became_active":"2024-05-22T23:57:33.329518+0000","last_became_peered":"2024-05-22T23:57:33.329518+0000","last_unstale":"2024-05-22T23:57:38.339090+0000","last_undegraded":"2024-05-22T23:57:38.339090+0000","last_fullsized":"2024-05-22T23:57:38.339090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:10:12.083766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336083+0000","last_change":"2024-05-22T23:57:37.787524+0000","last_active":"2024-05-22T23:57:38.336083+0000","last_peered":"2024-05-22T23:57:38.336083+0000","last_clean":"2024-05-22T23:57:38.336083+0000","last_became_active":"2024-05-22T23:57:37.787103+0000","last_became_peered":"2024-05-22T23:57:37.787103+0000","last_unstale":"2024-05-22T23:57:38.336083+0000","last_undegraded":"2024-05-22T23:57:38.336083+0000","last_fullsized":"2024-05-22T23:57:38.336083+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:19:16.963159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.255300+0000","last_change":"2024-05-22T23:57:32.318469+0000","last_active":"2024-05-22T23:58:04.255300+0000","last_peered":"2024-05-22T23:58:04.255300+0000","last_clean":"2024-05-22T23:58:04.255300+0000","last_became_active":"2024-05-22T23:57:32.317640+0000","last_became_peered":"2024-05-22T23:57:32.317640+0000","last_unstale":"2024-05-22T23:58:04.255300+0000","last_undegraded":"2024-05-22T23:58:04.255300+0000","last_fullsized":"2024-05-22T23:58:04.255300+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:01:55.710019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956539+0000","last_change":"2024-05-22T23:57:33.318604+0000","last_active":"2024-05-22T23:57:39.956539+0000","last_peered":"2024-05-22T23:57:39.956539+0000","last_clean":"2024-05-22T23:57:39.956539+0000","last_became_active":"2024-05-22T23:57:33.318184+0000","last_became_peered":"2024-05-22T23:57:33.318184+0000","last_unstale":"2024-05-22T23:57:39.956539+0000","last_undegraded":"2024-05-22T23:57:39.956539+0000","last_fullsized":"2024-05-22T23:57:39.956539+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:16.140799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335942+0000","last_change":"2024-05-22T23:57:37.796085+0000","last_active":"2024-05-22T23:57:38.335942+0000","last_peered":"2024-05-22T23:57:38.335942+0000","last_clean":"2024-05-22T23:57:38.335942+0000","last_became_active":"2024-05-22T23:57:37.795265+0000","last_became_peered":"2024-05-22T23:57:37.795265+0000","last_unstale":"2024-05-22T23:57:38.335942+0000","last_undegraded":"2024-05-22T23:57:38.335942+0000","last_fullsized":"2024-05-22T23:57:38.335942+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:43:55.577072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504490+0000","last_change":"2024-05-22T23:57:32.316346+0000","last_active":"2024-05-22T23:57:38.504490+0000","last_peered":"2024-05-22T23:57:38.504490+0000","last_clean":"2024-05-22T23:57:38.504490+0000","last_became_active":"2024-05-22T23:57:32.316038+0000","last_became_peered":"2024-05-22T23:57:32.316038+0000","last_unstale":"2024-05-22T23:57:38.504490+0000","last_undegraded":"2024-05-22T23:57:38.504490+0000","last_fullsized":"2024-05-22T23:57:38.504490+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:57:41.987250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955165+0000","last_change":"2024-05-22T23:57:33.329341+0000","last_active":"2024-05-22T23:57:39.955165+0000","last_peered":"2024-05-22T23:57:39.955165+0000","last_clean":"2024-05-22T23:57:39.955165+0000","last_became_active":"2024-05-22T23:57:33.329009+0000","last_became_peered":"2024-05-22T23:57:33.329009+0000","last_unstale":"2024-05-22T23:57:39.955165+0000","last_undegraded":"2024-05-22T23:57:39.955165+0000","last_fullsized":"2024-05-22T23:57:39.955165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:53:07.591497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336071+0000","last_change":"2024-05-22T23:57:37.779097+0000","last_active":"2024-05-22T23:57:38.336071+0000","last_peered":"2024-05-22T23:57:38.336071+0000","last_clean":"2024-05-22T23:57:38.336071+0000","last_became_active":"2024-05-22T23:57:37.778564+0000","last_became_peered":"2024-05-22T23:57:37.778564+0000","last_unstale":"2024-05-22T23:57:38.336071+0000","last_undegraded":"2024-05-22T23:57:38.336071+0000","last_fullsized":"2024-05-22T23:57:38.336071+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:09:16.351947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.243487+0000","last_change":"2024-05-22T23:57:32.316031+0000","last_active":"2024-05-22T23:58:03.243487+0000","last_peered":"2024-05-22T23:58:03.243487+0000","last_clean":"2024-05-22T23:58:03.243487+0000","last_became_active":"2024-05-22T23:57:32.315673+0000","last_became_peered":"2024-05-22T23:57:32.315673+0000","last_unstale":"2024-05-22T23:58:03.243487+0000","last_undegraded":"2024-05-22T23:58:03.243487+0000","last_fullsized":"2024-05-22T23:58:03.243487+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:28.316600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504150+0000","last_change":"2024-05-22T23:57:33.324587+0000","last_active":"2024-05-22T23:57:38.504150+0000","last_peered":"2024-05-22T23:57:38.504150+0000","last_clean":"2024-05-22T23:57:38.504150+0000","last_became_active":"2024-05-22T23:57:33.323972+0000","last_became_peered":"2024-05-22T23:57:33.323972+0000","last_unstale":"2024-05-22T23:57:38.504150+0000","last_undegraded":"2024-05-22T23:57:38.504150+0000","last_fullsized":"2024-05-22T23:57:38.504150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:23:52.551699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335999+0000","last_change":"2024-05-22T23:57:37.791726+0000","last_active":"2024-05-22T23:57:38.335999+0000","last_peered":"2024-05-22T23:57:38.335999+0000","last_clean":"2024-05-22T23:57:38.335999+0000","last_became_active":"2024-05-22T23:57:37.791185+0000","last_became_peered":"2024-05-22T23:57:37.791185+0000","last_unstale":"2024-05-22T23:57:38.335999+0000","last_undegraded":"2024-05-22T23:57:38.335999+0000","last_fullsized":"2024-05-22T23:57:38.335999+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:55:30.994272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.259539+0000","last_change":"2024-05-22T23:57:32.328130+0000","last_active":"2024-05-22T23:58:04.259539+0000","last_peered":"2024-05-22T23:58:04.259539+0000","last_clean":"2024-05-22T23:58:04.259539+0000","last_became_active":"2024-05-22T23:57:32.327835+0000","last_became_peered":"2024-05-22T23:57:32.327835+0000","last_unstale":"2024-05-22T23:58:04.259539+0000","last_undegraded":"2024-05-22T23:58:04.259539+0000","last_fullsized":"2024-05-22T23:58:04.259539+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:32:40.442824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955119+0000","last_change":"2024-05-22T23:57:33.336525+0000","last_active":"2024-05-22T23:57:39.955119+0000","last_peered":"2024-05-22T23:57:39.955119+0000","last_clean":"2024-05-22T23:57:39.955119+0000","last_became_active":"2024-05-22T23:57:33.336245+0000","last_became_peered":"2024-05-22T23:57:33.336245+0000","last_unstale":"2024-05-22T23:57:39.955119+0000","last_undegraded":"2024-05-22T23:57:39.955119+0000","last_fullsized":"2024-05-22T23:57:39.955119+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:14.329213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668430+0000","last_change":"2024-05-22T23:57:37.787910+0000","last_active":"2024-05-22T23:57:38.668430+0000","last_peered":"2024-05-22T23:57:38.668430+0000","last_clean":"2024-05-22T23:57:38.668430+0000","last_became_active":"2024-05-22T23:57:37.787465+0000","last_became_peered":"2024-05-22T23:57:37.787465+0000","last_unstale":"2024-05-22T23:57:38.668430+0000","last_undegraded":"2024-05-22T23:57:38.668430+0000","last_fullsized":"2024-05-22T23:57:38.668430+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:54:59.202855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957447+0000","last_change":"2024-05-22T23:57:32.329365+0000","last_active":"2024-05-22T23:57:39.957447+0000","last_peered":"2024-05-22T23:57:39.957447+0000","last_clean":"2024-05-22T23:57:39.957447+0000","last_became_active":"2024-05-22T23:57:32.327864+0000","last_became_peered":"2024-05-22T23:57:32.327864+0000","last_unstale":"2024-05-22T23:57:39.957447+0000","last_undegraded":"2024-05-22T23:57:39.957447+0000","last_fullsized":"2024-05-22T23:57:39.957447+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:26.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.877820+0000","last_change":"2024-05-22T23:57:33.313930+0000","last_active":"2024-05-22T23:57:39.877820+0000","last_peered":"2024-05-22T23:57:39.877820+0000","last_clean":"2024-05-22T23:57:39.877820+0000","last_became_active":"2024-05-22T23:57:33.313612+0000","last_became_peered":"2024-05-22T23:57:33.313612+0000","last_unstale":"2024-05-22T23:57:39.877820+0000","last_undegraded":"2024-05-22T23:57:39.877820+0000","last_fullsized":"2024-05-22T23:57:39.877820+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:45:25.800338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338306+0000","last_change":"2024-05-22T23:57:37.791537+0000","last_active":"2024-05-22T23:57:38.338306+0000","last_peered":"2024-05-22T23:57:38.338306+0000","last_clean":"2024-05-22T23:57:38.338306+0000","last_became_active":"2024-05-22T23:57:37.791052+0000","last_became_peered":"2024-05-22T23:57:37.791052+0000","last_unstale":"2024-05-22T23:57:38.338306+0000","last_undegraded":"2024-05-22T23:57:38.338306+0000","last_fullsized":"2024-05-22T23:57:38.338306+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:18:11.759840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954695+0000","last_change":"2024-05-22T23:57:32.315907+0000","last_active":"2024-05-22T23:57:39.954695+0000","last_peered":"2024-05-22T23:57:39.954695+0000","last_clean":"2024-05-22T23:57:39.954695+0000","last_became_active":"2024-05-22T23:57:32.315576+0000","last_became_peered":"2024-05-22T23:57:32.315576+0000","last_unstale":"2024-05-22T23:57:39.954695+0000","last_undegraded":"2024-05-22T23:57:39.954695+0000","last_fullsized":"2024-05-22T23:57:39.954695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:45:48.998377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957104+0000","last_change":"2024-05-22T23:57:33.338681+0000","last_active":"2024-05-22T23:57:39.957104+0000","last_peered":"2024-05-22T23:57:39.957104+0000","last_clean":"2024-05-22T23:57:39.957104+0000","last_became_active":"2024-05-22T23:57:33.334705+0000","last_became_peered":"2024-05-22T23:57:33.334705+0000","last_unstale":"2024-05-22T23:57:39.957104+0000","last_undegraded":"2024-05-22T23:57:39.957104+0000","last_fullsized":"2024-05-22T23:57:39.957104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:22.135137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878483+0000","last_change":"2024-05-22T23:57:37.780470+0000","last_active":"2024-05-22T23:57:39.878483+0000","last_peered":"2024-05-22T23:57:39.878483+0000","last_clean":"2024-05-22T23:57:39.878483+0000","last_became_active":"2024-05-22T23:57:37.779110+0000","last_became_peered":"2024-05-22T23:57:37.779110+0000","last_unstale":"2024-05-22T23:57:39.878483+0000","last_undegraded":"2024-05-22T23:57:39.878483+0000","last_fullsized":"2024-05-22T23:57:39.878483+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:55:45.486862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:06.271408+0000","last_change":"2024-05-22T23:57:32.322489+0000","last_active":"2024-05-22T23:58:06.271408+0000","last_peered":"2024-05-22T23:58:06.271408+0000","last_clean":"2024-05-22T23:58:06.271408+0000","last_became_active":"2024-05-22T23:57:32.322125+0000","last_became_peered":"2024-05-22T23:57:32.322125+0000","last_unstale":"2024-05-22T23:58:06.271408+0000","last_undegraded":"2024-05-22T23:58:06.271408+0000","last_fullsized":"2024-05-22T23:58:06.271408+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:11:47.707506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955706+0000","last_change":"2024-05-22T23:57:33.322417+0000","last_active":"2024-05-22T23:57:39.955706+0000","last_peered":"2024-05-22T23:57:39.955706+0000","last_clean":"2024-05-22T23:57:39.955706+0000","last_became_active":"2024-05-22T23:57:33.322102+0000","last_became_peered":"2024-05-22T23:57:33.322102+0000","last_unstale":"2024-05-22T23:57:39.955706+0000","last_undegraded":"2024-05-22T23:57:39.955706+0000","last_fullsized":"2024-05-22T23:57:39.955706+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:26:43.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956585+0000","last_change":"2024-05-22T23:57:37.784424+0000","last_active":"2024-05-22T23:57:39.956585+0000","last_peered":"2024-05-22T23:57:39.956585+0000","last_clean":"2024-05-22T23:57:39.956585+0000","last_became_active":"2024-05-22T23:57:37.783949+0000","last_became_peered":"2024-05-22T23:57:37.783949+0000","last_unstale":"2024-05-22T23:57:39.956585+0000","last_undegraded":"2024-05-22T23:57:39.956585+0000","last_fullsized":"2024-05-22T23:57:39.956585+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:52:23.763273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.240433+0000","last_change":"2024-05-22T23:57:32.317045+0000","last_active":"2024-05-22T23:58:03.240433+0000","last_peered":"2024-05-22T23:58:03.240433+0000","last_clean":"2024-05-22T23:58:03.240433+0000","last_became_active":"2024-05-22T23:57:32.316698+0000","last_became_peered":"2024-05-22T23:57:32.316698+0000","last_unstale":"2024-05-22T23:58:03.240433+0000","last_undegraded":"2024-05-22T23:58:03.240433+0000","last_fullsized":"2024-05-22T23:58:03.240433+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:40.861593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338431+0000","last_change":"2024-05-22T23:57:33.324237+0000","last_active":"2024-05-22T23:57:38.338431+0000","last_peered":"2024-05-22T23:57:38.338431+0000","last_clean":"2024-05-22T23:57:38.338431+0000","last_became_active":"2024-05-22T23:57:33.323870+0000","last_became_peered":"2024-05-22T23:57:33.323870+0000","last_unstale":"2024-05-22T23:57:38.338431+0000","last_undegraded":"2024-05-22T23:57:38.338431+0000","last_fullsized":"2024-05-22T23:57:38.338431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:41:29.328104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336375+0000","last_change":"2024-05-22T23:57:37.787795+0000","last_active":"2024-05-22T23:57:38.336375+0000","last_peered":"2024-05-22T23:57:38.336375+0000","last_clean":"2024-05-22T23:57:38.336375+0000","last_became_active":"2024-05-22T23:57:37.787210+0000","last_became_peered":"2024-05-22T23:57:37.787210+0000","last_unstale":"2024-05-22T23:57:38.336375+0000","last_undegraded":"2024-05-22T23:57:38.336375+0000","last_fullsized":"2024-05-22T23:57:38.336375+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:19:04.616758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955176+0000","last_change":"2024-05-22T23:57:32.321415+0000","last_active":"2024-05-22T23:57:39.955176+0000","last_peered":"2024-05-22T23:57:39.955176+0000","last_clean":"2024-05-22T23:57:39.955176+0000","last_became_active":"2024-05-22T23:57:32.321055+0000","last_became_peered":"2024-05-22T23:57:32.321055+0000","last_unstale":"2024-05-22T23:57:39.955176+0000","last_undegraded":"2024-05-22T23:57:39.955176+0000","last_fullsized":"2024-05-22T23:57:39.955176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:28:58.577548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956417+0000","last_change":"2024-05-22T23:57:33.317043+0000","last_active":"2024-05-22T23:57:39.956417+0000","last_peered":"2024-05-22T23:57:39.956417+0000","last_clean":"2024-05-22T23:57:39.956417+0000","last_became_active":"2024-05-22T23:57:33.314687+0000","last_became_peered":"2024-05-22T23:57:33.314687+0000","last_unstale":"2024-05-22T23:57:39.956417+0000","last_undegraded":"2024-05-22T23:57:39.956417+0000","last_fullsized":"2024-05-22T23:57:39.956417+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:18:02.718481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.960634+0000","last_change":"2024-05-22T23:57:37.781367+0000","last_active":"2024-05-22T23:57:39.960634+0000","last_peered":"2024-05-22T23:57:39.960634+0000","last_clean":"2024-05-22T23:57:39.960634+0000","last_became_active":"2024-05-22T23:57:37.780955+0000","last_became_peered":"2024-05-22T23:57:37.780955+0000","last_unstale":"2024-05-22T23:57:39.960634+0000","last_undegraded":"2024-05-22T23:57:39.960634+0000","last_fullsized":"2024-05-22T23:57:39.960634+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:36:18.962940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'25","reported_seq":58,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:07.276226+0000","last_change":"2024-05-22T23:57:32.313873+0000","last_active":"2024-05-22T23:58:07.276226+0000","last_peered":"2024-05-22T23:58:07.276226+0000","last_clean":"2024-05-22T23:58:07.276226+0000","last_became_active":"2024-05-22T23:57:32.313388+0000","last_became_peered":"2024-05-22T23:57:32.313388+0000","last_unstale":"2024-05-22T23:58:07.276226+0000","last_undegraded":"2024-05-22T23:58:07.276226+0000","last_fullsized":"2024-05-22T23:58:07.276226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:16.383384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40469,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":44,"num_write":26,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504454+0000","last_change":"2024-05-22T23:57:33.318751+0000","last_active":"2024-05-22T23:57:38.504454+0000","last_peered":"2024-05-22T23:57:38.504454+0000","last_clean":"2024-05-22T23:57:38.504454+0000","last_became_active":"2024-05-22T23:57:33.317710+0000","last_became_peered":"2024-05-22T23:57:33.317710+0000","last_unstale":"2024-05-22T23:57:38.504454+0000","last_undegraded":"2024-05-22T23:57:38.504454+0000","last_fullsized":"2024-05-22T23:57:38.504454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:57:50.852615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955499+0000","last_change":"2024-05-22T23:57:37.790073+0000","last_active":"2024-05-22T23:57:39.955499+0000","last_peered":"2024-05-22T23:57:39.955499+0000","last_clean":"2024-05-22T23:57:39.955499+0000","last_became_active":"2024-05-22T23:57:37.789477+0000","last_became_peered":"2024-05-22T23:57:37.789477+0000","last_unstale":"2024-05-22T23:57:39.955499+0000","last_undegraded":"2024-05-22T23:57:39.955499+0000","last_fullsized":"2024-05-22T23:57:39.955499+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:11.003201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.259989+0000","last_change":"2024-05-22T23:57:32.329846+0000","last_active":"2024-05-22T23:58:04.259989+0000","last_peered":"2024-05-22T23:58:04.259989+0000","last_clean":"2024-05-22T23:58:04.259989+0000","last_became_active":"2024-05-22T23:57:32.329414+0000","last_became_peered":"2024-05-22T23:57:32.329414+0000","last_unstale":"2024-05-22T23:58:04.259989+0000","last_undegraded":"2024-05-22T23:58:04.259989+0000","last_fullsized":"2024-05-22T23:58:04.259989+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:15:38.176790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958847+0000","last_change":"2024-05-22T23:57:33.333298+0000","last_active":"2024-05-22T23:57:39.958847+0000","last_peered":"2024-05-22T23:57:39.958847+0000","last_clean":"2024-05-22T23:57:39.958847+0000","last_became_active":"2024-05-22T23:57:33.332934+0000","last_became_peered":"2024-05-22T23:57:33.332934+0000","last_unstale":"2024-05-22T23:57:39.958847+0000","last_undegraded":"2024-05-22T23:57:39.958847+0000","last_fullsized":"2024-05-22T23:57:39.958847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:06:00.427627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668248+0000","last_change":"2024-05-22T23:57:37.795628+0000","last_active":"2024-05-22T23:57:38.668248+0000","last_peered":"2024-05-22T23:57:38.668248+0000","last_clean":"2024-05-22T23:57:38.668248+0000","last_became_active":"2024-05-22T23:57:37.795208+0000","last_became_peered":"2024-05-22T23:57:37.795208+0000","last_unstale":"2024-05-22T23:57:38.668248+0000","last_undegraded":"2024-05-22T23:57:38.668248+0000","last_fullsized":"2024-05-22T23:57:38.668248+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:03:12.684474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.245200+0000","last_change":"2024-05-22T23:57:32.334039+0000","last_active":"2024-05-22T23:58:03.245200+0000","last_peered":"2024-05-22T23:58:03.245200+0000","last_clean":"2024-05-22T23:58:03.245200+0000","last_became_active":"2024-05-22T23:57:32.333709+0000","last_became_peered":"2024-05-22T23:57:32.333709+0000","last_unstale":"2024-05-22T23:58:03.245200+0000","last_undegraded":"2024-05-22T23:58:03.245200+0000","last_fullsized":"2024-05-22T23:58:03.245200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:41:43.319094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956045+0000","last_change":"2024-05-22T23:57:33.328056+0000","last_active":"2024-05-22T23:57:39.956045+0000","last_peered":"2024-05-22T23:57:39.956045+0000","last_clean":"2024-05-22T23:57:39.956045+0000","last_became_active":"2024-05-22T23:57:33.326559+0000","last_became_peered":"2024-05-22T23:57:33.326559+0000","last_unstale":"2024-05-22T23:57:39.956045+0000","last_undegraded":"2024-05-22T23:57:39.956045+0000","last_fullsized":"2024-05-22T23:57:39.956045+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:06.463837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.960354+0000","last_change":"2024-05-22T23:57:37.795892+0000","last_active":"2024-05-22T23:57:39.960354+0000","last_peered":"2024-05-22T23:57:39.960354+0000","last_clean":"2024-05-22T23:57:39.960354+0000","last_became_active":"2024-05-22T23:57:37.792246+0000","last_became_peered":"2024-05-22T23:57:37.792246+0000","last_unstale":"2024-05-22T23:57:39.960354+0000","last_undegraded":"2024-05-22T23:57:39.960354+0000","last_fullsized":"2024-05-22T23:57:39.960354+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:18.462267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.253582+0000","last_change":"2024-05-22T23:57:32.319379+0000","last_active":"2024-05-22T23:58:04.253582+0000","last_peered":"2024-05-22T23:58:04.253582+0000","last_clean":"2024-05-22T23:58:04.253582+0000","last_became_active":"2024-05-22T23:57:32.318803+0000","last_became_peered":"2024-05-22T23:57:32.318803+0000","last_unstale":"2024-05-22T23:58:04.253582+0000","last_undegraded":"2024-05-22T23:58:04.253582+0000","last_fullsized":"2024-05-22T23:58:04.253582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:59:15.689119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955097+0000","last_change":"2024-05-22T23:57:33.325294+0000","last_active":"2024-05-22T23:57:39.955097+0000","last_peered":"2024-05-22T23:57:39.955097+0000","last_clean":"2024-05-22T23:57:39.955097+0000","last_became_active":"2024-05-22T23:57:33.324339+0000","last_became_peered":"2024-05-22T23:57:33.324339+0000","last_unstale":"2024-05-22T23:57:39.955097+0000","last_undegraded":"2024-05-22T23:57:39.955097+0000","last_fullsized":"2024-05-22T23:57:39.955097+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:29:47.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.959996+0000","last_change":"2024-05-22T23:57:37.797961+0000","last_active":"2024-05-22T23:57:39.959996+0000","last_peered":"2024-05-22T23:57:39.959996+0000","last_clean":"2024-05-22T23:57:39.959996+0000","last_became_active":"2024-05-22T23:57:37.797532+0000","last_became_peered":"2024-05-22T23:57:37.797532+0000","last_unstale":"2024-05-22T23:57:39.959996+0000","last_undegraded":"2024-05-22T23:57:39.959996+0000","last_fullsized":"2024-05-22T23:57:39.959996+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:51:12.997674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.258018+0000","last_change":"2024-05-22T23:57:32.317434+0000","last_active":"2024-05-22T23:58:04.258018+0000","last_peered":"2024-05-22T23:58:04.258018+0000","last_clean":"2024-05-22T23:58:04.258018+0000","last_became_active":"2024-05-22T23:57:32.316987+0000","last_became_peered":"2024-05-22T23:57:32.316987+0000","last_unstale":"2024-05-22T23:58:04.258018+0000","last_undegraded":"2024-05-22T23:58:04.258018+0000","last_fullsized":"2024-05-22T23:58:04.258018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:46:50.306756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954821+0000","last_change":"2024-05-22T23:57:33.328670+0000","last_active":"2024-05-22T23:57:39.954821+0000","last_peered":"2024-05-22T23:57:39.954821+0000","last_clean":"2024-05-22T23:57:39.954821+0000","last_became_active":"2024-05-22T23:57:33.328294+0000","last_became_peered":"2024-05-22T23:57:33.328294+0000","last_unstale":"2024-05-22T23:57:39.954821+0000","last_undegraded":"2024-05-22T23:57:39.954821+0000","last_fullsized":"2024-05-22T23:57:39.954821+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:09:10.807844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954816+0000","last_change":"2024-05-22T23:57:37.788885+0000","last_active":"2024-05-22T23:57:39.954816+0000","last_peered":"2024-05-22T23:57:39.954816+0000","last_clean":"2024-05-22T23:57:39.954816+0000","last_became_active":"2024-05-22T23:57:37.788145+0000","last_became_peered":"2024-05-22T23:57:37.788145+0000","last_unstale":"2024-05-22T23:57:39.954816+0000","last_undegraded":"2024-05-22T23:57:39.954816+0000","last_fullsized":"2024-05-22T23:57:39.954816+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:14:50.027785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:05.268296+0000","last_change":"2024-05-22T23:57:32.317855+0000","last_active":"2024-05-22T23:58:05.268296+0000","last_peered":"2024-05-22T23:58:05.268296+0000","last_clean":"2024-05-22T23:58:05.268296+0000","last_became_active":"2024-05-22T23:57:32.317087+0000","last_became_peered":"2024-05-22T23:57:32.317087+0000","last_unstale":"2024-05-22T23:58:05.268296+0000","last_undegraded":"2024-05-22T23:58:05.268296+0000","last_fullsized":"2024-05-22T23:58:05.268296+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:49:40.486222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668239+0000","last_change":"2024-05-22T23:57:33.323897+0000","last_active":"2024-05-22T23:57:38.668239+0000","last_peered":"2024-05-22T23:57:38.668239+0000","last_clean":"2024-05-22T23:57:38.668239+0000","last_became_active":"2024-05-22T23:57:33.323567+0000","last_became_peered":"2024-05-22T23:57:33.323567+0000","last_unstale":"2024-05-22T23:57:38.668239+0000","last_undegraded":"2024-05-22T23:57:38.668239+0000","last_fullsized":"2024-05-22T23:57:38.668239+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:17:48.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.504206+0000","last_change":"2024-05-22T23:57:38.218313+0000","last_active":"2024-05-22T23:57:38.504206+0000","last_peered":"2024-05-22T23:57:38.504206+0000","last_clean":"2024-05-22T23:57:38.504206+0000","last_became_active":"2024-05-22T23:57:38.217756+0000","last_became_peered":"2024-05-22T23:57:38.217756+0000","last_unstale":"2024-05-22T23:57:38.504206+0000","last_undegraded":"2024-05-22T23:57:38.504206+0000","last_fullsized":"2024-05-22T23:57:38.504206+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:07:55.122260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958715+0000","last_change":"2024-05-22T23:57:32.325729+0000","last_active":"2024-05-22T23:57:39.958715+0000","last_peered":"2024-05-22T23:57:39.958715+0000","last_clean":"2024-05-22T23:57:39.958715+0000","last_became_active":"2024-05-22T23:57:32.325383+0000","last_became_peered":"2024-05-22T23:57:32.325383+0000","last_unstale":"2024-05-22T23:57:39.958715+0000","last_undegraded":"2024-05-22T23:57:39.958715+0000","last_fullsized":"2024-05-22T23:57:39.958715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:57.978806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956927+0000","last_change":"2024-05-22T23:57:33.337499+0000","last_active":"2024-05-22T23:57:39.956927+0000","last_peered":"2024-05-22T23:57:39.956927+0000","last_clean":"2024-05-22T23:57:39.956927+0000","last_became_active":"2024-05-22T23:57:33.337271+0000","last_became_peered":"2024-05-22T23:57:33.337271+0000","last_unstale":"2024-05-22T23:57:39.956927+0000","last_undegraded":"2024-05-22T23:57:39.956927+0000","last_fullsized":"2024-05-22T23:57:39.956927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:32:39.133984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958026+0000","last_change":"2024-05-22T23:57:37.792707+0000","last_active":"2024-05-22T23:57:39.958026+0000","last_peered":"2024-05-22T23:57:39.958026+0000","last_clean":"2024-05-22T23:57:39.958026+0000","last_became_active":"2024-05-22T23:57:37.792209+0000","last_became_peered":"2024-05-22T23:57:37.792209+0000","last_unstale":"2024-05-22T23:57:39.958026+0000","last_undegraded":"2024-05-22T23:57:39.958026+0000","last_fullsized":"2024-05-22T23:57:39.958026+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:36:34.089424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955194+0000","last_change":"2024-05-22T23:57:32.330400+0000","last_active":"2024-05-22T23:57:39.955194+0000","last_peered":"2024-05-22T23:57:39.955194+0000","last_clean":"2024-05-22T23:57:39.955194+0000","last_became_active":"2024-05-22T23:57:32.330051+0000","last_became_peered":"2024-05-22T23:57:32.330051+0000","last_unstale":"2024-05-22T23:57:39.955194+0000","last_undegraded":"2024-05-22T23:57:39.955194+0000","last_fullsized":"2024-05-22T23:57:39.955194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:26:57.966530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956191+0000","last_change":"2024-05-22T23:57:33.328129+0000","last_active":"2024-05-22T23:57:39.956191+0000","last_peered":"2024-05-22T23:57:39.956191+0000","last_clean":"2024-05-22T23:57:39.956191+0000","last_became_active":"2024-05-22T23:57:33.327732+0000","last_became_peered":"2024-05-22T23:57:33.327732+0000","last_unstale":"2024-05-22T23:57:39.956191+0000","last_undegraded":"2024-05-22T23:57:39.956191+0000","last_fullsized":"2024-05-22T23:57:39.956191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:25:14.105235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.959156+0000","last_change":"2024-05-22T23:57:38.220884+0000","last_active":"2024-05-22T23:57:39.959156+0000","last_peered":"2024-05-22T23:57:39.959156+0000","last_clean":"2024-05-22T23:57:39.959156+0000","last_became_active":"2024-05-22T23:57:38.219365+0000","last_became_peered":"2024-05-22T23:57:38.219365+0000","last_unstale":"2024-05-22T23:57:39.959156+0000","last_undegraded":"2024-05-22T23:57:39.959156+0000","last_fullsized":"2024-05-22T23:57:39.959156+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:54:56.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954873+0000","last_change":"2024-05-22T23:57:33.328272+0000","last_active":"2024-05-22T23:57:39.954873+0000","last_peered":"2024-05-22T23:57:39.954873+0000","last_clean":"2024-05-22T23:57:39.954873+0000","last_became_active":"2024-05-22T23:57:33.327915+0000","last_became_peered":"2024-05-22T23:57:33.327915+0000","last_unstale":"2024-05-22T23:57:39.954873+0000","last_undegraded":"2024-05-22T23:57:39.954873+0000","last_fullsized":"2024-05-22T23:57:39.954873+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:59:09.085649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957387+0000","last_change":"2024-05-22T23:57:32.331979+0000","last_active":"2024-05-22T23:57:39.957387+0000","last_peered":"2024-05-22T23:57:39.957387+0000","last_clean":"2024-05-22T23:57:39.957387+0000","last_became_active":"2024-05-22T23:57:32.330983+0000","last_became_peered":"2024-05-22T23:57:32.330983+0000","last_unstale":"2024-05-22T23:57:39.957387+0000","last_undegraded":"2024-05-22T23:57:39.957387+0000","last_fullsized":"2024-05-22T23:57:39.957387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:07:19.137652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956118+0000","last_change":"2024-05-22T23:57:37.787935+0000","last_active":"2024-05-22T23:57:39.956118+0000","last_peered":"2024-05-22T23:57:39.956118+0000","last_clean":"2024-05-22T23:57:39.956118+0000","last_became_active":"2024-05-22T23:57:37.787495+0000","last_became_peered":"2024-05-22T23:57:37.787495+0000","last_unstale":"2024-05-22T23:57:39.956118+0000","last_undegraded":"2024-05-22T23:57:39.956118+0000","last_fullsized":"2024-05-22T23:57:39.956118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:37.682711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878181+0000","last_change":"2024-05-22T23:57:33.329451+0000","last_active":"2024-05-22T23:57:39.878181+0000","last_peered":"2024-05-22T23:57:39.878181+0000","last_clean":"2024-05-22T23:57:39.878181+0000","last_became_active":"2024-05-22T23:57:33.329107+0000","last_became_peered":"2024-05-22T23:57:33.329107+0000","last_unstale":"2024-05-22T23:57:39.878181+0000","last_undegraded":"2024-05-22T23:57:39.878181+0000","last_fullsized":"2024-05-22T23:57:39.878181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:44:50.198040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.247682+0000","last_change":"2024-05-22T23:57:32.321873+0000","last_active":"2024-05-22T23:58:03.247682+0000","last_peered":"2024-05-22T23:58:03.247682+0000","last_clean":"2024-05-22T23:58:03.247682+0000","last_became_active":"2024-05-22T23:57:32.321550+0000","last_became_peered":"2024-05-22T23:57:32.321550+0000","last_unstale":"2024-05-22T23:58:03.247682+0000","last_undegraded":"2024-05-22T23:58:03.247682+0000","last_fullsized":"2024-05-22T23:58:03.247682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:30:17.479238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":356,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954789+0000","last_change":"2024-05-22T23:56:11.327788+0000","last_active":"2024-05-22T23:57:39.954789+0000","last_peered":"2024-05-22T23:57:39.954789+0000","last_clean":"2024-05-22T23:57:39.954789+0000","last_became_active":"2024-05-22T23:56:11.327232+0000","last_became_peered":"2024-05-22T23:56:11.327232+0000","last_unstale":"2024-05-22T23:57:39.954789+0000","last_undegraded":"2024-05-22T23:57:39.954789+0000","last_fullsized":"2024-05-22T23:57:39.954789+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_clean_scrub_stamp":"2024-05-22T23:52:16.719643+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:26:13.077470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955505+0000","last_change":"2024-05-22T23:57:38.219326+0000","last_active":"2024-05-22T23:57:39.955505+0000","last_peered":"2024-05-22T23:57:39.955505+0000","last_clean":"2024-05-22T23:57:39.955505+0000","last_became_active":"2024-05-22T23:57:38.218835+0000","last_became_peered":"2024-05-22T23:57:38.218835+0000","last_unstale":"2024-05-22T23:57:39.955505+0000","last_undegraded":"2024-05-22T23:57:39.955505+0000","last_fullsized":"2024-05-22T23:57:39.955505+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:29.722862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957209+0000","last_change":"2024-05-22T23:57:33.316760+0000","last_active":"2024-05-22T23:57:39.957209+0000","last_peered":"2024-05-22T23:57:39.957209+0000","last_clean":"2024-05-22T23:57:39.957209+0000","last_became_active":"2024-05-22T23:57:33.316328+0000","last_became_peered":"2024-05-22T23:57:33.316328+0000","last_unstale":"2024-05-22T23:57:39.957209+0000","last_undegraded":"2024-05-22T23:57:39.957209+0000","last_fullsized":"2024-05-22T23:57:39.957209+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:53:11.592134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.254957+0000","last_change":"2024-05-22T23:57:32.319154+0000","last_active":"2024-05-22T23:58:04.254957+0000","last_peered":"2024-05-22T23:58:04.254957+0000","last_clean":"2024-05-22T23:58:04.254957+0000","last_became_active":"2024-05-22T23:57:32.318808+0000","last_became_peered":"2024-05-22T23:57:32.318808+0000","last_unstale":"2024-05-22T23:58:04.254957+0000","last_undegraded":"2024-05-22T23:58:04.254957+0000","last_fullsized":"2024-05-22T23:58:04.254957+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:37:47.271767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668241+0000","last_change":"2024-05-22T23:57:37.791464+0000","last_active":"2024-05-22T23:57:38.668241+0000","last_peered":"2024-05-22T23:57:38.668241+0000","last_clean":"2024-05-22T23:57:38.668241+0000","last_became_active":"2024-05-22T23:57:37.791044+0000","last_became_peered":"2024-05-22T23:57:37.791044+0000","last_unstale":"2024-05-22T23:57:38.668241+0000","last_undegraded":"2024-05-22T23:57:38.668241+0000","last_fullsized":"2024-05-22T23:57:38.668241+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:47:00.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.246405+0000","last_change":"2024-05-22T23:57:32.322740+0000","last_active":"2024-05-22T23:58:03.246405+0000","last_peered":"2024-05-22T23:58:03.246405+0000","last_clean":"2024-05-22T23:58:03.246405+0000","last_became_active":"2024-05-22T23:57:32.322425+0000","last_became_peered":"2024-05-22T23:57:32.322425+0000","last_unstale":"2024-05-22T23:58:03.246405+0000","last_undegraded":"2024-05-22T23:58:03.246405+0000","last_fullsized":"2024-05-22T23:58:03.246405+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:05:10.522462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958552+0000","last_change":"2024-05-22T23:57:33.324667+0000","last_active":"2024-05-22T23:57:39.958552+0000","last_peered":"2024-05-22T23:57:39.958552+0000","last_clean":"2024-05-22T23:57:39.958552+0000","last_became_active":"2024-05-22T23:57:33.324251+0000","last_became_peered":"2024-05-22T23:57:33.324251+0000","last_unstale":"2024-05-22T23:57:39.958552+0000","last_undegraded":"2024-05-22T23:57:39.958552+0000","last_fullsized":"2024-05-22T23:57:39.958552+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:49:57.203996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.877970+0000","last_change":"2024-05-22T23:57:38.222346+0000","last_active":"2024-05-22T23:57:39.877970+0000","last_peered":"2024-05-22T23:57:39.877970+0000","last_clean":"2024-05-22T23:57:39.877970+0000","last_became_active":"2024-05-22T23:57:38.220248+0000","last_became_peered":"2024-05-22T23:57:38.220248+0000","last_unstale":"2024-05-22T23:57:39.877970+0000","last_undegraded":"2024-05-22T23:57:39.877970+0000","last_fullsized":"2024-05-22T23:57:39.877970+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:26:44.026641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.260409+0000","last_change":"2024-05-22T23:57:32.336308+0000","last_active":"2024-05-22T23:58:04.260409+0000","last_peered":"2024-05-22T23:58:04.260409+0000","last_clean":"2024-05-22T23:58:04.260409+0000","last_became_active":"2024-05-22T23:57:32.334872+0000","last_became_peered":"2024-05-22T23:57:32.334872+0000","last_unstale":"2024-05-22T23:58:04.260409+0000","last_undegraded":"2024-05-22T23:58:04.260409+0000","last_fullsized":"2024-05-22T23:58:04.260409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:44:16.560930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878851+0000","last_change":"2024-05-22T23:57:33.315266+0000","last_active":"2024-05-22T23:57:39.878851+0000","last_peered":"2024-05-22T23:57:39.878851+0000","last_clean":"2024-05-22T23:57:39.878851+0000","last_became_active":"2024-05-22T23:57:33.314212+0000","last_became_peered":"2024-05-22T23:57:33.314212+0000","last_unstale":"2024-05-22T23:57:39.878851+0000","last_undegraded":"2024-05-22T23:57:39.878851+0000","last_fullsized":"2024-05-22T23:57:39.878851+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:22.896162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335426+0000","last_change":"2024-05-22T23:57:37.778524+0000","last_active":"2024-05-22T23:57:38.335426+0000","last_peered":"2024-05-22T23:57:38.335426+0000","last_clean":"2024-05-22T23:57:38.335426+0000","last_became_active":"2024-05-22T23:57:37.777827+0000","last_became_peered":"2024-05-22T23:57:37.777827+0000","last_unstale":"2024-05-22T23:57:38.335426+0000","last_undegraded":"2024-05-22T23:57:38.335426+0000","last_fullsized":"2024-05-22T23:57:38.335426+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:30:59.154921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.243324+0000","last_change":"2024-05-22T23:57:32.317649+0000","last_active":"2024-05-22T23:58:03.243324+0000","last_peered":"2024-05-22T23:58:03.243324+0000","last_clean":"2024-05-22T23:58:03.243324+0000","last_became_active":"2024-05-22T23:57:32.317280+0000","last_became_peered":"2024-05-22T23:57:32.317280+0000","last_unstale":"2024-05-22T23:58:03.243324+0000","last_undegraded":"2024-05-22T23:58:03.243324+0000","last_fullsized":"2024-05-22T23:58:03.243324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:01:16.917079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338452+0000","last_change":"2024-05-22T23:57:33.339296+0000","last_active":"2024-05-22T23:57:38.338452+0000","last_peered":"2024-05-22T23:57:38.338452+0000","last_clean":"2024-05-22T23:57:38.338452+0000","last_became_active":"2024-05-22T23:57:33.338457+0000","last_became_peered":"2024-05-22T23:57:33.338457+0000","last_unstale":"2024-05-22T23:57:38.338452+0000","last_undegraded":"2024-05-22T23:57:38.338452+0000","last_fullsized":"2024-05-22T23:57:38.338452+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:13:05.927893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336383+0000","last_change":"2024-05-22T23:57:38.218649+0000","last_active":"2024-05-22T23:57:38.336383+0000","last_peered":"2024-05-22T23:57:38.336383+0000","last_clean":"2024-05-22T23:57:38.336383+0000","last_became_active":"2024-05-22T23:57:38.218189+0000","last_became_peered":"2024-05-22T23:57:38.218189+0000","last_unstale":"2024-05-22T23:57:38.336383+0000","last_undegraded":"2024-05-22T23:57:38.336383+0000","last_fullsized":"2024-05-22T23:57:38.336383+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:30.747293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.246280+0000","last_change":"2024-05-22T23:57:32.309379+0000","last_active":"2024-05-22T23:58:03.246280+0000","last_peered":"2024-05-22T23:58:03.246280+0000","last_clean":"2024-05-22T23:58:03.246280+0000","last_became_active":"2024-05-22T23:57:32.309068+0000","last_became_peered":"2024-05-22T23:57:32.309068+0000","last_unstale":"2024-05-22T23:58:03.246280+0000","last_undegraded":"2024-05-22T23:58:03.246280+0000","last_fullsized":"2024-05-22T23:58:03.246280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:43:54.787996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.503994+0000","last_change":"2024-05-22T23:57:33.320253+0000","last_active":"2024-05-22T23:57:38.503994+0000","last_peered":"2024-05-22T23:57:38.503994+0000","last_clean":"2024-05-22T23:57:38.503994+0000","last_became_active":"2024-05-22T23:57:33.319604+0000","last_became_peered":"2024-05-22T23:57:33.319604+0000","last_unstale":"2024-05-22T23:57:38.503994+0000","last_undegraded":"2024-05-22T23:57:38.503994+0000","last_fullsized":"2024-05-22T23:57:38.503994+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:23:51.745040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.960879+0000","last_change":"2024-05-22T23:57:37.795368+0000","last_active":"2024-05-22T23:57:39.960879+0000","last_peered":"2024-05-22T23:57:39.960879+0000","last_clean":"2024-05-22T23:57:39.960879+0000","last_became_active":"2024-05-22T23:57:37.792848+0000","last_became_peered":"2024-05-22T23:57:37.792848+0000","last_unstale":"2024-05-22T23:57:39.960879+0000","last_undegraded":"2024-05-22T23:57:39.960879+0000","last_fullsized":"2024-05-22T23:57:39.960879+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:45.859867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.247516+0000","last_change":"2024-05-22T23:57:32.329147+0000","last_active":"2024-05-22T23:58:03.247516+0000","last_peered":"2024-05-22T23:58:03.247516+0000","last_clean":"2024-05-22T23:58:03.247516+0000","last_became_active":"2024-05-22T23:57:32.327254+0000","last_became_peered":"2024-05-22T23:57:32.327254+0000","last_unstale":"2024-05-22T23:58:03.247516+0000","last_undegraded":"2024-05-22T23:58:03.247516+0000","last_fullsized":"2024-05-22T23:58:03.247516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:55:24.915817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.667879+0000","last_change":"2024-05-22T23:57:33.332843+0000","last_active":"2024-05-22T23:57:38.667879+0000","last_peered":"2024-05-22T23:57:38.667879+0000","last_clean":"2024-05-22T23:57:38.667879+0000","last_became_active":"2024-05-22T23:57:33.332497+0000","last_became_peered":"2024-05-22T23:57:33.332497+0000","last_unstale":"2024-05-22T23:57:38.667879+0000","last_undegraded":"2024-05-22T23:57:38.667879+0000","last_fullsized":"2024-05-22T23:57:38.667879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:55:16.109676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336600+0000","last_change":"2024-05-22T23:57:37.787609+0000","last_active":"2024-05-22T23:57:38.336600+0000","last_peered":"2024-05-22T23:57:38.336600+0000","last_clean":"2024-05-22T23:57:38.336600+0000","last_became_active":"2024-05-22T23:57:37.787153+0000","last_became_peered":"2024-05-22T23:57:37.787153+0000","last_unstale":"2024-05-22T23:57:38.336600+0000","last_undegraded":"2024-05-22T23:57:38.336600+0000","last_fullsized":"2024-05-22T23:57:38.336600+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:57:51.640037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335396+0000","last_change":"2024-05-22T23:57:32.329784+0000","last_active":"2024-05-22T23:57:38.335396+0000","last_peered":"2024-05-22T23:57:38.335396+0000","last_clean":"2024-05-22T23:57:38.335396+0000","last_became_active":"2024-05-22T23:57:32.329393+0000","last_became_peered":"2024-05-22T23:57:32.329393+0000","last_unstale":"2024-05-22T23:57:38.335396+0000","last_undegraded":"2024-05-22T23:57:38.335396+0000","last_fullsized":"2024-05-22T23:57:38.335396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:17:54.811962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878789+0000","last_change":"2024-05-22T23:57:33.318803+0000","last_active":"2024-05-22T23:57:39.878789+0000","last_peered":"2024-05-22T23:57:39.878789+0000","last_clean":"2024-05-22T23:57:39.878789+0000","last_became_active":"2024-05-22T23:57:33.318170+0000","last_became_peered":"2024-05-22T23:57:33.318170+0000","last_unstale":"2024-05-22T23:57:39.878789+0000","last_undegraded":"2024-05-22T23:57:39.878789+0000","last_fullsized":"2024-05-22T23:57:39.878789+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:56:56.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.336577+0000","last_change":"2024-05-22T23:57:38.218583+0000","last_active":"2024-05-22T23:57:38.336577+0000","last_peered":"2024-05-22T23:57:38.336577+0000","last_clean":"2024-05-22T23:57:38.336577+0000","last_became_active":"2024-05-22T23:57:38.218048+0000","last_became_peered":"2024-05-22T23:57:38.218048+0000","last_unstale":"2024-05-22T23:57:38.336577+0000","last_undegraded":"2024-05-22T23:57:38.336577+0000","last_fullsized":"2024-05-22T23:57:38.336577+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:10:57.579656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:03.242063+0000","last_change":"2024-05-22T23:57:32.317908+0000","last_active":"2024-05-22T23:58:03.242063+0000","last_peered":"2024-05-22T23:58:03.242063+0000","last_clean":"2024-05-22T23:58:03.242063+0000","last_became_active":"2024-05-22T23:57:32.317580+0000","last_became_peered":"2024-05-22T23:57:32.317580+0000","last_unstale":"2024-05-22T23:58:03.242063+0000","last_undegraded":"2024-05-22T23:58:03.242063+0000","last_fullsized":"2024-05-22T23:58:03.242063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:50:02.044817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955261+0000","last_change":"2024-05-22T23:57:33.321184+0000","last_active":"2024-05-22T23:57:39.955261+0000","last_peered":"2024-05-22T23:57:39.955261+0000","last_clean":"2024-05-22T23:57:39.955261+0000","last_became_active":"2024-05-22T23:57:33.320418+0000","last_became_peered":"2024-05-22T23:57:33.320418+0000","last_unstale":"2024-05-22T23:57:39.955261+0000","last_undegraded":"2024-05-22T23:57:39.955261+0000","last_fullsized":"2024-05-22T23:57:39.955261+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:22:43.019809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954795+0000","last_change":"2024-05-22T23:57:37.795718+0000","last_active":"2024-05-22T23:57:39.954795+0000","last_peered":"2024-05-22T23:57:39.954795+0000","last_clean":"2024-05-22T23:57:39.954795+0000","last_became_active":"2024-05-22T23:57:37.795297+0000","last_became_peered":"2024-05-22T23:57:37.795297+0000","last_unstale":"2024-05-22T23:57:39.954795+0000","last_undegraded":"2024-05-22T23:57:39.954795+0000","last_fullsized":"2024-05-22T23:57:39.954795+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:56:47.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:58:04.257734+0000","last_change":"2024-05-22T23:57:32.311921+0000","last_active":"2024-05-22T23:58:04.257734+0000","last_peered":"2024-05-22T23:58:04.257734+0000","last_clean":"2024-05-22T23:58:04.257734+0000","last_became_active":"2024-05-22T23:57:32.311600+0000","last_became_peered":"2024-05-22T23:57:32.311600+0000","last_unstale":"2024-05-22T23:58:04.257734+0000","last_undegraded":"2024-05-22T23:58:04.257734+0000","last_fullsized":"2024-05-22T23:58:04.257734+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:35:21.882773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.956456+0000","last_change":"2024-05-22T23:57:33.321289+0000","last_active":"2024-05-22T23:57:39.956456+0000","last_peered":"2024-05-22T23:57:39.956456+0000","last_clean":"2024-05-22T23:57:39.956456+0000","last_became_active":"2024-05-22T23:57:33.320458+0000","last_became_peered":"2024-05-22T23:57:33.320458+0000","last_unstale":"2024-05-22T23:57:39.956456+0000","last_undegraded":"2024-05-22T23:57:39.956456+0000","last_fullsized":"2024-05-22T23:57:39.956456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:02:05.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338005+0000","last_change":"2024-05-22T23:57:37.786062+0000","last_active":"2024-05-22T23:57:38.338005+0000","last_peered":"2024-05-22T23:57:38.338005+0000","last_clean":"2024-05-22T23:57:38.338005+0000","last_became_active":"2024-05-22T23:57:37.785624+0000","last_became_peered":"2024-05-22T23:57:37.785624+0000","last_unstale":"2024-05-22T23:57:38.338005+0000","last_undegraded":"2024-05-22T23:57:38.338005+0000","last_fullsized":"2024-05-22T23:57:38.338005+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:38:56.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955273+0000","last_change":"2024-05-22T23:57:32.333909+0000","last_active":"2024-05-22T23:57:39.955273+0000","last_peered":"2024-05-22T23:57:39.955273+0000","last_clean":"2024-05-22T23:57:39.955273+0000","last_became_active":"2024-05-22T23:57:32.333531+0000","last_became_peered":"2024-05-22T23:57:32.333531+0000","last_unstale":"2024-05-22T23:57:39.955273+0000","last_undegraded":"2024-05-22T23:57:39.955273+0000","last_fullsized":"2024-05-22T23:57:39.955273+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:53:21.843150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955703+0000","last_change":"2024-05-22T23:57:33.334366+0000","last_active":"2024-05-22T23:57:39.955703+0000","last_peered":"2024-05-22T23:57:39.955703+0000","last_clean":"2024-05-22T23:57:39.955703+0000","last_became_active":"2024-05-22T23:57:33.334037+0000","last_became_peered":"2024-05-22T23:57:33.334037+0000","last_unstale":"2024-05-22T23:57:39.955703+0000","last_undegraded":"2024-05-22T23:57:39.955703+0000","last_fullsized":"2024-05-22T23:57:39.955703+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:15:10.631087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954652+0000","last_change":"2024-05-22T23:57:37.780241+0000","last_active":"2024-05-22T23:57:39.954652+0000","last_peered":"2024-05-22T23:57:39.954652+0000","last_clean":"2024-05-22T23:57:39.954652+0000","last_became_active":"2024-05-22T23:57:37.779816+0000","last_became_peered":"2024-05-22T23:57:37.779816+0000","last_unstale":"2024-05-22T23:57:39.954652+0000","last_undegraded":"2024-05-22T23:57:39.954652+0000","last_fullsized":"2024-05-22T23:57:39.954652+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:11.397681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.959726+0000","last_change":"2024-05-22T23:57:32.334829+0000","last_active":"2024-05-22T23:57:39.959726+0000","last_peered":"2024-05-22T23:57:39.959726+0000","last_clean":"2024-05-22T23:57:39.959726+0000","last_became_active":"2024-05-22T23:57:32.334368+0000","last_became_peered":"2024-05-22T23:57:32.334368+0000","last_unstale":"2024-05-22T23:57:39.959726+0000","last_undegraded":"2024-05-22T23:57:39.959726+0000","last_fullsized":"2024-05-22T23:57:39.959726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:17:42.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955807+0000","last_change":"2024-05-22T23:57:33.318740+0000","last_active":"2024-05-22T23:57:39.955807+0000","last_peered":"2024-05-22T23:57:39.955807+0000","last_clean":"2024-05-22T23:57:39.955807+0000","last_became_active":"2024-05-22T23:57:33.318363+0000","last_became_peered":"2024-05-22T23:57:33.318363+0000","last_unstale":"2024-05-22T23:57:39.955807+0000","last_undegraded":"2024-05-22T23:57:39.955807+0000","last_fullsized":"2024-05-22T23:57:39.955807+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:51:42.319667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668014+0000","last_change":"2024-05-22T23:57:33.325582+0000","last_active":"2024-05-22T23:57:38.668014+0000","last_peered":"2024-05-22T23:57:38.668014+0000","last_clean":"2024-05-22T23:57:38.668014+0000","last_became_active":"2024-05-22T23:57:33.325268+0000","last_became_peered":"2024-05-22T23:57:33.325268+0000","last_unstale":"2024-05-22T23:57:38.668014+0000","last_undegraded":"2024-05-22T23:57:38.668014+0000","last_fullsized":"2024-05-22T23:57:38.668014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:02:18.968634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335721+0000","last_change":"2024-05-22T23:57:33.325875+0000","last_active":"2024-05-22T23:57:38.335721+0000","last_peered":"2024-05-22T23:57:38.335721+0000","last_clean":"2024-05-22T23:57:38.335721+0000","last_became_active":"2024-05-22T23:57:33.325540+0000","last_became_peered":"2024-05-22T23:57:33.325540+0000","last_unstale":"2024-05-22T23:57:38.335721+0000","last_undegraded":"2024-05-22T23:57:38.335721+0000","last_fullsized":"2024-05-22T23:57:38.335721+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:59:42.817107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335334+0000","last_change":"2024-05-22T23:57:33.332935+0000","last_active":"2024-05-22T23:57:38.335334+0000","last_peered":"2024-05-22T23:57:38.335334+0000","last_clean":"2024-05-22T23:57:38.335334+0000","last_became_active":"2024-05-22T23:57:33.332607+0000","last_became_peered":"2024-05-22T23:57:33.332607+0000","last_unstale":"2024-05-22T23:57:38.335334+0000","last_undegraded":"2024-05-22T23:57:38.335334+0000","last_fullsized":"2024-05-22T23:57:38.335334+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:09.589896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.338514+0000","last_change":"2024-05-22T23:57:33.333269+0000","last_active":"2024-05-22T23:57:38.338514+0000","last_peered":"2024-05-22T23:57:38.338514+0000","last_clean":"2024-05-22T23:57:38.338514+0000","last_became_active":"2024-05-22T23:57:33.329017+0000","last_became_peered":"2024-05-22T23:57:33.329017+0000","last_unstale":"2024-05-22T23:57:38.338514+0000","last_undegraded":"2024-05-22T23:57:38.338514+0000","last_fullsized":"2024-05-22T23:57:38.338514+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:35:42.156434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.958815+0000","last_change":"2024-05-22T23:57:33.328721+0000","last_active":"2024-05-22T23:57:39.958815+0000","last_peered":"2024-05-22T23:57:39.958815+0000","last_clean":"2024-05-22T23:57:39.958815+0000","last_became_active":"2024-05-22T23:57:33.328318+0000","last_became_peered":"2024-05-22T23:57:33.328318+0000","last_unstale":"2024-05-22T23:57:39.958815+0000","last_undegraded":"2024-05-22T23:57:39.958815+0000","last_fullsized":"2024-05-22T23:57:39.958815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:14.772688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668343+0000","last_change":"2024-05-22T23:57:33.324777+0000","last_active":"2024-05-22T23:57:38.668343+0000","last_peered":"2024-05-22T23:57:38.668343+0000","last_clean":"2024-05-22T23:57:38.668343+0000","last_became_active":"2024-05-22T23:57:33.324446+0000","last_became_peered":"2024-05-22T23:57:33.324446+0000","last_unstale":"2024-05-22T23:57:38.668343+0000","last_undegraded":"2024-05-22T23:57:38.668343+0000","last_fullsized":"2024-05-22T23:57:38.668343+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:20:42.327094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.335801+0000","last_change":"2024-05-22T23:57:33.318783+0000","last_active":"2024-05-22T23:57:38.335801+0000","last_peered":"2024-05-22T23:57:38.335801+0000","last_clean":"2024-05-22T23:57:38.335801+0000","last_became_active":"2024-05-22T23:57:33.318481+0000","last_became_peered":"2024-05-22T23:57:33.318481+0000","last_unstale":"2024-05-22T23:57:38.335801+0000","last_undegraded":"2024-05-22T23:57:38.335801+0000","last_fullsized":"2024-05-22T23:57:38.335801+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:14:35.745683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878949+0000","last_change":"2024-05-22T23:57:33.319939+0000","last_active":"2024-05-22T23:57:39.878949+0000","last_peered":"2024-05-22T23:57:39.878949+0000","last_clean":"2024-05-22T23:57:39.878949+0000","last_became_active":"2024-05-22T23:57:33.316029+0000","last_became_peered":"2024-05-22T23:57:33.316029+0000","last_unstale":"2024-05-22T23:57:39.878949+0000","last_undegraded":"2024-05-22T23:57:39.878949+0000","last_fullsized":"2024-05-22T23:57:39.878949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:46:34.383953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.954758+0000","last_change":"2024-05-22T23:57:33.334625+0000","last_active":"2024-05-22T23:57:39.954758+0000","last_peered":"2024-05-22T23:57:39.954758+0000","last_clean":"2024-05-22T23:57:39.954758+0000","last_became_active":"2024-05-22T23:57:33.334341+0000","last_became_peered":"2024-05-22T23:57:33.334341+0000","last_unstale":"2024-05-22T23:57:39.954758+0000","last_undegraded":"2024-05-22T23:57:39.954758+0000","last_fullsized":"2024-05-22T23:57:39.954758+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:38.319721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.878077+0000","last_change":"2024-05-22T23:57:33.330958+0000","last_active":"2024-05-22T23:57:39.878077+0000","last_peered":"2024-05-22T23:57:39.878077+0000","last_clean":"2024-05-22T23:57:39.878077+0000","last_became_active":"2024-05-22T23:57:33.317355+0000","last_became_peered":"2024-05-22T23:57:33.317355+0000","last_unstale":"2024-05-22T23:57:39.878077+0000","last_undegraded":"2024-05-22T23:57:39.878077+0000","last_fullsized":"2024-05-22T23:57:39.878077+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:49:43.784780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.955348+0000","last_change":"2024-05-22T23:57:33.319530+0000","last_active":"2024-05-22T23:57:39.955348+0000","last_peered":"2024-05-22T23:57:39.955348+0000","last_clean":"2024-05-22T23:57:39.955348+0000","last_became_active":"2024-05-22T23:57:33.319122+0000","last_became_peered":"2024-05-22T23:57:33.319122+0000","last_unstale":"2024-05-22T23:57:39.955348+0000","last_undegraded":"2024-05-22T23:57:39.955348+0000","last_fullsized":"2024-05-22T23:57:39.955348+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:31:26.937804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:39.957776+0000","last_change":"2024-05-22T23:57:33.323722+0000","last_active":"2024-05-22T23:57:39.957776+0000","last_peered":"2024-05-22T23:57:39.957776+0000","last_clean":"2024-05-22T23:57:39.957776+0000","last_became_active":"2024-05-22T23:57:33.323343+0000","last_became_peered":"2024-05-22T23:57:33.323343+0000","last_unstale":"2024-05-22T23:57:39.957776+0000","last_undegraded":"2024-05-22T23:57:39.957776+0000","last_fullsized":"2024-05-22T23:57:39.957776+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:24:59.575536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-22T23:57:38.668203+0000","last_change":"2024-05-22T23:57:33.320925+0000","last_active":"2024-05-22T23:57:38.668203+0000","last_peered":"2024-05-22T23:57:38.668203+0000","last_clean":"2024-05-22T23:57:38.668203+0000","last_became_active":"2024-05-22T23:57:33.320570+0000","last_became_peered":"2024-05-22T23:57:33.320570+0000","last_unstale":"2024-05-22T23:57:38.668203+0000","last_undegraded":"2024-05-22T23:57:38.668203+0000","last_fullsized":"2024-05-22T23:57:38.668203+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:38:27.659025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244946,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":43989,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":19,"num_read_kb":53,"num_write":137,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":79359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318292,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":485342,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579930,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1114112,"data_stored":1117565,"data_compressed":50343,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":489674,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005093,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":485498,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266732,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1052672,"data_stored":1081796,"data_compressed":50343,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430255,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":484760,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":488601,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":485454,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":489951,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":485044,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888124,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":484782,"data_compressed":14647,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691896,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":1076604,"data_compressed":50343,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":23139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":42437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-22T23:58:09.662 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-22T23:58:09.663 INFO:tasks.cephadm:Setting up client nodes... 2024-05-22T23:58:09.663 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-22T23:58:10.140 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:10.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:10 smithi105 ceph-mon[36426]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:58:10.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:10 smithi105 ceph-mon[36426]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.9 KiB/s wr, 11 op/s 2024-05-22T23:58:10.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:10 smithi103 ceph-mon[31370]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:58:10.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:10 smithi103 ceph-mon[31370]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.9 KiB/s wr, 11 op/s 2024-05-22T23:58:10.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:10 smithi174 ceph-mon[36179]: from='client.14769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:58:10.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:10 smithi174 ceph-mon[36179]: pgmap v341: 129 pgs: 129 active+clean; 621 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.9 KiB/s wr, 11 op/s 2024-05-22T23:58:12.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.409+0000 7f3aaf000700 1 -- 172.21.15.103:0/1015639922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8102410 msgr2=0x7f3aa8102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:12.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.409+0000 7f3aaf000700 1 --2- 172.21.15.103:0/1015639922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8102410 0x7f3aa8102830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3a980099c0 tx=0x7f3a98009cd0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.409+0000 7f3aaf000700 1 -- 172.21.15.103:0/1015639922 shutdown_connections 2024-05-22T23:58:12.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.409+0000 7f3aaf000700 1 --2- 172.21.15.103:0/1015639922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aa8103fb0 0x7f3aa810a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.413 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.409+0000 7f3aaf000700 1 --2- 172.21.15.103:0/1015639922 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8102410 0x7f3aa8102830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.409+0000 7f3aaf000700 1 --2- 172.21.15.103:0/1015639922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3aa8103610 0x7f3aa8103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.409+0000 7f3aaf000700 1 -- 172.21.15.103:0/1015639922 >> 172.21.15.103:0/1015639922 conn(0x7f3aa80fd9b0 msgr2=0x7f3aa80ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:12.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.410+0000 7f3aaf000700 1 -- 172.21.15.103:0/1015639922 shutdown_connections 2024-05-22T23:58:12.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.410+0000 7f3aaf000700 1 -- 172.21.15.103:0/1015639922 wait complete. 2024-05-22T23:58:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.411+0000 7f3aaf000700 1 Processor -- start 2024-05-22T23:58:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.411+0000 7f3aaf000700 1 -- start start 2024-05-22T23:58:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3aaf000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aa8102410 0x7f3aa8076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3aaf000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8103610 0x7f3aa8074750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3aaf000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3aa8103fb0 0x7f3aa8074c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3aaf000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa810e050 con 0x7f3aa8103610 2024-05-22T23:58:12.415 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3aaf000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa810ded0 con 0x7f3aa8103fb0 2024-05-22T23:58:12.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3aaf000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa810e1d0 con 0x7f3aa8102410 2024-05-22T23:58:12.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3a9f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8103610 0x7f3aa8074750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:12.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3aa7e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aa8102410 0x7f3aa8076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:12.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3a9f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8103610 0x7f3aa8074750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58574/0 (socket says 172.21.15.103:58574) 2024-05-22T23:58:12.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3a9f400700 1 -- 172.21.15.103:0/2020557048 learned_addr learned my addr 172.21.15.103:0/2020557048 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:12.416 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.412+0000 7f3aad000700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3aa8103fb0 0x7f3aa8074c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aa7e00700 1 -- 172.21.15.103:0/2020557048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3aa8103fb0 msgr2=0x7f3aa8074c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aa7e00700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3aa8103fb0 0x7f3aa8074c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aa7e00700 1 -- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8103610 msgr2=0x7f3aa8074750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aa7e00700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8103610 0x7f3aa8074750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aa7e00700 1 -- 172.21.15.103:0/2020557048 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa8075540 con 0x7f3aa8102410 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3a9f400700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8103610 0x7f3aa8074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aad000700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3aa8103fb0 0x7f3aa8074c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aa7e00700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aa8102410 0x7f3aa8076100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3a980098e0 tx=0x7f3a98009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aa5600700 1 -- 172.21.15.103:0/2020557048 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98027070 con 0x7f3aa8102410 2024-05-22T23:58:12.417 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aa5600700 1 -- 172.21.15.103:0/2020557048 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a98022470 con 0x7f3aa8102410 2024-05-22T23:58:12.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.413+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa81a26a0 con 0x7f3aa8102410 2024-05-22T23:58:12.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.414+0000 7f3aa5600700 1 -- 172.21.15.103:0/2020557048 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a98009e60 con 0x7f3aa8102410 2024-05-22T23:58:12.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.414+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa81a2af0 con 0x7f3aa8102410 2024-05-22T23:58:12.418 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.415+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa80508d0 con 0x7f3aa8102410 2024-05-22T23:58:12.419 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.417+0000 7f3aa5600700 1 -- 172.21.15.103:0/2020557048 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f3a9802a070 con 0x7f3aa8102410 2024-05-22T23:58:12.421 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.418+0000 7f3aa5600700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3a8c0a2de0 0x7f3a8c0a52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:12.421 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.418+0000 7f3aa5600700 1 -- 172.21.15.103:0/2020557048 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f3a980d02d0 con 0x7f3aa8102410 2024-05-22T23:58:12.421 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.418+0000 7f3a9f400700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3a8c0a2de0 0x7f3a8c0a52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:12.422 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.419+0000 7f3a9f400700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3a8c0a2de0 0x7f3a8c0a52a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a90011550 tx=0x7f3a90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:12.423 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.420+0000 7f3aa5600700 1 -- 172.21.15.103:0/2020557048 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3a98013070 con 0x7f3aa8102410 2024-05-22T23:58:12.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:12 smithi105 ceph-mon[36426]: pgmap v342: 129 pgs: 129 active+clean; 621 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-05-22T23:58:12.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:12 smithi103 ceph-mon[31370]: pgmap v342: 129 pgs: 129 active+clean; 621 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-05-22T23:58:12.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:12 smithi174 ceph-mon[36179]: pgmap v342: 129 pgs: 129 active+clean; 621 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-05-22T23:58:12.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.734+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3aa802ddf0 con 0x7f3aa8102410 2024-05-22T23:58:12.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.741+0000 7f3aa5600700 1 -- 172.21.15.103:0/2020557048 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f3a9809cec0 con 0x7f3aa8102410 2024-05-22T23:58:12.744 INFO:teuthology.orchestra.run.smithi103.stdout:[client.0] 2024-05-22T23:58:12.745 INFO:teuthology.orchestra.run.smithi103.stdout: key = AQCUhk5m2pHjKxAAQSN8ePG852R/SrZ92qzVhg== 2024-05-22T23:58:12.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.744+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3a8c0a2de0 msgr2=0x7f3a8c0a52a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.744+0000 7f3aaf000700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3a8c0a2de0 0x7f3a8c0a52a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a90011550 tx=0x7f3a90009260 comp rx=0 tx=0).stop 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.744+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aa8102410 msgr2=0x7f3aa8076100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.744+0000 7f3aaf000700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aa8102410 0x7f3aa8076100 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3a980098e0 tx=0x7f3a98009580 comp rx=0 tx=0).stop 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.744+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 shutdown_connections 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.744+0000 7f3aaf000700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3aa8103fb0 0x7f3aa8074c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.745+0000 7f3aaf000700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f3a8c0a2de0 0x7f3a8c0a52a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.745+0000 7f3aaf000700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aa8102410 0x7f3aa8076100 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.745+0000 7f3aaf000700 1 --2- 172.21.15.103:0/2020557048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3aa8103610 0x7f3aa8074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.745+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 >> 172.21.15.103:0/2020557048 conn(0x7f3aa80fd9b0 msgr2=0x7f3aa81068b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.745+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 shutdown_connections 2024-05-22T23:58:12.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:12.745+0000 7f3aaf000700 1 -- 172.21.15.103:0/2020557048 wait complete. 2024-05-22T23:58:13.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:13 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2020557048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-22T23:58:13.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:13 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-22T23:58:13.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:13 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-22T23:58:13.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:13 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2020557048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-22T23:58:13.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:13 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-22T23:58:13.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:13 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-22T23:58:13.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:13 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2020557048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-22T23:58:13.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:13 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-22T23:58:13.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:13 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-22T23:58:13.679 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-22T23:58:13.679 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-22T23:58:13.679 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-22T23:58:13.808 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-22T23:58:13.809 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-22T23:58:13.809 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph mgr dump --format=json 2024-05-22T23:58:14.287 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:14.467 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:14 smithi103 ceph-mon[31370]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-05-22T23:58:14.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:14 smithi105 ceph-mon[36426]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-05-22T23:58:14.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:14 smithi174 ceph-mon[36179]: pgmap v343: 129 pgs: 129 active+clean; 629 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 14 op/s 2024-05-22T23:58:16.309 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:16 smithi103 ceph-mon[31370]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-05-22T23:58:16.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:16 smithi105 ceph-mon[36426]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-05-22T23:58:16.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.604+0000 7ff402400700 1 -- 172.21.15.103:0/1337774595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 msgr2=0x7ff3fc06aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:16.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.604+0000 7ff402400700 1 --2- 172.21.15.103:0/1337774595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 0x7ff3fc06aa30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec009990 tx=0x7ff3ec009ca0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.605+0000 7ff402400700 1 -- 172.21.15.103:0/1337774595 shutdown_connections 2024-05-22T23:58:16.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.605+0000 7ff402400700 1 --2- 172.21.15.103:0/1337774595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3fc10ecb0 0x7ff3fc1110a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.605+0000 7ff402400700 1 --2- 172.21.15.103:0/1337774595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 0x7ff3fc06aa30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.605+0000 7ff402400700 1 --2- 172.21.15.103:0/1337774595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3fc06af70 0x7ff3fc06b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.605+0000 7ff402400700 1 -- 172.21.15.103:0/1337774595 >> 172.21.15.103:0/1337774595 conn(0x7ff3fc078210 msgr2=0x7ff3fc078620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:16.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.605+0000 7ff402400700 1 -- 172.21.15.103:0/1337774595 shutdown_connections 2024-05-22T23:58:16.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.605+0000 7ff402400700 1 -- 172.21.15.103:0/1337774595 wait complete. 2024-05-22T23:58:16.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.606+0000 7ff402400700 1 Processor -- start 2024-05-22T23:58:16.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.606+0000 7ff402400700 1 -- start start 2024-05-22T23:58:16.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff402400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3fc06af70 0x7ff3fc19ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:16.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff402400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 0x7ff3fc19b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:16.609 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff402400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3fc10ecb0 0x7ff3fc19f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:16.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff402400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc1148f0 con 0x7ff3fc103270 2024-05-22T23:58:16.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff402400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc114770 con 0x7ff3fc06af70 2024-05-22T23:58:16.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff402400700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff3fc114a70 con 0x7ff3fc10ecb0 2024-05-22T23:58:16.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3faa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 0x7ff3fc19b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:16.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3faa00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 0x7ff3fc19b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51930/0 (socket says 172.21.15.103:51930) 2024-05-22T23:58:16.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3faa00700 1 -- 172.21.15.103:0/3318345979 learned_addr learned my addr 172.21.15.103:0/3318345979 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:16.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3fbe00700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3fc10ecb0 0x7ff3fc19f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:16.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3fb400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3fc06af70 0x7ff3fc19ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:16.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3faa00700 1 -- 172.21.15.103:0/3318345979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3fc10ecb0 msgr2=0x7ff3fc19f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:16.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3faa00700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3fc10ecb0 0x7ff3fc19f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.611 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3faa00700 1 -- 172.21.15.103:0/3318345979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3fc06af70 msgr2=0x7ff3fc19ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3faa00700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3fc06af70 0x7ff3fc19ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.607+0000 7ff3faa00700 1 -- 172.21.15.103:0/3318345979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3fc19fd70 con 0x7ff3fc103270 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.608+0000 7ff3faa00700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 0x7ff3fc19b2a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800eb30 tx=0x7ff3e800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.608+0000 7ff3f3e00700 1 -- 172.21.15.103:0/3318345979 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e800cbd0 con 0x7ff3fc103270 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.608+0000 7ff3f3e00700 1 -- 172.21.15.103:0/3318345979 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3e8004500 con 0x7ff3fc103270 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.608+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3fc1a0060 con 0x7ff3fc103270 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.608+0000 7ff3f3e00700 1 -- 172.21.15.103:0/3318345979 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8010430 con 0x7ff3fc103270 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.608+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3fc1a78e0 con 0x7ff3fc103270 2024-05-22T23:58:16.612 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.609+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3fc1018a0 con 0x7ff3fc103270 2024-05-22T23:58:16.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.612+0000 7ff3f3e00700 1 -- 172.21.15.103:0/3318345979 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff3e8003f00 con 0x7ff3fc103270 2024-05-22T23:58:16.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.613+0000 7ff3f3e00700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff3e00a2cb0 0x7ff3e00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:16.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.614+0000 7ff3f3e00700 1 -- 172.21.15.103:0/3318345979 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7ff3e80cf7d0 con 0x7ff3fc103270 2024-05-22T23:58:16.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.614+0000 7ff3fb400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff3e00a2cb0 0x7ff3e00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:16.617 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.614+0000 7ff3fb400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff3e00a2cb0 0x7ff3e00a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec006010 tx=0x7ff3ec00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:16.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:16 smithi174 ceph-mon[36179]: pgmap v344: 129 pgs: 129 active+clean; 629 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2024-05-22T23:58:16.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.618+0000 7ff3f3e00700 1 -- 172.21.15.103:0/3318345979 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff3e809c330 con 0x7ff3fc103270 2024-05-22T23:58:16.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.906+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7ff3fc069050 con 0x7ff3fc103270 2024-05-22T23:58:16.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.924+0000 7ff3f3e00700 1 -- 172.21.15.103:0/3318345979 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263208 (secure 0 0 0) 0x7ff3fc069050 con 0x7ff3fc103270 2024-05-22T23:58:16.928 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:16.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.930+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff3e00a2cb0 msgr2=0x7ff3e00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:16.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.930+0000 7ff402400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff3e00a2cb0 0x7ff3e00a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff3ec006010 tx=0x7ff3ec00b260 comp rx=0 tx=0).stop 2024-05-22T23:58:16.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.930+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 msgr2=0x7ff3fc19b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:16.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.930+0000 7ff402400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 0x7ff3fc19b2a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff3e800eb30 tx=0x7ff3e800ee40 comp rx=0 tx=0).stop 2024-05-22T23:58:16.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.931+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 shutdown_connections 2024-05-22T23:58:16.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.931+0000 7ff402400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3fc06af70 0x7ff3fc19ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.931+0000 7ff402400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff3e00a2cb0 0x7ff3e00a5170 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.931+0000 7ff402400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3fc10ecb0 0x7ff3fc19f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.931+0000 7ff402400700 1 --2- 172.21.15.103:0/3318345979 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff3fc103270 0x7ff3fc19b2a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:16.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.931+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 >> 172.21.15.103:0/3318345979 conn(0x7ff3fc078210 msgr2=0x7ff3fc100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:16.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.931+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 shutdown_connections 2024-05-22T23:58:16.935 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:16.931+0000 7ff402400700 1 -- 172.21.15.103:0/3318345979 wait complete. 2024-05-22T23:58:17.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:17 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3318345979' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-22T23:58:17.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:17 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3318345979' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-22T23:58:17.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:17 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3318345979' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-22T23:58:17.830 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":1744216601},{"type":"v1","addr":"172.21.15.103:6801","nonce":1744216601}]},"active_addr":"172.21.15.103:6801/1744216601","active_change":"2024-05-22T23:49:29.953378+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.103:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3827978298}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":284924627}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":2531882208}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":3929139004}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.103:0","nonce":874212393}]}]} 2024-05-22T23:58:17.839 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-22T23:58:17.839 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-22T23:58:17.840 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-22T23:58:18.327 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:18.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:18 smithi103 ceph-mon[31370]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-05-22T23:58:18.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:18 smithi105 ceph-mon[36426]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-05-22T23:58:18.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:18 smithi174 ceph-mon[36179]: pgmap v345: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-05-22T23:58:20.303 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:20 smithi103 ceph-mon[31370]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-05-22T23:58:20.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:20 smithi105 ceph-mon[36426]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-05-22T23:58:20.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:20 smithi174 ceph-mon[36179]: pgmap v346: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.619+0000 7f583b600700 1 -- 172.21.15.103:0/3270560067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58341072d0 msgr2=0x7f58341076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.619+0000 7f583b600700 1 --2- 172.21.15.103:0/3270560067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58341072d0 0x7f58341076b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f58280098f0 tx=0x7f5828009c00 comp rx=0 tx=0).stop 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.619+0000 7f583b600700 1 -- 172.21.15.103:0/3270560067 shutdown_connections 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.619+0000 7f583b600700 1 --2- 172.21.15.103:0/3270560067 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58341072d0 0x7f58341076b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.619+0000 7f583b600700 1 --2- 172.21.15.103:0/3270560067 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5834101c20 0x7f583410e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.619+0000 7f583b600700 1 --2- 172.21.15.103:0/3270560067 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5834101260 0x7f58341016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.619+0000 7f583b600700 1 -- 172.21.15.103:0/3270560067 >> 172.21.15.103:0/3270560067 conn(0x7f5834078210 msgr2=0x7f5834078620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.620+0000 7f583b600700 1 -- 172.21.15.103:0/3270560067 shutdown_connections 2024-05-22T23:58:20.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.620+0000 7f583b600700 1 -- 172.21.15.103:0/3270560067 wait complete. 2024-05-22T23:58:20.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.621+0000 7f583b600700 1 Processor -- start 2024-05-22T23:58:20.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.621+0000 7f583b600700 1 -- start start 2024-05-22T23:58:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.621+0000 7f583b600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5834101260 0x7f5834102f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.621+0000 7f583b600700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5834101c20 0x7f5834103480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.621+0000 7f583b600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58341072d0 0x7f58341039c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.621+0000 7f583b600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5834111590 con 0x7f5834101260 2024-05-22T23:58:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.621+0000 7f583b600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5834111410 con 0x7f58341072d0 2024-05-22T23:58:20.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.621+0000 7f583b600700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5834111710 con 0x7f5834101c20 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5838c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5834101260 0x7f5834102f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5838c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5834101260 0x7f5834102f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51946/0 (socket says 172.21.15.103:51946) 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5838c00700 1 -- 172.21.15.103:0/1872790243 learned_addr learned my addr 172.21.15.103:0/1872790243 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5833e00700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5834101c20 0x7f5834103480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5839600700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58341072d0 0x7f58341039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5838c00700 1 -- 172.21.15.103:0/1872790243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5834101c20 msgr2=0x7f5834103480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5838c00700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5834101c20 0x7f5834103480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5838c00700 1 -- 172.21.15.103:0/1872790243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58341072d0 msgr2=0x7f58341039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5838c00700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58341072d0 0x7f58341039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5838c00700 1 -- 172.21.15.103:0/1872790243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58341a70e0 con 0x7f5834101260 2024-05-22T23:58:20.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.622+0000 7f5833e00700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5834101c20 0x7f5834103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.623+0000 7f5838c00700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5834101260 0x7f5834102f40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5828005950 tx=0x7f582800fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.623+0000 7f5831600700 1 -- 172.21.15.103:0/1872790243 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828005140 con 0x7f5834101260 2024-05-22T23:58:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.623+0000 7f5831600700 1 -- 172.21.15.103:0/1872790243 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f582800b560 con 0x7f5834101260 2024-05-22T23:58:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.623+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58341a7370 con 0x7f5834101260 2024-05-22T23:58:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.623+0000 7f5831600700 1 -- 172.21.15.103:0/1872790243 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5828003ae0 con 0x7f5834101260 2024-05-22T23:58:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.623+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58341a77c0 con 0x7f5834101260 2024-05-22T23:58:20.628 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.625+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5834069050 con 0x7f5834101260 2024-05-22T23:58:20.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.628+0000 7f5831600700 1 -- 172.21.15.103:0/1872790243 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f582800ba80 con 0x7f5834101260 2024-05-22T23:58:20.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.629+0000 7f5831600700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f581c0a2cb0 0x7f581c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:20.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.630+0000 7f5831600700 1 -- 172.21.15.103:0/1872790243 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7f58280d0590 con 0x7f5834101260 2024-05-22T23:58:20.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.630+0000 7f5833e00700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f581c0a2cb0 0x7f581c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:20.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.631+0000 7f5833e00700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f581c0a2cb0 0x7f581c0a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5834078810 tx=0x7f5820009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:20.634 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.631+0000 7f5831600700 1 -- 172.21.15.103:0/1872790243 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f582805e850 con 0x7f5834101260 2024-05-22T23:58:20.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.862+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f58341a7b10 con 0x7f5834101260 2024-05-22T23:58:20.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.864+0000 7f5831600700 1 -- 172.21.15.103:0/1872790243 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7f5828013030 con 0x7f5834101260 2024-05-22T23:58:20.868 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:20.868 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:58:20.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f581c0a2cb0 msgr2=0x7f581c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:20.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f581c0a2cb0 0x7f581c0a5170 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5834078810 tx=0x7f5820009260 comp rx=0 tx=0).stop 2024-05-22T23:58:20.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5834101260 msgr2=0x7f5834102f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:20.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5834101260 0x7f5834102f40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5828005950 tx=0x7f582800fb20 comp rx=0 tx=0).stop 2024-05-22T23:58:20.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 shutdown_connections 2024-05-22T23:58:20.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58341072d0 0x7f58341039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f581c0a2cb0 0x7f581c0a5170 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5834101260 0x7f5834102f40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 --2- 172.21.15.103:0/1872790243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5834101c20 0x7f5834103480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:20.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.867+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 >> 172.21.15.103:0/1872790243 conn(0x7f5834078210 msgr2=0x7f5834100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:20.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.868+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 shutdown_connections 2024-05-22T23:58:20.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:20.868+0000 7f583b600700 1 -- 172.21.15.103:0/1872790243 wait complete. 2024-05-22T23:58:21.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:21 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1872790243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:21.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:21 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1872790243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:21.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:21 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1872790243' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:21.728 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-22T23:58:21.728 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd dump --format=json 2024-05-22T23:58:22.172 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:22.506 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:22 smithi103 ceph-mon[31370]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-05-22T23:58:22.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:22 smithi105 ceph-mon[36426]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-05-22T23:58:22.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:22 smithi174 ceph-mon[36179]: pgmap v347: 129 pgs: 129 active+clean; 638 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2024-05-22T23:58:24.408 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:24 smithi103 ceph-mon[31370]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-05-22T23:58:24.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.451+0000 7fa5b5000700 1 -- 172.21.15.103:0/3016566410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5b0104570 msgr2=0x7fa5b01049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.451+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3016566410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5b0104570 0x7fa5b01049f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa59c009990 tx=0x7fa59c009ca0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.451+0000 7fa5b5000700 1 -- 172.21.15.103:0/3016566410 shutdown_connections 2024-05-22T23:58:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.451+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3016566410 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5b010a620 0x7fa5b010aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.451+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3016566410 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b0104f30 0x7fa5b01113f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.451+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3016566410 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5b0104570 0x7fa5b01049f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.451+0000 7fa5b5000700 1 -- 172.21.15.103:0/3016566410 >> 172.21.15.103:0/3016566410 conn(0x7fa5b0100040 msgr2=0x7fa5b0102460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.452+0000 7fa5b5000700 1 -- 172.21.15.103:0/3016566410 shutdown_connections 2024-05-22T23:58:24.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.452+0000 7fa5b5000700 1 -- 172.21.15.103:0/3016566410 wait complete. 2024-05-22T23:58:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5b5000700 1 Processor -- start 2024-05-22T23:58:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5b5000700 1 -- start start 2024-05-22T23:58:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5b5000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5b0104570 0x7fa5b01a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5b5000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b0104f30 0x7fa5b01a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:24.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5b5000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5b010a620 0x7fa5b019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5b5000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b01148d0 con 0x7fa5b0104f30 2024-05-22T23:58:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5b5000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0114750 con 0x7fa5b0104570 2024-05-22T23:58:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5b5000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0114a50 con 0x7fa5b010a620 2024-05-22T23:58:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5ade00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5b0104570 0x7fa5b01a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5ade00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5b0104570 0x7fa5b01a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:33060/0 (socket says 172.21.15.103:33060) 2024-05-22T23:58:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5ade00700 1 -- 172.21.15.103:0/3376467480 learned_addr learned my addr 172.21.15.103:0/3376467480 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.453+0000 7fa5ad400700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b0104f30 0x7fa5b01a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:24.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.454+0000 7fa5ae800700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5b010a620 0x7fa5b019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.454+0000 7fa5ade00700 1 -- 172.21.15.103:0/3376467480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5b010a620 msgr2=0x7fa5b019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.454+0000 7fa5ade00700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5b010a620 0x7fa5b019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.454+0000 7fa5ade00700 1 -- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b0104f30 msgr2=0x7fa5b01a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.454+0000 7fa5ade00700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b0104f30 0x7fa5b01a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.454+0000 7fa5ade00700 1 -- 172.21.15.103:0/3376467480 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b019b230 con 0x7fa5b0104570 2024-05-22T23:58:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.454+0000 7fa5ade00700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5b0104570 0x7fa5b01a0910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000e9c0 tx=0x7fa5a000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.454+0000 7fa5a6a00700 1 -- 172.21.15.103:0/3376467480 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a000cbd0 con 0x7fa5b0104570 2024-05-22T23:58:24.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.455+0000 7fa5a6a00700 1 -- 172.21.15.103:0/3376467480 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5a000ee50 con 0x7fa5b0104570 2024-05-22T23:58:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.455+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b019b520 con 0x7fa5b0104570 2024-05-22T23:58:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.455+0000 7fa5a6a00700 1 -- 172.21.15.103:0/3376467480 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0005680 con 0x7fa5b0104570 2024-05-22T23:58:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.455+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b01a77e0 con 0x7fa5b0104570 2024-05-22T23:58:24.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.456+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b0069050 con 0x7fa5b0104570 2024-05-22T23:58:24.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.457+0000 7fa5a6a00700 1 -- 172.21.15.103:0/3376467480 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fa5a0044b60 con 0x7fa5b0104570 2024-05-22T23:58:24.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.459+0000 7fa5a6a00700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa5940a2d30 0x7fa5940a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:24.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.459+0000 7fa5a6a00700 1 -- 172.21.15.103:0/3376467480 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8802+0+0 (secure 0 0 0) 0x7fa5a00cf670 con 0x7fa5b0104570 2024-05-22T23:58:24.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.459+0000 7fa5ad400700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa5940a2d30 0x7fa5940a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:24.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.460+0000 7fa5ad400700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa5940a2d30 0x7fa5940a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0109c30 tx=0x7fa59c00b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:24.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.461+0000 7fa5a6a00700 1 -- 172.21.15.103:0/3376467480 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa5a009c1d0 con 0x7fa5b0104570 2024-05-22T23:58:24.688 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.685+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5b019bfc0 con 0x7fa5b0104570 2024-05-22T23:58:24.690 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.687+0000 7fa5a6a00700 1 -- 172.21.15.103:0/3376467480 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24187 (secure 0 0 0) 0x7fa5a009bad0 con 0x7fa5b0104570 2024-05-22T23:58:24.691 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:58:24.691 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":91,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:57:38.243351+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:58:24.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa5940a2d30 msgr2=0x7fa5940a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:24.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa5940a2d30 0x7fa5940a51f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0109c30 tx=0x7fa59c00b280 comp rx=0 tx=0).stop 2024-05-22T23:58:24.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5b0104570 msgr2=0x7fa5b01a0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:24.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5b0104570 0x7fa5b01a0910 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000e9c0 tx=0x7fa5a000ecd0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 shutdown_connections 2024-05-22T23:58:24.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa5b0104570 0x7fa5b01a0910 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa5940a2d30 0x7fa5940a51f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5b010a620 0x7fa5b019aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 --2- 172.21.15.103:0/3376467480 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa5b0104f30 0x7fa5b01a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:24.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 >> 172.21.15.103:0/3376467480 conn(0x7fa5b0100040 msgr2=0x7fa5b01016b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:24.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.690+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 shutdown_connections 2024-05-22T23:58:24.694 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:24.691+0000 7fa5b5000700 1 -- 172.21.15.103:0/3376467480 wait complete. 2024-05-22T23:58:24.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:24 smithi105 ceph-mon[36426]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-05-22T23:58:24.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:24 smithi174 ceph-mon[36179]: pgmap v348: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2024-05-22T23:58:25.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-22T23:58:25.581 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-22T23:58:25.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-22T23:58:25.582 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-22T23:58:25.583 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-22T23:58:25.583 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-22T23:58:25.583 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-22T23:58:25.584 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-22T23:58:25.584 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-22T23:58:25.585 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-22T23:58:25.585 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-22T23:58:25.585 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-22T23:58:25.704 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:25 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3376467480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:25.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:25 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3376467480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:25.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:25 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3376467480' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:58:26.578 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:26.600 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:26.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:26 smithi105 ceph-mon[36426]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-22T23:58:26.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:26 smithi103 ceph-mon[31370]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-22T23:58:26.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:26 smithi174 ceph-mon[36179]: pgmap v349: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-22T23:58:28.238 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.325 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:28 smithi103 ceph-mon[31370]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-22T23:58:28.365 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.365 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.367 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.374 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.378 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.378 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.379 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.383 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:28.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:28 smithi105 ceph-mon[36426]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-22T23:58:28.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:28 smithi174 ceph-mon[36179]: pgmap v350: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-22T23:58:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2024-05-22T23:58:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-22T23:58:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-22T23:58:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-22T23:58:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-22T23:58:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-22T23:58:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-05-22T23:58:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-05-22T23:58:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-05-22T23:58:30.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-05-22T23:58:30.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]: dispatch 2024-05-22T23:58:30.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]: dispatch 2024-05-22T23:58:30.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]: dispatch 2024-05-22T23:58:31.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]': finished 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: osdmap e92: 12 total, 12 up, 12 in 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-22T23:58:31.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:31 smithi103 ceph-mon[31370]: osdmap e93: 12 total, 12 up, 12 in 2024-05-22T23:58:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2024-05-22T23:58:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-22T23:58:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-22T23:58:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-22T23:58:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-22T23:58:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-22T23:58:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-05-22T23:58:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-05-22T23:58:31.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-05-22T23:58:31.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]': finished 2024-05-22T23:58:31.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: osdmap e92: 12 total, 12 up, 12 in 2024-05-22T23:58:31.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-22T23:58:31.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:31 smithi105 ceph-mon[36426]: osdmap e93: 12 total, 12 up, 12 in 2024-05-22T23:58:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2024-05-22T23:58:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-22T23:58:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-22T23:58:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-22T23:58:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-22T23:58:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-22T23:58:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-05-22T23:58:31.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-05-22T23:58:31.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8]}]': finished 2024-05-22T23:58:31.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 7]}]': finished 2024-05-22T23:58:31.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: osdmap e92: 12 total, 12 up, 12 in 2024-05-22T23:58:31.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: pgmap v352: 129 pgs: 129 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2024-05-22T23:58:31.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:31 smithi174 ceph-mon[36179]: osdmap e93: 12 total, 12 up, 12 in 2024-05-22T23:58:32.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:32 smithi103 ceph-mon[31370]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:32.443 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:32 smithi103 ceph-mon[31370]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-22T23:58:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:32 smithi105 ceph-mon[36426]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:32.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:32 smithi105 ceph-mon[36426]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-22T23:58:32.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:32 smithi174 ceph-mon[36179]: pgmap v354: 129 pgs: 2 peering, 127 active+clean; 640 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:32.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:32 smithi174 ceph-mon[36179]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-22T23:58:32.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 -- 172.21.15.103:0/449453370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f912c10a650 msgr2=0x7f912c10aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:32.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 --2- 172.21.15.103:0/449453370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f912c10a650 0x7f912c10aa30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f91200099c0 tx=0x7f9120009cd0 comp rx=0 tx=0).stop 2024-05-22T23:58:32.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 -- 172.21.15.103:0/449453370 shutdown_connections 2024-05-22T23:58:32.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 --2- 172.21.15.103:0/449453370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f912c10a650 0x7f912c10aa30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:32.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 --2- 172.21.15.103:0/449453370 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912c104f60 0x7f912c111420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:32.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 --2- 172.21.15.103:0/449453370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f912c1045a0 0x7f912c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:32.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 -- 172.21.15.103:0/449453370 >> 172.21.15.103:0/449453370 conn(0x7f912c100070 msgr2=0x7f912c102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:32.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 -- 172.21.15.103:0/449453370 shutdown_connections 2024-05-22T23:58:32.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.960+0000 7f9133c00700 1 -- 172.21.15.103:0/449453370 wait complete. 2024-05-22T23:58:32.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.961+0000 7f9133c00700 1 Processor -- start 2024-05-22T23:58:32.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f9133c00700 1 -- start start 2024-05-22T23:58:32.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f9133c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f912c1045a0 0x7f912c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:32.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f9133c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f912c104f60 0x7f912c1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:32.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f9133c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912c10a650 0x7f912c19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:32.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f9133c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f912c1148f0 con 0x7f912c10a650 2024-05-22T23:58:32.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f9133c00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f912c114770 con 0x7f912c1045a0 2024-05-22T23:58:32.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f9133c00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f912c114a70 con 0x7f912c104f60 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f912be00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f912c104f60 0x7f912c1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f912be00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f912c104f60 0x7f912c1a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:44936/0 (socket says 172.21.15.103:44936) 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.962+0000 7f912be00700 1 -- 172.21.15.103:0/1492260972 learned_addr learned my addr 172.21.15.103:0/1492260972 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.963+0000 7f9131200700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f912c1045a0 0x7f912c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.963+0000 7f9131c00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912c10a650 0x7f912c19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.963+0000 7f912be00700 1 -- 172.21.15.103:0/1492260972 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f912c1045a0 msgr2=0x7f912c1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.963+0000 7f912be00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f912c1045a0 0x7f912c1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.963+0000 7f912be00700 1 -- 172.21.15.103:0/1492260972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912c10a650 msgr2=0x7f912c19aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.963+0000 7f912be00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f912c10a650 0x7f912c19aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.963+0000 7f912be00700 1 -- 172.21.15.103:0/1492260972 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912c19b210 con 0x7f912c104f60 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.963+0000 7f912be00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f912c104f60 0x7f912c1a0de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f911800d950 tx=0x7f911800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:32.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.964+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9118016070 con 0x7f912c104f60 2024-05-22T23:58:32.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.964+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f912c19b500 con 0x7f912c104f60 2024-05-22T23:58:32.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.964+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9118009e20 con 0x7f912c104f60 2024-05-22T23:58:32.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.964+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911800f620 con 0x7f912c104f60 2024-05-22T23:58:32.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.964+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f912c1a7770 con 0x7f912c104f60 2024-05-22T23:58:32.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.965+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f912c062af0 con 0x7f912c104f60 2024-05-22T23:58:32.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.967+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f9118019070 con 0x7f912c104f60 2024-05-22T23:58:32.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.968+0000 7f9129600700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91140a2cb0 0x7f91140a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:32.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.969+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f91180cf3d0 con 0x7f912c104f60 2024-05-22T23:58:32.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.969+0000 7f9131200700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91140a2cb0 0x7f91140a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:32.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.969+0000 7f9129600700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] conn(0x7f91140ade10 0x7f91140b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:32.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.969+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 --> [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9118005690 con 0x7f91140ade10 2024-05-22T23:58:32.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.969+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f91180cf780 con 0x7f912c104f60 2024-05-22T23:58:32.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.969+0000 7f9131c00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] conn(0x7f91140ade10 0x7f91140b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:32.974 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.970+0000 7f9131200700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91140a2cb0 0x7f91140a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9120009990 tx=0x7f9120005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:32.974 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.970+0000 7f9131c00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] conn(0x7f91140ade10 0x7f91140b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:32.975 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.971+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 <== osd.10 v2:172.21.15.105:6824/3074362984 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9118005690 con 0x7f91140ade10 2024-05-22T23:58:32.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.991+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 --> [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f912c062f70 con 0x7f91140ade10 2024-05-22T23:58:32.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.991+0000 7f9129600700 1 -- 172.21.15.103:0/1492260972 <== osd.10 v2:172.21.15.105:6824/3074362984 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f912c062f70 con 0x7f91140ade10 2024-05-22T23:58:32.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 >> [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] conn(0x7f91140ade10 msgr2=0x7f91140b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:32.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9133c00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] conn(0x7f91140ade10 0x7f91140b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:32.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91140a2cb0 msgr2=0x7f91140a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:32.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9133c00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91140a2cb0 0x7f91140a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9120009990 tx=0x7f9120005ab0 comp rx=0 tx=0).stop 2024-05-22T23:58:32.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f912c104f60 msgr2=0x7f912c1a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:32.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9133c00700 1 --2- 172.21.15.103:0/1492260972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f912c104f60 0x7f912c1a0de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f911800d950 tx=0x7f911800dc60 comp rx=0 tx=0).stop 2024-05-22T23:58:32.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9131c00700 1 -- 172.21.15.103:0/1492260972 reap_dead start 2024-05-22T23:58:32.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 shutdown_connections 2024-05-22T23:58:32.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.992+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 >> 172.21.15.103:0/1492260972 conn(0x7f912c100070 msgr2=0x7f912c1016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:32.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.993+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 shutdown_connections 2024-05-22T23:58:32.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:32.993+0000 7f9133c00700 1 -- 172.21.15.103:0/1492260972 wait complete. 2024-05-22T23:58:34.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:33 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:58:34.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:33 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:58:34.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:33 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-22T23:58:34.989 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:34 smithi174 ceph-mon[36179]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:34 smithi105 ceph-mon[36426]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:35.291 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:34 smithi103 ceph-mon[31370]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:37.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:36 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:36 smithi105 ceph-mon[36426]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s; 126 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:37.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:36 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:36 smithi103 ceph-mon[31370]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s; 126 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:37.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:36 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:58:37.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:36 smithi174 ceph-mon[36179]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s; 126 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:38.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:37 smithi105 ceph-mon[36426]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-22T23:58:38.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:37 smithi105 ceph-mon[36426]: Cluster is now healthy 2024-05-22T23:58:38.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:37 smithi103 ceph-mon[31370]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-22T23:58:38.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:37 smithi103 ceph-mon[31370]: Cluster is now healthy 2024-05-22T23:58:38.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:37 smithi174 ceph-mon[36179]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-22T23:58:38.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:37 smithi174 ceph-mon[36179]: Cluster is now healthy 2024-05-22T23:58:38.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:38 smithi103 ceph-mon[31370]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 254 B/s wr, 5 op/s; 125 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:39.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:38 smithi105 ceph-mon[36426]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 254 B/s wr, 5 op/s; 125 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:39.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:38 smithi174 ceph-mon[36179]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 254 B/s wr, 5 op/s; 125 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:39.364 INFO:teuthology.orchestra.run.smithi103.stdout:317827579936 2024-05-22T23:58:39.365 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-22T23:58:40.436 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 -- 172.21.15.103:0/3605865620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 msgr2=0x7f4cf010f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 --2- 172.21.15.103:0/3605865620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 0x7f4cf010f220 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8009f90 tx=0x7f4ce800b330 comp rx=0 tx=0).stop 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 -- 172.21.15.103:0/3605865620 shutdown_connections 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 --2- 172.21.15.103:0/3605865620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 0x7f4cf010f220 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 --2- 172.21.15.103:0/3605865620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf01083e0 0x7f4cf01087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 --2- 172.21.15.103:0/3605865620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf006a070 0x7f4cf006a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 -- 172.21.15.103:0/3605865620 >> 172.21.15.103:0/3605865620 conn(0x7f4cf0078210 msgr2=0x7f4cf0078620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 -- 172.21.15.103:0/3605865620 shutdown_connections 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.386+0000 7f4cf5000700 1 -- 172.21.15.103:0/3605865620 wait complete. 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.387+0000 7f4cf5000700 1 Processor -- start 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.387+0000 7f4cf5000700 1 -- start start 2024-05-22T23:58:40.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4cf5000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf006a070 0x7f4cf0101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4cf5000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 0x7f4cf0101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4cf5000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf01083e0 0x7f4cf0101db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4cf5000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf0112630 con 0x7f4cf006aa10 2024-05-22T23:58:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4cf5000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf01124b0 con 0x7f4cf006a070 2024-05-22T23:58:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4cf5000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cf01127b0 con 0x7f4cf01083e0 2024-05-22T23:58:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4ced400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 0x7f4cf0101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:40.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4ced400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 0x7f4cf0101870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46058/0 (socket says 172.21.15.103:46058) 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4ced400700 1 -- 172.21.15.103:0/2273028369 learned_addr learned my addr 172.21.15.103:0/2273028369 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4cede00700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf006a070 0x7f4cf0101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.388+0000 7f4cee800700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf01083e0 0x7f4cf0101db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ced400700 1 -- 172.21.15.103:0/2273028369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf01083e0 msgr2=0x7f4cf0101db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ced400700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cf01083e0 0x7f4cf0101db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ced400700 1 -- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf006a070 msgr2=0x7f4cf0101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ced400700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf006a070 0x7f4cf0101330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ced400700 1 -- 172.21.15.103:0/2273028369 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cf01a6fd0 con 0x7f4cf006aa10 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4cede00700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf006a070 0x7f4cf0101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ced400700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 0x7f4cf0101870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00ec10 tx=0x7f4cdc00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc00cc90 con 0x7f4cf006aa10 2024-05-22T23:58:40.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cf01a72c0 con 0x7f4cf006aa10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4cdc00ce30 con 0x7f4cf006aa10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cdc010690 con 0x7f4cf006aa10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.389+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cf01a7800 con 0x7f4cf006aa10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.391+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4cf0062a30 con 0x7f4cf006aa10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.393+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4cdc003680 con 0x7f4cf006aa10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.395+0000 7f4ce6a00700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4cd40a2cb0 0x7f4cd40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.395+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4cdc0cf8f0 con 0x7f4cf006aa10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.395+0000 7f4cede00700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4cd40a2cb0 0x7f4cd40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.395+0000 7f4ce6a00700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] conn(0x7f4cd40ade10 0x7f4cd40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.395+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 --> [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4cd40b0980 con 0x7f4cd40ade10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.395+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f4cdc0d42a0 con 0x7f4cf006aa10 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.396+0000 7f4cee800700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] conn(0x7f4cd40ade10 0x7f4cd40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.396+0000 7f4cede00700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4cd40a2cb0 0x7f4cd40a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009680 tx=0x7f4ce0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:40.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.396+0000 7f4cee800700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] conn(0x7f4cd40ade10 0x7f4cd40b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.397+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 <== osd.4 v2:172.21.15.105:6808/4091773686 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4cd40b0980 con 0x7f4cd40ade10 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.415+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 --> [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4cf0062eb0 con 0x7f4cd40ade10 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.416+0000 7f4ce6a00700 1 -- 172.21.15.103:0/2273028369 <== osd.4 v2:172.21.15.105:6808/4091773686 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4cf0062eb0 con 0x7f4cd40ade10 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.416+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] conn(0x7f4cd40ade10 msgr2=0x7f4cd40b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.416+0000 7f4cf5000700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] conn(0x7f4cd40ade10 0x7f4cd40b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4cd40a2cb0 msgr2=0x7f4cd40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cf5000700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4cd40a2cb0 0x7f4cd40a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009680 tx=0x7f4ce0009260 comp rx=0 tx=0).stop 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 msgr2=0x7f4cf0101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cf5000700 1 --2- 172.21.15.103:0/2273028369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4cf006aa10 0x7f4cf0101870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4cdc00ec10 tx=0x7f4cdc00ef20 comp rx=0 tx=0).stop 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cee800700 1 -- 172.21.15.103:0/2273028369 reap_dead start 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 shutdown_connections 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 >> 172.21.15.103:0/2273028369 conn(0x7f4cf0078210 msgr2=0x7f4cf00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:40.441 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 shutdown_connections 2024-05-22T23:58:40.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:40.417+0000 7f4cf5000700 1 -- 172.21.15.103:0/2273028369 wait complete. 2024-05-22T23:58:40.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:40 smithi105 ceph-mon[36426]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 5 op/s; 101 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:40.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:40 smithi103 ceph-mon[31370]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 5 op/s; 101 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:40.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:40 smithi174 ceph-mon[36179]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 5 op/s; 101 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:42.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 -- 172.21.15.103:0/1912469313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f891c10ed60 msgr2=0x7f891c111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 --2- 172.21.15.103:0/1912469313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f891c10ed60 0x7f891c111150 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f891400b0a0 tx=0x7f891400b3b0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 -- 172.21.15.103:0/1912469313 shutdown_connections 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 --2- 172.21.15.103:0/1912469313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f891c074590 0x7f891c074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 --2- 172.21.15.103:0/1912469313 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f891c10ed60 0x7f891c111150 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 --2- 172.21.15.103:0/1912469313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f891c074eb0 0x7f891c10e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 -- 172.21.15.103:0/1912469313 >> 172.21.15.103:0/1912469313 conn(0x7f891c0794c0 msgr2=0x7f891c0798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 -- 172.21.15.103:0/1912469313 shutdown_connections 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.942+0000 7f8923800700 1 -- 172.21.15.103:0/1912469313 wait complete. 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.943+0000 7f8923800700 1 Processor -- start 2024-05-22T23:58:42.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8923800700 1 -- start start 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8923800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f891c074590 0x7f891c1a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8923800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f891c074eb0 0x7f891c1a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8923800700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f891c10ed60 0x7f891c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8923800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f891c1148a0 con 0x7f891c074eb0 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8923800700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f891c114720 con 0x7f891c074590 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8923800700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f891c114a20 con 0x7f891c10ed60 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8913e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f891c074eb0 0x7f891c1a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8920e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f891c074590 0x7f891c1a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8920e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f891c074590 0x7f891c1a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:57680/0 (socket says 172.21.15.103:57680) 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8920e00700 1 -- 172.21.15.103:0/4136629482 learned_addr learned my addr 172.21.15.103:0/4136629482 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.944+0000 7f8921800700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f891c10ed60 0x7f891c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f8920e00700 1 -- 172.21.15.103:0/4136629482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f891c10ed60 msgr2=0x7f891c19aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f8920e00700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f891c10ed60 0x7f891c19aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f8920e00700 1 -- 172.21.15.103:0/4136629482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f891c074eb0 msgr2=0x7f891c1a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f8920e00700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f891c074eb0 0x7f891c1a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f8920e00700 1 -- 172.21.15.103:0/4136629482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891c19b200 con 0x7f891c074590 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f8920e00700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f891c074590 0x7f891c1a08e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f890c00eb30 tx=0x7f890c00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c00cbd0 con 0x7f891c074590 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f890c004500 con 0x7f891c074590 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f890c010430 con 0x7f891c074590 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.945+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f891c19b4f0 con 0x7f891c074590 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.946+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f891c1a77b0 con 0x7f891c074590 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.946+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f891c0ffef0 con 0x7f891c074590 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.949+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f890c003f00 con 0x7f891c074590 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.950+0000 7f891a000700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89000a2cb0 0x7f89000a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.950+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f890c0cfb80 con 0x7f891c074590 2024-05-22T23:58:42.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.950+0000 7f8913e00700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89000a2cb0 0x7f89000a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.950+0000 7f891a000700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] conn(0x7f89000ade10 0x7f89000b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.950+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 --> [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f890c023e50 con 0x7f89000ade10 2024-05-22T23:58:42.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.950+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f890c0cff30 con 0x7f891c074590 2024-05-22T23:58:42.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.951+0000 7f8921800700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] conn(0x7f89000ade10 0x7f89000b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.951+0000 7f8913e00700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89000a2cb0 0x7f89000a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f891c110920 tx=0x7f8904009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:42.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.951+0000 7f8921800700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] conn(0x7f89000ade10 0x7f89000b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.952+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 <== osd.11 v2:172.21.15.174:6824/1551576781 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f890c023e50 con 0x7f89000ade10 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.970+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 --> [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f891c075c80 con 0x7f89000ade10 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.971+0000 7f891a000700 1 -- 172.21.15.103:0/4136629482 <== osd.11 v2:172.21.15.174:6824/1551576781 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f891c075c80 con 0x7f89000ade10 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.971+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 >> [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] conn(0x7f89000ade10 msgr2=0x7f89000b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.971+0000 7f8923800700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] conn(0x7f89000ade10 0x7f89000b02b0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89000a2cb0 msgr2=0x7f89000a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8923800700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89000a2cb0 0x7f89000a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f891c110920 tx=0x7f8904009260 comp rx=0 tx=0).stop 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f891c074590 msgr2=0x7f891c1a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8923800700 1 --2- 172.21.15.103:0/4136629482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f891c074590 0x7f891c1a08e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f890c00eb30 tx=0x7f890c00ee40 comp rx=0 tx=0).stop 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8921800700 1 -- 172.21.15.103:0/4136629482 reap_dead start 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 shutdown_connections 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 >> 172.21.15.103:0/4136629482 conn(0x7f891c0794c0 msgr2=0x7f891c0ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 shutdown_connections 2024-05-22T23:58:42.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:41.972+0000 7f8923800700 1 -- 172.21.15.103:0/4136629482 wait complete. 2024-05-22T23:58:42.525 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:42 smithi103 ceph-mon[31370]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 185 B/s wr, 5 op/s; 91 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:42.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:42 smithi105 ceph-mon[36426]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 185 B/s wr, 5 op/s; 91 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:42.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:42 smithi174 ceph-mon[36179]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 185 B/s wr, 5 op/s; 91 B/s, 0 keys/s, 0 objects/s recovering 2024-05-22T23:58:42.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 -- 172.21.15.103:0/3042317052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 msgr2=0x7f4874108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 --2- 172.21.15.103:0/3042317052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 0x7f4874108e10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48600098f0 tx=0x7f4860009c00 comp rx=0 tx=0).stop 2024-05-22T23:58:42.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 -- 172.21.15.103:0/3042317052 shutdown_connections 2024-05-22T23:58:42.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 --2- 172.21.15.103:0/3042317052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4874109350 0x7f4874115860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 --2- 172.21.15.103:0/3042317052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f487410e990 0x7f487410ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 --2- 172.21.15.103:0/3042317052 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 0x7f4874108e10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 -- 172.21.15.103:0/3042317052 >> 172.21.15.103:0/3042317052 conn(0x7f4874075b30 msgr2=0x7f4874107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 -- 172.21.15.103:0/3042317052 shutdown_connections 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.574+0000 7f487b200700 1 -- 172.21.15.103:0/3042317052 wait complete. 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.575+0000 7f487b200700 1 Processor -- start 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f487b200700 1 -- start start 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f487b200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 0x7f487419f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f487b200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4874109350 0x7f487419f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f487b200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f487410e990 0x7f48741a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f487b200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4874118d70 con 0x7f4874109350 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f487b200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4874118bf0 con 0x7f487410e990 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f487b200700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4874118ef0 con 0x7f4874108990 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f4873400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4874109350 0x7f487419f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f4873e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 0x7f487419f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f4873e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 0x7f487419f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:43936/0 (socket says 172.21.15.103:43936) 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.576+0000 7f4873e00700 1 -- 172.21.15.103:0/587298927 learned_addr learned my addr 172.21.15.103:0/587298927 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4879200700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f487410e990 0x7f48741a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4873400700 1 -- 172.21.15.103:0/587298927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 msgr2=0x7f487419f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4873400700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 0x7f487419f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4873400700 1 -- 172.21.15.103:0/587298927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f487410e990 msgr2=0x7f48741a3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4873400700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f487410e990 0x7f48741a3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4873400700 1 -- 172.21.15.103:0/587298927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48741a4190 con 0x7f4874109350 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4873e00700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4874108990 0x7f487419f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4879200700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f487410e990 0x7f48741a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.577+0000 7f4873400700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4874109350 0x7f487419f6c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4860000c00 tx=0x7f486000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.578+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4860005040 con 0x7f4874109350 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.578+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48741a4420 con 0x7f4874109350 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.578+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48600056c0 con 0x7f4874109350 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.578+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4860003a00 con 0x7f4874109350 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.578+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48741abd00 con 0x7f4874109350 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.579+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4874062af0 con 0x7f4874109350 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.581+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f486000b560 con 0x7f4874109350 2024-05-22T23:58:42.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.583+0000 7f4870c00700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f48580a2cb0 0x7f48580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.583+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f48600d0890 con 0x7f4874109350 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.583+0000 7f4873e00700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f48580a2cb0 0x7f48580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.583+0000 7f4870c00700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] conn(0x7f48580adce0 0x7f48580b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.583+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 --> [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f486000b0b0 con 0x7f48580adce0 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.583+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f48600d0c40 con 0x7f4874109350 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.584+0000 7f4879200700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] conn(0x7f48580adce0 0x7f48580b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.584+0000 7f4873e00700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f48580a2cb0 0x7f48580a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4864005950 tx=0x7f4864009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.584+0000 7f4879200700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] conn(0x7f48580adce0 0x7f48580b0180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:42.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.585+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 <== osd.2 v2:172.21.15.174:6800/3252167839 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f486000b0b0 con 0x7f48580adce0 2024-05-22T23:58:42.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.603+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 --> [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4874062f70 con 0x7f48580adce0 2024-05-22T23:58:42.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.604+0000 7f4870c00700 1 -- 172.21.15.103:0/587298927 <== osd.2 v2:172.21.15.174:6800/3252167839 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4874062f70 con 0x7f48580adce0 2024-05-22T23:58:42.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.604+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 >> [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] conn(0x7f48580adce0 msgr2=0x7f48580b0180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.604+0000 7f487b200700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] conn(0x7f48580adce0 0x7f48580b0180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:42.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.604+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f48580a2cb0 msgr2=0x7f48580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.604+0000 7f487b200700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f48580a2cb0 0x7f48580a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4864005950 tx=0x7f4864009310 comp rx=0 tx=0).stop 2024-05-22T23:58:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.604+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4874109350 msgr2=0x7f487419f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.604+0000 7f487b200700 1 --2- 172.21.15.103:0/587298927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4874109350 0x7f487419f6c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4860000c00 tx=0x7f486000fa00 comp rx=0 tx=0).stop 2024-05-22T23:58:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.604+0000 7f4879200700 1 -- 172.21.15.103:0/587298927 reap_dead start 2024-05-22T23:58:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.605+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 shutdown_connections 2024-05-22T23:58:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.605+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 >> 172.21.15.103:0/587298927 conn(0x7f4874075b30 msgr2=0x7f4874076070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.605+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 shutdown_connections 2024-05-22T23:58:42.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:42.605+0000 7f487b200700 1 -- 172.21.15.103:0/587298927 wait complete. 2024-05-22T23:58:43.266 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:44.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 -- 172.21.15.103:0/2368688337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfac1088a0 msgr2=0x7fcfac108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 --2- 172.21.15.103:0/2368688337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfac1088a0 0x7fcfac108d20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009990 tx=0x7fcf98009ca0 comp rx=0 tx=0).stop 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 -- 172.21.15.103:0/2368688337 shutdown_connections 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 --2- 172.21.15.103:0/2368688337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfac109260 0x7fcfac115700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 --2- 172.21.15.103:0/2368688337 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfac10d770 0x7fcfac10db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 --2- 172.21.15.103:0/2368688337 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfac1088a0 0x7fcfac108d20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 -- 172.21.15.103:0/2368688337 >> 172.21.15.103:0/2368688337 conn(0x7fcfac0780f0 msgr2=0x7fcfac107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 -- 172.21.15.103:0/2368688337 shutdown_connections 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.811+0000 7fcfb2000700 1 -- 172.21.15.103:0/2368688337 wait complete. 2024-05-22T23:58:44.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.812+0000 7fcfb2000700 1 Processor -- start 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb2000700 1 -- start start 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb2000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfac1088a0 0x7fcfac19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb2000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfac109260 0x7fcfac19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb2000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfac10d770 0x7fcfac1a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb2000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfac118bd0 con 0x7fcfac10d770 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb2000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcfac118a50 con 0x7fcfac1088a0 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb2000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfac118d50 con 0x7fcfac109260 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfabe00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfac109260 0x7fcfac19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb0c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfac1088a0 0x7fcfac19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfb1600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfac10d770 0x7fcfac1a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfabe00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfac109260 0x7fcfac19f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:43874/0 (socket says 172.21.15.103:43874) 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.813+0000 7fcfabe00700 1 -- 172.21.15.103:0/3847617423 learned_addr learned my addr 172.21.15.103:0/3847617423 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:44.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfb0c00700 1 -- 172.21.15.103:0/3847617423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfac109260 msgr2=0x7fcfac19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfb0c00700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfac109260 0x7fcfac19f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfb0c00700 1 -- 172.21.15.103:0/3847617423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfac10d770 msgr2=0x7fcfac1a3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfb0c00700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfac10d770 0x7fcfac1a3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfb0c00700 1 -- 172.21.15.103:0/3847617423 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfac1a4080 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfb0c00700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfac1088a0 0x7fcfac19f0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000e9f0 tx=0x7fcfa000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000c9c0 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfac1a4370 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfa000cb60 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.814+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0018680 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.815+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfac1abbf0 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.815+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcfac062af0 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.818+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fcfa0010430 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.819+0000 7fcfa9600700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcf940a2cb0 0x7fcf940a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.819+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fcfa00cfa30 con 0x7fcfac1088a0 2024-05-22T23:58:44.185 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.820+0000 7fcfabe00700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcf940a2cb0 0x7fcf940a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.820+0000 7fcfa9600700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] conn(0x7fcf940ade10 0x7fcf940b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.820+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 --> [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf940b0980 con 0x7fcf940ade10 2024-05-22T23:58:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.820+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fcfa00cfe80 con 0x7fcfac1088a0 2024-05-22T23:58:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.820+0000 7fcfb1600700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] conn(0x7fcf940ade10 0x7fcf940b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.820+0000 7fcfabe00700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcf940a2cb0 0x7fcf940a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009750 tx=0x7fcf9800b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.821+0000 7fcfb1600700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] conn(0x7fcf940ade10 0x7fcf940b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:44.186 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.822+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 <== osd.1 v2:172.21.15.105:6800/732590607 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcf940b0980 con 0x7fcf940ade10 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.840+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 --> [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcfac062ef0 con 0x7fcf940ade10 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.841+0000 7fcfa9600700 1 -- 172.21.15.103:0/3847617423 <== osd.1 v2:172.21.15.105:6800/732590607 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcfac062ef0 con 0x7fcf940ade10 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.841+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 >> [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] conn(0x7fcf940ade10 msgr2=0x7fcf940b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.841+0000 7fcfb2000700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] conn(0x7fcf940ade10 0x7fcf940b02b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.841+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcf940a2cb0 msgr2=0x7fcf940a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.841+0000 7fcfb2000700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcf940a2cb0 0x7fcf940a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009750 tx=0x7fcf9800b280 comp rx=0 tx=0).stop 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.842+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfac1088a0 msgr2=0x7fcfac19f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.842+0000 7fcfb2000700 1 --2- 172.21.15.103:0/3847617423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfac1088a0 0x7fcfac19f0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000e9f0 tx=0x7fcfa000ed00 comp rx=0 tx=0).stop 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.842+0000 7fcfb1600700 1 -- 172.21.15.103:0/3847617423 reap_dead start 2024-05-22T23:58:44.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.842+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 shutdown_connections 2024-05-22T23:58:44.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.842+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 >> 172.21.15.103:0/3847617423 conn(0x7fcfac0780f0 msgr2=0x7fcfac078750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:44.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.842+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 shutdown_connections 2024-05-22T23:58:44.188 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:39.842+0000 7fcfb2000700 1 -- 172.21.15.103:0/3847617423 wait complete. 2024-05-22T23:58:44.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:44 smithi103 ceph-mon[31370]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s; 84 B/s, 0 objects/s recovering 2024-05-22T23:58:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:44 smithi105 ceph-mon[36426]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s; 84 B/s, 0 objects/s recovering 2024-05-22T23:58:44.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:44 smithi174 ceph-mon[36179]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s; 84 B/s, 0 objects/s recovering 2024-05-22T23:58:45.856 INFO:teuthology.orchestra.run.smithi103.stdout:55834574932 2024-05-22T23:58:45.857 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-22T23:58:46.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.040+0000 7fb3dfc00700 1 -- 172.21.15.103:0/4075989786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d810a670 msgr2=0x7fb3d810aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.040+0000 7fb3dfc00700 1 --2- 172.21.15.103:0/4075989786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d810a670 0x7fb3d810aa50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc0099c0 tx=0x7fb3cc009cd0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.041+0000 7fb3dfc00700 1 -- 172.21.15.103:0/4075989786 shutdown_connections 2024-05-22T23:58:46.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.041+0000 7fb3dfc00700 1 --2- 172.21.15.103:0/4075989786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d810a670 0x7fb3d810aa50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.041+0000 7fb3dfc00700 1 --2- 172.21.15.103:0/4075989786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d8104f80 0x7fb3d8111440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.041+0000 7fb3dfc00700 1 --2- 172.21.15.103:0/4075989786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d81045c0 0x7fb3d8104a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.041+0000 7fb3dfc00700 1 -- 172.21.15.103:0/4075989786 >> 172.21.15.103:0/4075989786 conn(0x7fb3d8100090 msgr2=0x7fb3d81024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.041+0000 7fb3dfc00700 1 -- 172.21.15.103:0/4075989786 shutdown_connections 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.041+0000 7fb3dfc00700 1 -- 172.21.15.103:0/4075989786 wait complete. 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.042+0000 7fb3dfc00700 1 Processor -- start 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.042+0000 7fb3dfc00700 1 -- start start 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3dfc00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d81045c0 0x7fb3d81a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3dfc00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d8104f80 0x7fb3d81a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3dfc00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d810a670 0x7fb3d819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3dfc00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d81148e0 con 0x7fb3d810a670 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3dfc00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d8114760 con 0x7fb3d81045c0 2024-05-22T23:58:46.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3dfc00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d8114a60 con 0x7fb3d8104f80 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3dd200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d81045c0 0x7fb3d81a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3d7e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d8104f80 0x7fb3d81a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3d7e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d8104f80 0x7fb3d81a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:60812/0 (socket says 172.21.15.103:60812) 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3d7e00700 1 -- 172.21.15.103:0/906604814 learned_addr learned my addr 172.21.15.103:0/906604814 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3dd200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d81045c0 0x7fb3d81a08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:40942/0 (socket says 172.21.15.103:40942) 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.043+0000 7fb3ddc00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d810a670 0x7fb3d819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3dd200700 1 -- 172.21.15.103:0/906604814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d8104f80 msgr2=0x7fb3d81a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3dd200700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d8104f80 0x7fb3d81a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3dd200700 1 -- 172.21.15.103:0/906604814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d810a670 msgr2=0x7fb3d819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3dd200700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb3d810a670 0x7fb3d819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3dd200700 1 -- 172.21.15.103:0/906604814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d819b260 con 0x7fb3d81045c0 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3d7e00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3d8104f80 0x7fb3d81a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:46.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3dd200700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d81045c0 0x7fb3d81a08f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00b580 tx=0x7fb3cc003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc003680 con 0x7fb3d81045c0 2024-05-22T23:58:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc0055a0 con 0x7fb3d81045c0 2024-05-22T23:58:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc00fd00 con 0x7fb3d81045c0 2024-05-22T23:58:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.044+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d819b4f0 con 0x7fb3d81045c0 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.045+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d81a77c0 con 0x7fb3d81045c0 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.045+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3d8062af0 con 0x7fb3d81045c0 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.048+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb3cc00f460 con 0x7fb3d81045c0 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.049+0000 7fb3d5600700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb3c00a2ed0 0x7fb3c00a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.049+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb3cc0d03e0 con 0x7fb3d81045c0 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.049+0000 7fb3d7e00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb3c00a2ed0 0x7fb3c00a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.050+0000 7fb3d5600700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] conn(0x7fb3c00ae030 0x7fb3c00b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.050+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 --> [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3cc001d30 con 0x7fb3c00ae030 2024-05-22T23:58:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.050+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7fb3cc0d0830 con 0x7fb3d81045c0 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.050+0000 7fb3ddc00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] conn(0x7fb3c00ae030 0x7fb3c00b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.050+0000 7fb3d7e00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb3c00a2ed0 0x7fb3c00a5390 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8109c50 tx=0x7fb3c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.051+0000 7fb3ddc00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] conn(0x7fb3c00ae030 0x7fb3c00b04d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.051+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 <== osd.5 v2:172.21.15.174:6808/1001493350 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3cc001d30 con 0x7fb3c00ae030 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.072+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 --> [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb3d8062fc0 con 0x7fb3c00ae030 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.073+0000 7fb3d5600700 1 -- 172.21.15.103:0/906604814 <== osd.5 v2:172.21.15.174:6808/1001493350 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb3d8062fc0 con 0x7fb3c00ae030 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.073+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 >> [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] conn(0x7fb3c00ae030 msgr2=0x7fb3c00b04d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.073+0000 7fb3dfc00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] conn(0x7fb3c00ae030 0x7fb3c00b04d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.073+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb3c00a2ed0 msgr2=0x7fb3c00a5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.073+0000 7fb3dfc00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb3c00a2ed0 0x7fb3c00a5390 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8109c50 tx=0x7fb3c4009260 comp rx=0 tx=0).stop 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.074+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d81045c0 msgr2=0x7fb3d81a08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.110 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.074+0000 7fb3dfc00700 1 --2- 172.21.15.103:0/906604814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3d81045c0 0x7fb3d81a08f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00b580 tx=0x7fb3cc003a90 comp rx=0 tx=0).stop 2024-05-22T23:58:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.074+0000 7fb3ddc00700 1 -- 172.21.15.103:0/906604814 reap_dead start 2024-05-22T23:58:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.074+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 shutdown_connections 2024-05-22T23:58:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.074+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 >> 172.21.15.103:0/906604814 conn(0x7fb3d8100090 msgr2=0x7fb3d81016f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.074+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 shutdown_connections 2024-05-22T23:58:46.111 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.074+0000 7fb3dfc00700 1 -- 172.21.15.103:0/906604814 wait complete. 2024-05-22T23:58:46.464 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:46 smithi103 ceph-mon[31370]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-05-22T23:58:46.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 -- 172.21.15.103:0/3618723615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201024e0 msgr2=0x7f65201028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 --2- 172.21.15.103:0/3618723615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201024e0 0x7f65201028c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6510009990 tx=0x7f6510009ca0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 -- 172.21.15.103:0/3618723615 shutdown_connections 2024-05-22T23:58:46.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 --2- 172.21.15.103:0/3618723615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201024e0 0x7f65201028c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 --2- 172.21.15.103:0/3618723615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65201075c0 0x7f652010f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 --2- 172.21.15.103:0/3618723615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6520102e90 0x7f6520106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 -- 172.21.15.103:0/3618723615 >> 172.21.15.103:0/3618723615 conn(0x7f65200fddb0 msgr2=0x7f65201001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 -- 172.21.15.103:0/3618723615 shutdown_connections 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.355+0000 7f6524e00700 1 -- 172.21.15.103:0/3618723615 wait complete. 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.356+0000 7f6524e00700 1 Processor -- start 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.356+0000 7f6524e00700 1 -- start start 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f6524e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65201024e0 0x7f652019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f6524e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6520102e90 0x7f652019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f6524e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201075c0 0x7f6520198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f6524e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f65201125d0 con 0x7f6520102e90 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f6524e00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6520112450 con 0x7f65201075c0 2024-05-22T23:58:46.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f6524e00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6520112750 con 0x7f65201024e0 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f651fe00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201075c0 0x7f6520198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f651ea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6520102e90 0x7f652019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f651f400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65201024e0 0x7f652019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f651fe00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201075c0 0x7f6520198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:40962/0 (socket says 172.21.15.103:40962) 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.357+0000 7f651fe00700 1 -- 172.21.15.103:0/1578812448 learned_addr learned my addr 172.21.15.103:0/1578812448 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f651fe00700 1 -- 172.21.15.103:0/1578812448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65201024e0 msgr2=0x7f652019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f651fe00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65201024e0 0x7f652019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f651fe00700 1 -- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6520102e90 msgr2=0x7f652019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f651fe00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6520102e90 0x7f652019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f651fe00700 1 -- 172.21.15.103:0/1578812448 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6520198fd0 con 0x7f65201075c0 2024-05-22T23:58:46.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f651fe00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201075c0 0x7f6520198870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f651800ef30 tx=0x7f651800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65180092e0 con 0x7f65201075c0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6518004bb0 con 0x7f65201075c0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6520199260 con 0x7f65201075c0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.358+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518004280 con 0x7f65201075c0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.359+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65201a5580 con 0x7f65201075c0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.359+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6520062a30 con 0x7f65201075c0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.362+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f651801b030 con 0x7f65201075c0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.363+0000 7f6517e00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65040a2cb0 0x7f65040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.363+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f65180cf920 con 0x7f65201075c0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.363+0000 7f651f400700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65040a2cb0 0x7f65040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.363+0000 7f6517e00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] conn(0x7f65040ade10 0x7f65040b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.363+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 --> [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f651801fe50 con 0x7f65040ade10 2024-05-22T23:58:46.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.363+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f65180cfcd0 con 0x7f65201075c0 2024-05-22T23:58:46.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.364+0000 7f651ea00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] conn(0x7f65040ade10 0x7f65040b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:46.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.364+0000 7f651f400700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65040a2cb0 0x7f65040a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f651000b2e0 tx=0x7f651001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:46.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.364+0000 7f651ea00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] conn(0x7f65040ade10 0x7f65040b02b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:46.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.365+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 <== osd.3 v2:172.21.15.103:6810/29866944 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f651801fe50 con 0x7f65040ade10 2024-05-22T23:58:46.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.384+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 --> [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6520062eb0 con 0x7f65040ade10 2024-05-22T23:58:46.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.384+0000 7f6517e00700 1 -- 172.21.15.103:0/1578812448 <== osd.3 v2:172.21.15.103:6810/29866944 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6520062eb0 con 0x7f65040ade10 2024-05-22T23:58:46.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] conn(0x7f65040ade10 msgr2=0x7f65040b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] conn(0x7f65040ade10 0x7f65040b02b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65040a2cb0 msgr2=0x7f65040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65040a2cb0 0x7f65040a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f651000b2e0 tx=0x7f651001a040 comp rx=0 tx=0).stop 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201075c0 msgr2=0x7f6520198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 --2- 172.21.15.103:0/1578812448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65201075c0 0x7f6520198870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f651800ef30 tx=0x7f651800c3a0 comp rx=0 tx=0).stop 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f651fe00700 1 -- 172.21.15.103:0/1578812448 reap_dead start 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 shutdown_connections 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 >> 172.21.15.103:0/1578812448 conn(0x7f65200fddb0 msgr2=0x7f65200ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.385+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 shutdown_connections 2024-05-22T23:58:46.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:45.386+0000 7f6524e00700 1 -- 172.21.15.103:0/1578812448 wait complete. 2024-05-22T23:58:46.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:46 smithi105 ceph-mon[36426]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-05-22T23:58:46.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:46 smithi174 ceph-mon[36179]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 84 B/s, 0 objects/s recovering 2024-05-22T23:58:47.169 INFO:teuthology.orchestra.run.smithi103.stdout:146028888132 2024-05-22T23:58:47.169 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-22T23:58:47.993 INFO:teuthology.orchestra.run.smithi103.stdout:352187318300 2024-05-22T23:58:47.994 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-22T23:58:48.456 INFO:teuthology.orchestra.run.smithi103.stdout:171798691905 2024-05-22T23:58:48.457 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-22T23:58:48.510 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:48 smithi103 ceph-mon[31370]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:58:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:48 smithi105 ceph-mon[36426]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:58:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:48 smithi174 ceph-mon[36179]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:58:49.804 INFO:teuthology.orchestra.run.smithi103.stdout:111669149770 2024-05-22T23:58:49.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-22T23:58:50.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:50 smithi103 ceph-mon[31370]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:58:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:50 smithi105 ceph-mon[36426]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:58:50.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:50 smithi174 ceph-mon[36179]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:58:51.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.321+0000 7ff38ba00700 1 -- 172.21.15.103:0/46630936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102610 msgr2=0x7ff3841029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.321+0000 7ff38ba00700 1 --2- 172.21.15.103:0/46630936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102610 0x7ff3841029f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff378009990 tx=0x7ff378009ca0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.321+0000 7ff38ba00700 1 -- 172.21.15.103:0/46630936 shutdown_connections 2024-05-22T23:58:51.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.321+0000 7ff38ba00700 1 --2- 172.21.15.103:0/46630936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3841076a0 0x7ff38410f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.321+0000 7ff38ba00700 1 --2- 172.21.15.103:0/46630936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102610 0x7ff3841029f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.321+0000 7ff38ba00700 1 --2- 172.21.15.103:0/46630936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff384102fc0 0x7ff384106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.321+0000 7ff38ba00700 1 -- 172.21.15.103:0/46630936 >> 172.21.15.103:0/46630936 conn(0x7ff3840794c0 msgr2=0x7ff3840798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:51.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.322+0000 7ff38ba00700 1 -- 172.21.15.103:0/46630936 shutdown_connections 2024-05-22T23:58:51.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.322+0000 7ff38ba00700 1 -- 172.21.15.103:0/46630936 wait complete. 2024-05-22T23:58:51.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.323+0000 7ff38ba00700 1 Processor -- start 2024-05-22T23:58:51.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.323+0000 7ff38ba00700 1 -- start start 2024-05-22T23:58:51.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.323+0000 7ff38ba00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff384102610 0x7ff38419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.323+0000 7ff38ba00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102fc0 0x7ff38419b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.323+0000 7ff38ba00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3841076a0 0x7ff38419f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.323+0000 7ff38ba00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff3841126e0 con 0x7ff384102fc0 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.323+0000 7ff38ba00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff384112560 con 0x7ff384102610 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.323+0000 7ff38ba00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff384112860 con 0x7ff3841076a0 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff383e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102fc0 0x7ff38419b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff383e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102fc0 0x7ff38419b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:58560/0 (socket says 172.21.15.103:58560) 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff383e00700 1 -- 172.21.15.103:0/1337863619 learned_addr learned my addr 172.21.15.103:0/1337863619 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff389000700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff384102610 0x7ff38419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff389a00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3841076a0 0x7ff38419f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff383e00700 1 -- 172.21.15.103:0/1337863619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3841076a0 msgr2=0x7ff38419f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff383e00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3841076a0 0x7ff38419f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff383e00700 1 -- 172.21.15.103:0/1337863619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff384102610 msgr2=0x7ff38419ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff383e00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff384102610 0x7ff38419ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff383e00700 1 -- 172.21.15.103:0/1337863619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38419fda0 con 0x7ff384102fc0 2024-05-22T23:58:51.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.324+0000 7ff389000700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff384102610 0x7ff38419ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:51.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.325+0000 7ff383e00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102fc0 0x7ff38419b2d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff37000e970 tx=0x7ff37000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:51.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.325+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37000cbd0 con 0x7ff384102fc0 2024-05-22T23:58:51.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.325+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3841a0090 con 0x7ff384102fc0 2024-05-22T23:58:51.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.325+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3841a7910 con 0x7ff384102fc0 2024-05-22T23:58:51.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.325+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff37000cd70 con 0x7ff384102fc0 2024-05-22T23:58:51.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.325+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff370005660 con 0x7ff384102fc0 2024-05-22T23:58:51.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.326+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff364001030 con 0x7ff384102fc0 2024-05-22T23:58:51.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.329+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff370004750 con 0x7ff384102fc0 2024-05-22T23:58:51.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.331+0000 7ff381600700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff36c0a2d30 0x7ff36c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.331+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff3700cfaa0 con 0x7ff384102fc0 2024-05-22T23:58:51.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.331+0000 7ff389000700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff36c0a2d30 0x7ff36c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.331+0000 7ff381600700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] conn(0x7ff36c0ade90 0x7ff36c0b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.331+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 --> [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff37001ee50 con 0x7ff36c0ade90 2024-05-22T23:58:51.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.331+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7ff3700cfee0 con 0x7ff384102fc0 2024-05-22T23:58:51.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.332+0000 7ff389a00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] conn(0x7ff36c0ade90 0x7ff36c0b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.332+0000 7ff389000700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff36c0a2d30 0x7ff36c0a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff378006010 tx=0x7ff37800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:51.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.332+0000 7ff389a00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] conn(0x7ff36c0ade90 0x7ff36c0b0330 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:51.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.333+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 <== osd.9 v2:172.21.15.103:6826/3886855673 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff37001ee50 con 0x7ff36c0ade90 2024-05-22T23:58:51.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.351+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 --> [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff364002e60 con 0x7ff36c0ade90 2024-05-22T23:58:51.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff381600700 1 -- 172.21.15.103:0/1337863619 <== osd.9 v2:172.21.15.103:6826/3886855673 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff364002e60 con 0x7ff36c0ade90 2024-05-22T23:58:51.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] conn(0x7ff36c0ade90 msgr2=0x7ff36c0b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff38ba00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] conn(0x7ff36c0ade90 0x7ff36c0b0330 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff36c0a2d30 msgr2=0x7ff36c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff38ba00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff36c0a2d30 0x7ff36c0a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff378006010 tx=0x7ff37800b260 comp rx=0 tx=0).stop 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102fc0 msgr2=0x7ff38419b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff38ba00700 1 --2- 172.21.15.103:0/1337863619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff384102fc0 0x7ff38419b2d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff37000e970 tx=0x7ff37000ec80 comp rx=0 tx=0).stop 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff389a00700 1 -- 172.21.15.103:0/1337863619 reap_dead start 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 shutdown_connections 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.352+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 >> 172.21.15.103:0/1337863619 conn(0x7ff3840794c0 msgr2=0x7ff384101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.353+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 shutdown_connections 2024-05-22T23:58:51.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.353+0000 7ff38ba00700 1 -- 172.21.15.103:0/1337863619 wait complete. 2024-05-22T23:58:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 -- 172.21.15.103:0/3528960239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3234074680 msgr2=0x7f3234074a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 --2- 172.21.15.103:0/3528960239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3234074680 0x7f3234074a60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f32240098f0 tx=0x7f3224009c00 comp rx=0 tx=0).stop 2024-05-22T23:58:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 -- 172.21.15.103:0/3528960239 shutdown_connections 2024-05-22T23:58:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 --2- 172.21.15.103:0/3528960239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3234074680 0x7f3234074a60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 --2- 172.21.15.103:0/3528960239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f323410eb40 0x7f3234110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 --2- 172.21.15.103:0/3528960239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234074fa0 0x7f323410e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 -- 172.21.15.103:0/3528960239 >> 172.21.15.103:0/3528960239 conn(0x7f32340fddb0 msgr2=0x7f32341001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 -- 172.21.15.103:0/3528960239 shutdown_connections 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.031+0000 7f3233e00700 1 -- 172.21.15.103:0/3528960239 wait complete. 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.032+0000 7f3233e00700 1 Processor -- start 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.032+0000 7f3233e00700 1 -- start start 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f3233e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234074680 0x7f323419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f3233e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3234074fa0 0x7f323419b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f3233e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f323410eb40 0x7f323419f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f3233e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32341147d0 con 0x7f3234074680 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f3233e00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3234114650 con 0x7f323410eb40 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f3233e00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3234114950 con 0x7f3234074fa0 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f322be00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3234074fa0 0x7f323419b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.447 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f322be00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3234074fa0 0x7f323419b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:60846/0 (socket says 172.21.15.103:60846) 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f322be00700 1 -- 172.21.15.103:0/244424899 learned_addr learned my addr 172.21.15.103:0/244424899 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f3232a00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234074680 0x7f323419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f3233400700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f323410eb40 0x7f323419f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f322be00700 1 -- 172.21.15.103:0/244424899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f323410eb40 msgr2=0x7f323419f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.033+0000 7f322be00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f323410eb40 0x7f323419f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f322be00700 1 -- 172.21.15.103:0/244424899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234074680 msgr2=0x7f323419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f322be00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3234074680 0x7f323419ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f322be00700 1 -- 172.21.15.103:0/244424899 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323419fd70 con 0x7f3234074fa0 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f322be00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3234074fa0 0x7f323419b2a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f321c00e9a0 tx=0x7f321c00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c00ca10 con 0x7f3234074fa0 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32341a0060 con 0x7f3234074fa0 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f321c00cbb0 con 0x7f3234074fa0 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f321c018680 con 0x7f3234074fa0 2024-05-22T23:58:51.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.034+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32341a78e0 con 0x7f3234074fa0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.035+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3234062a70 con 0x7f3234074fa0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.038+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f321c010430 con 0x7f3234074fa0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.039+0000 7f322b400700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f32180a2cb0 0x7f32180a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.039+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f321c0cffb0 con 0x7f3234074fa0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.039+0000 7f3232a00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f32180a2cb0 0x7f32180a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.039+0000 7f322b400700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] conn(0x7f32180ade10 0x7f32180b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.039+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 --> [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f321c01bec0 con 0x7f32180ade10 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.039+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f321c0d0360 con 0x7f3234074fa0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.040+0000 7f3233400700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] conn(0x7f32180ade10 0x7f32180b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.040+0000 7f3232a00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f32180a2cb0 0x7f32180a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3224009810 tx=0x7f3224019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.040+0000 7f3233400700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] conn(0x7f32180ade10 0x7f32180b02b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.041+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 <== osd.7 v2:172.21.15.105:6816/4084427396 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f321c01bec0 con 0x7f32180ade10 2024-05-22T23:58:51.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.073+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 --> [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3234062e70 con 0x7f32180ade10 2024-05-22T23:58:51.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.074+0000 7f322b400700 1 -- 172.21.15.103:0/244424899 <== osd.7 v2:172.21.15.105:6816/4084427396 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3234062e70 con 0x7f32180ade10 2024-05-22T23:58:51.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.074+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 >> [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] conn(0x7f32180ade10 msgr2=0x7f32180b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.074+0000 7f3233e00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] conn(0x7f32180ade10 0x7f32180b02b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f32180a2cb0 msgr2=0x7f32180a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233e00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f32180a2cb0 0x7f32180a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3224009810 tx=0x7f3224019040 comp rx=0 tx=0).stop 2024-05-22T23:58:51.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3234074fa0 msgr2=0x7f323419b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:51.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233e00700 1 --2- 172.21.15.103:0/244424899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3234074fa0 0x7f323419b2a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f321c00e9a0 tx=0x7f321c00ecb0 comp rx=0 tx=0).stop 2024-05-22T23:58:51.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233400700 1 -- 172.21.15.103:0/244424899 reap_dead start 2024-05-22T23:58:51.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 shutdown_connections 2024-05-22T23:58:51.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 >> 172.21.15.103:0/244424899 conn(0x7f32340fddb0 msgr2=0x7f32340febc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:51.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 shutdown_connections 2024-05-22T23:58:51.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:51.075+0000 7f3233e00700 1 -- 172.21.15.103:0/244424899 wait complete. 2024-05-22T23:58:52.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:52 smithi105 ceph-mon[36426]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:58:52.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:52 smithi103 ceph-mon[31370]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:58:52.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:52 smithi174 ceph-mon[36179]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:58:52.874 INFO:teuthology.orchestra.run.smithi103.stdout:227633266741 2024-05-22T23:58:52.874 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-22T23:58:53.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.246+0000 7f7285c00700 1 -- 172.21.15.103:0/1530352999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7280074590 msgr2=0x7f7280074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:53.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.246+0000 7f7285c00700 1 --2- 172.21.15.103:0/1530352999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7280074590 0x7f7280074970 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7270009990 tx=0x7f7270009ca0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.246+0000 7f7285c00700 1 -- 172.21.15.103:0/1530352999 shutdown_connections 2024-05-22T23:58:53.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.246+0000 7f7285c00700 1 --2- 172.21.15.103:0/1530352999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f728010edc0 0x7f72801111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.246+0000 7f7285c00700 1 --2- 172.21.15.103:0/1530352999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7280074590 0x7f7280074970 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.246+0000 7f7285c00700 1 --2- 172.21.15.103:0/1530352999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7280074eb0 0x7f728010e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.246+0000 7f7285c00700 1 -- 172.21.15.103:0/1530352999 >> 172.21.15.103:0/1530352999 conn(0x7f72800794c0 msgr2=0x7f72800798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:53.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.246+0000 7f7285c00700 1 -- 172.21.15.103:0/1530352999 shutdown_connections 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.247+0000 7f7285c00700 1 -- 172.21.15.103:0/1530352999 wait complete. 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.248+0000 7f7285c00700 1 Processor -- start 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.248+0000 7f7285c00700 1 -- start start 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f7285c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7280074590 0x7f72801a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f7285c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7280074eb0 0x7f72801a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f7285c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728010edc0 0x7f728019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f7285c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7280114920 con 0x7f728010edc0 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f7285c00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f72801147a0 con 0x7f7280074590 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f7285c00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7280114aa0 con 0x7f7280074eb0 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728010edc0 0x7f728019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727ea00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7280074590 0x7f72801a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:53.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727e000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7280074eb0 0x7f72801a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727ea00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7280074590 0x7f72801a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:41002/0 (socket says 172.21.15.103:41002) 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727ea00700 1 -- 172.21.15.103:0/1271939771 learned_addr learned my addr 172.21.15.103:0/1271939771 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727ea00700 1 -- 172.21.15.103:0/1271939771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7280074eb0 msgr2=0x7f72801a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727ea00700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7280074eb0 0x7f72801a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727ea00700 1 -- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728010edc0 msgr2=0x7f728019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.249+0000 7f727ea00700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f728010edc0 0x7f728019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.250+0000 7f727ea00700 1 -- 172.21.15.103:0/1271939771 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f728019b200 con 0x7f7280074590 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.250+0000 7f727ea00700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7280074590 0x7f72801a08e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7270005910 tx=0x7f727000bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.250+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7270027070 con 0x7f7280074590 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.250+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f728019b490 con 0x7f7280074590 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.250+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7270003d70 con 0x7f7280074590 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.250+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727000f460 con 0x7f7280074590 2024-05-22T23:58:53.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.250+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72801a77b0 con 0x7f7280074590 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.251+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f72800fff50 con 0x7f7280074590 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.253+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f727002a070 con 0x7f7280074590 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.255+0000 7f7277400700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72640a2d30 0x7f72640a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.255+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f72700d0a70 con 0x7f7280074590 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.255+0000 7f727e000700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72640a2d30 0x7f72640a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.255+0000 7f7277400700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] conn(0x7f72640ade90 0x7f72640b0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.255+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 --> [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72640b0a00 con 0x7f72640ade90 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.255+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f72700d0e20 con 0x7f7280074590 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.256+0000 7f727f400700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] conn(0x7f72640ade90 0x7f72640b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.256+0000 7f727e000700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72640a2d30 0x7f72640a51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7280110930 tx=0x7f726c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.256+0000 7f727f400700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] conn(0x7f72640ade90 0x7f72640b0330 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.257+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 <== osd.0 v2:172.21.15.103:6802/2549120661 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f72640b0a00 con 0x7f72640ade90 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.275+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 --> [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7280075c50 con 0x7f72640ade90 2024-05-22T23:58:53.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.275+0000 7f7277400700 1 -- 172.21.15.103:0/1271939771 <== osd.0 v2:172.21.15.103:6802/2549120661 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7280075c50 con 0x7f72640ade90 2024-05-22T23:58:53.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] conn(0x7f72640ade90 msgr2=0x7f72640b0330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:53.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] conn(0x7f72640ade90 0x7f72640b0330 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72640a2d30 msgr2=0x7f72640a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:53.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f72640a2d30 0x7f72640a51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7280110930 tx=0x7f726c0073d0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7280074590 msgr2=0x7f72801a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:53.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 --2- 172.21.15.103:0/1271939771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7280074590 0x7f72801a08e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7270005910 tx=0x7f727000bab0 comp rx=0 tx=0).stop 2024-05-22T23:58:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f727f400700 1 -- 172.21.15.103:0/1271939771 reap_dead start 2024-05-22T23:58:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 shutdown_connections 2024-05-22T23:58:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 >> 172.21.15.103:0/1271939771 conn(0x7f72800794c0 msgr2=0x7f72800ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 shutdown_connections 2024-05-22T23:58:53.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:53.276+0000 7f7285c00700 1 -- 172.21.15.103:0/1271939771 wait complete. 2024-05-22T23:58:54.407 INFO:teuthology.orchestra.run.smithi103.stdout:34359738459 2024-05-22T23:58:54.407 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-22T23:58:54.479 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:54 smithi103 ceph-mon[31370]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:58:54.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:54 smithi105 ceph-mon[36426]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:58:54.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:54 smithi174 ceph-mon[36179]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:58:55.278 INFO:teuthology.orchestra.run.smithi103.stdout:296352743464 2024-05-22T23:58:55.278 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-22T23:58:55.289 INFO:teuthology.orchestra.run.smithi103.stdout:77309411407 2024-05-22T23:58:55.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-22T23:58:55.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.176+0000 7ff03be00700 1 -- 172.21.15.103:0/2066928145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff03c1024f0 msgr2=0x7ff03c1028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:55.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.176+0000 7ff03be00700 1 --2- 172.21.15.103:0/2066928145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff03c1024f0 0x7ff03c1028d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009920 tx=0x7ff02c009c30 comp rx=0 tx=0).stop 2024-05-22T23:58:55.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.176+0000 7ff03be00700 1 -- 172.21.15.103:0/2066928145 shutdown_connections 2024-05-22T23:58:55.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.176+0000 7ff03be00700 1 --2- 172.21.15.103:0/2066928145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff03c1075d0 0x7ff03c10f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:55.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.176+0000 7ff03be00700 1 --2- 172.21.15.103:0/2066928145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff03c1024f0 0x7ff03c1028d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:55.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.176+0000 7ff03be00700 1 --2- 172.21.15.103:0/2066928145 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff03c102ea0 0x7ff03c106ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:55.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.176+0000 7ff03be00700 1 -- 172.21.15.103:0/2066928145 >> 172.21.15.103:0/2066928145 conn(0x7ff03c0fdde0 msgr2=0x7ff03c100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.176+0000 7ff03be00700 1 -- 172.21.15.103:0/2066928145 shutdown_connections 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.177+0000 7ff03be00700 1 -- 172.21.15.103:0/2066928145 wait complete. 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.178+0000 7ff03be00700 1 Processor -- start 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.178+0000 7ff03be00700 1 -- start start 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff03be00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff03c1024f0 0x7ff03c074660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff03be00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff03c102ea0 0x7ff03c074ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff03be00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff03c1075d0 0x7ff03c0750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff03be00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c112620 con 0x7ff03c102ea0 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff03be00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1124a0 con 0x7ff03c1075d0 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff03be00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff03c1127a0 con 0x7ff03c1024f0 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff032000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff03c102ea0 0x7ff03c074ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:55.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff032000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff03c102ea0 0x7ff03c074ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54050/0 (socket says 172.21.15.103:54050) 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff032000700 1 -- 172.21.15.103:0/1169317958 learned_addr learned my addr 172.21.15.103:0/1169317958 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff03aa00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff03c1024f0 0x7ff03c074660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.179+0000 7ff03b400700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff03c1075d0 0x7ff03c0750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff032000700 1 -- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff03c1024f0 msgr2=0x7ff03c074660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff032000700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff03c1024f0 0x7ff03c074660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff032000700 1 -- 172.21.15.103:0/1169317958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff03c1075d0 msgr2=0x7ff03c0750e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff032000700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff03c1075d0 0x7ff03c0750e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff032000700 1 -- 172.21.15.103:0/1169317958 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff03c1a7050 con 0x7ff03c102ea0 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff03aa00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff03c1024f0 0x7ff03c074660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff03b400700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff03c1075d0 0x7ff03c0750e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff032000700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff03c102ea0 0x7ff03c074ba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff02400e9f0 tx=0x7ff02400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:55.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024004a30 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff02400cca0 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff03c1a7340 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.180+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0240056c0 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.181+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03c1a7830 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.182+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff03c062a70 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.184+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff02400c7c0 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.185+0000 7ff033e00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff0200a2cb0 0x7ff0200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.185+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff0240cfb60 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.185+0000 7ff03aa00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff0200a2cb0 0x7ff0200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.186+0000 7ff033e00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] conn(0x7ff0200ade10 0x7ff0200b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.186+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 --> [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff02400fe50 con 0x7ff0200ade10 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.186+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7ff0240cff10 con 0x7ff03c102ea0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.186+0000 7ff03aa00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff0200a2cb0 0x7ff0200a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0098f0 tx=0x7ff02c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:55.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.186+0000 7ff03b400700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] conn(0x7ff0200ade10 0x7ff0200b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:55.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.187+0000 7ff03b400700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] conn(0x7ff0200ade10 0x7ff0200b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:55.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.188+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 <== osd.8 v2:172.21.15.174:6816/1749666610 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff02400fe50 con 0x7ff0200ade10 2024-05-22T23:58:55.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.206+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 --> [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff03c062ef0 con 0x7ff0200ade10 2024-05-22T23:58:55.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.207+0000 7ff033e00700 1 -- 172.21.15.103:0/1169317958 <== osd.8 v2:172.21.15.174:6816/1749666610 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff03c062ef0 con 0x7ff0200ade10 2024-05-22T23:58:55.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.207+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] conn(0x7ff0200ade10 msgr2=0x7ff0200b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:55.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.207+0000 7ff03be00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] conn(0x7ff0200ade10 0x7ff0200b02b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:55.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.208+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff0200a2cb0 msgr2=0x7ff0200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:55.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.208+0000 7ff03be00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff0200a2cb0 0x7ff0200a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0098f0 tx=0x7ff02c019040 comp rx=0 tx=0).stop 2024-05-22T23:58:55.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.208+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff03c102ea0 msgr2=0x7ff03c074ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:55.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.208+0000 7ff03be00700 1 --2- 172.21.15.103:0/1169317958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff03c102ea0 0x7ff03c074ba0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff02400e9f0 tx=0x7ff02400ed00 comp rx=0 tx=0).stop 2024-05-22T23:58:55.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.208+0000 7ff03b400700 1 -- 172.21.15.103:0/1169317958 reap_dead start 2024-05-22T23:58:55.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.209+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 shutdown_connections 2024-05-22T23:58:55.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.209+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 >> 172.21.15.103:0/1169317958 conn(0x7ff03c0fdde0 msgr2=0x7ff03c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:55.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.209+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 shutdown_connections 2024-05-22T23:58:55.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:54.209+0000 7ff03be00700 1 -- 172.21.15.103:0/1169317958 wait complete. 2024-05-22T23:58:56.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.447+0000 7f4b5c800700 1 -- 172.21.15.103:0/1539802981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b541098a0 msgr2=0x7f4b54109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:56.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.447+0000 7f4b5c800700 1 --2- 172.21.15.103:0/1539802981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b541098a0 0x7f4b54109d20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009990 tx=0x7f4b44009ca0 comp rx=0 tx=0).stop 2024-05-22T23:58:56.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.447+0000 7f4b5c800700 1 -- 172.21.15.103:0/1539802981 shutdown_connections 2024-05-22T23:58:56.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.447+0000 7f4b5c800700 1 --2- 172.21.15.103:0/1539802981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b541173f0 0x7f4b541197e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:56.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.447+0000 7f4b5c800700 1 --2- 172.21.15.103:0/1539802981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b5410ca50 0x7f4b54109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:56.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.447+0000 7f4b5c800700 1 --2- 172.21.15.103:0/1539802981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b541098a0 0x7f4b54109d20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:56.452 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.447+0000 7f4b5c800700 1 -- 172.21.15.103:0/1539802981 >> 172.21.15.103:0/1539802981 conn(0x7f4b5406d8b0 msgr2=0x7f4b5406dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:56.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.448+0000 7f4b5c800700 1 -- 172.21.15.103:0/1539802981 shutdown_connections 2024-05-22T23:58:56.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.448+0000 7f4b5c800700 1 -- 172.21.15.103:0/1539802981 wait complete. 2024-05-22T23:58:56.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.449+0000 7f4b5c800700 1 Processor -- start 2024-05-22T23:58:56.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.449+0000 7f4b5c800700 1 -- start start 2024-05-22T23:58:56.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b5c800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b541098a0 0x7f4b541a3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:56.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b5c800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b5410ca50 0x7f4b541a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:56.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b5c800700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b541173f0 0x7f4b541a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:56.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b5c800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5411d130 con 0x7f4b541098a0 2024-05-22T23:58:56.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b5c800700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5411cfb0 con 0x7f4b5410ca50 2024-05-22T23:58:56.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b5c800700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4b5411d2b0 con 0x7f4b541173f0 2024-05-22T23:58:56.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b59400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b5410ca50 0x7f4b541a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:56.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b59e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b541098a0 0x7f4b541a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:56.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b59400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b5410ca50 0x7f4b541a3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49230/0 (socket says 172.21.15.103:49230) 2024-05-22T23:58:56.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b59400700 1 -- 172.21.15.103:0/2761965418 learned_addr learned my addr 172.21.15.103:0/2761965418 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:58:56.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b5a800700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b541173f0 0x7f4b541a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:56.455 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b59400700 1 -- 172.21.15.103:0/2761965418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b541173f0 msgr2=0x7f4b541a7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:56.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.450+0000 7f4b59400700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b541173f0 0x7f4b541a7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:56.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b59400700 1 -- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b541098a0 msgr2=0x7f4b541a3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:56.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b59400700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b541098a0 0x7f4b541a3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:56.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b59400700 1 -- 172.21.15.103:0/2761965418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b541a8590 con 0x7f4b5410ca50 2024-05-22T23:58:56.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b59e00700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b541098a0 0x7f4b541a3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:58:56.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b59400700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b5410ca50 0x7f4b541a3ac0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4b44005910 tx=0x7f4b4400bb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:56.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44026070 con 0x7f4b5410ca50 2024-05-22T23:58:56.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b541a8820 con 0x7f4b5410ca50 2024-05-22T23:58:56.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b44003d70 con 0x7f4b5410ca50 2024-05-22T23:58:56.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.451+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b44021410 con 0x7f4b5410ca50 2024-05-22T23:58:56.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.452+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b541b0100 con 0x7f4b5410ca50 2024-05-22T23:58:56.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.453+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4b54119540 con 0x7f4b5410ca50 2024-05-22T23:58:56.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.455+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f4b44029070 con 0x7f4b5410ca50 2024-05-22T23:58:56.459 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.456+0000 7f4b4ea00700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4b3c0a2cf0 0x7f4b3c0a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:56.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.456+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4b440d03e0 con 0x7f4b5410ca50 2024-05-22T23:58:56.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.456+0000 7f4b59e00700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4b3c0a2cf0 0x7f4b3c0a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:56.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.457+0000 7f4b4ea00700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] conn(0x7f4b3c0ade50 0x7f4b3c0b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:58:56.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.457+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 --> [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b440043d0 con 0x7f4b3c0ade50 2024-05-22T23:58:56.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.457+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=93) ==== 24+0+0 (secure 0 0 0) 0x7f4b440d7050 con 0x7f4b5410ca50 2024-05-22T23:58:56.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.457+0000 7f4b5a800700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] conn(0x7f4b3c0ade50 0x7f4b3c0b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:58:56.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.457+0000 7f4b59e00700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4b3c0a2cf0 0x7f4b3c0a51b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009cb0 tx=0x7f4b48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:56.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.458+0000 7f4b5a800700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] conn(0x7f4b3c0ade50 0x7f4b3c0b02f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:58:56.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.459+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 <== osd.6 v2:172.21.15.103:6818/1312276577 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b440043d0 con 0x7f4b3c0ade50 2024-05-22T23:58:56.463 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:56 smithi103 ceph-mon[31370]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:58:56.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.477+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 --> [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b5406ea50 con 0x7f4b3c0ade50 2024-05-22T23:58:56.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.477+0000 7f4b4ea00700 1 -- 172.21.15.103:0/2761965418 <== osd.6 v2:172.21.15.103:6818/1312276577 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4b5406ea50 con 0x7f4b3c0ade50 2024-05-22T23:58:56.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] conn(0x7f4b3c0ade50 msgr2=0x7f4b3c0b02f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:56.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] conn(0x7f4b3c0ade50 0x7f4b3c0b02f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:58:56.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4b3c0a2cf0 msgr2=0x7f4b3c0a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:56.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4b3c0a2cf0 0x7f4b3c0a51b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f4b48009cb0 tx=0x7f4b48009260 comp rx=0 tx=0).stop 2024-05-22T23:58:56.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b5410ca50 msgr2=0x7f4b541a3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:58:56.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 --2- 172.21.15.103:0/2761965418 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b5410ca50 0x7f4b541a3ac0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4b44005910 tx=0x7f4b4400bb60 comp rx=0 tx=0).stop 2024-05-22T23:58:56.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5a800700 1 -- 172.21.15.103:0/2761965418 reap_dead start 2024-05-22T23:58:56.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 shutdown_connections 2024-05-22T23:58:56.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 >> 172.21.15.103:0/2761965418 conn(0x7f4b5406d8b0 msgr2=0x7f4b54118ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:58:56.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 shutdown_connections 2024-05-22T23:58:56.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:58:56.478+0000 7f4b5c800700 1 -- 172.21.15.103:0/2761965418 wait complete. 2024-05-22T23:58:56.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:56 smithi105 ceph-mon[36426]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:58:56.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:56 smithi174 ceph-mon[36179]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:58:57.332 INFO:teuthology.orchestra.run.smithi103.stdout:261993005103 2024-05-22T23:58:57.332 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-22T23:58:57.407 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:57.408 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:57.413 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:57.416 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:57.548 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:58:58.124 INFO:teuthology.orchestra.run.smithi103.stdout:206158430266 2024-05-22T23:58:58.124 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-22T23:58:58.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:58:58 smithi103 ceph-mon[31370]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-05-22T23:58:58.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:58:58 smithi105 ceph-mon[36426]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-05-22T23:58:58.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:58:58 smithi174 ceph-mon[36179]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-05-22T23:59:00.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:00 smithi105 ceph-mon[36426]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:00.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:00 smithi103 ceph-mon[31370]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:00.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:00 smithi174 ceph-mon[36179]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:01.138 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:02.430 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:02 smithi103 ceph-mon[31370]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:02.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:02 smithi105 ceph-mon[36426]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:02.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:02 smithi174 ceph-mon[36179]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:04.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:04 smithi105 ceph-mon[36426]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:04.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:04 smithi103 ceph-mon[31370]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:04.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:04 smithi174 ceph-mon[36179]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:04.633 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:06.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:05 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1534624970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-22T23:59:06.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:05 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1534624970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-22T23:59:06.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:05 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1534624970' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-22T23:59:07.006 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:06 smithi103 ceph-mon[31370]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:07.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:06 smithi105 ceph-mon[36426]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:07.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:06 smithi174 ceph-mon[36179]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:07.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 -- 172.21.15.103:0/1526453686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9441088a0 msgr2=0x7fb944108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:07.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 --2- 172.21.15.103:0/1526453686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9441088a0 0x7fb944108d20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb930009990 tx=0x7fb930009ca0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 -- 172.21.15.103:0/1526453686 shutdown_connections 2024-05-22T23:59:07.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 --2- 172.21.15.103:0/1526453686 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb944109260 0x7fb944115760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 --2- 172.21.15.103:0/1526453686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb94410e890 0x7fb94410ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 --2- 172.21.15.103:0/1526453686 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9441088a0 0x7fb944108d20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 -- 172.21.15.103:0/1526453686 >> 172.21.15.103:0/1526453686 conn(0x7fb9440780c0 msgr2=0x7fb944107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:07.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 -- 172.21.15.103:0/1526453686 shutdown_connections 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.465+0000 7fb94a400700 1 -- 172.21.15.103:0/1526453686 wait complete. 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.467+0000 7fb94a400700 1 Processor -- start 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.467+0000 7fb94a400700 1 -- start start 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb94a400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9441088a0 0x7fb94419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb94a400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb944109260 0x7fb94419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb94a400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb94410e890 0x7fb9441a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb94a400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb944118be0 con 0x7fb944109260 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb94a400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb944118a60 con 0x7fb94410e890 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb94a400700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb944118d60 con 0x7fb9441088a0 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb949000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9441088a0 0x7fb94419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:07.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb949a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb94410e890 0x7fb9441a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb943e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb944109260 0x7fb94419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb949000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9441088a0 0x7fb94419f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:48202/0 (socket says 172.21.15.103:48202) 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb949a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb94410e890 0x7fb9441a3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:42272/0 (socket says 172.21.15.103:42272) 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.468+0000 7fb949000700 1 -- 172.21.15.103:0/1534624970 learned_addr learned my addr 172.21.15.103:0/1534624970 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.469+0000 7fb949000700 1 -- 172.21.15.103:0/1534624970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb94410e890 msgr2=0x7fb9441a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.469+0000 7fb949000700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb94410e890 0x7fb9441a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.469+0000 7fb949000700 1 -- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb944109260 msgr2=0x7fb94419f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.469+0000 7fb949000700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb944109260 0x7fb94419f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.469+0000 7fb949000700 1 -- 172.21.15.103:0/1534624970 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9441a3fd0 con 0x7fb9441088a0 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.469+0000 7fb949a00700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb94410e890 0x7fb9441a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:07.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.469+0000 7fb949000700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9441088a0 0x7fb94419f010 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb93800e9a0 tx=0x7fb93800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:07.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.469+0000 7fb941600700 1 -- 172.21.15.103:0/1534624970 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93800ca10 con 0x7fb9441088a0 2024-05-22T23:59:07.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.470+0000 7fb941600700 1 -- 172.21.15.103:0/1534624970 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb93800cbb0 con 0x7fb9441088a0 2024-05-22T23:59:07.243 INFO:teuthology.orchestra.run.smithi103.stdout:146028888136 2024-05-22T23:59:07.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.470+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9441a42c0 con 0x7fb9441088a0 2024-05-22T23:59:07.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.470+0000 7fb941600700 1 -- 172.21.15.103:0/1534624970 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938018680 con 0x7fb9441088a0 2024-05-22T23:59:07.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.470+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9441abb40 con 0x7fb9441088a0 2024-05-22T23:59:07.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.471+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb944069050 con 0x7fb9441088a0 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.473+0000 7fb941600700 1 -- 172.21.15.103:0/1534624970 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb938010430 con 0x7fb9441088a0 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.475+0000 7fb941600700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb92c0a2d30 0x7fb92c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.476+0000 7fb941600700 1 -- 172.21.15.103:0/1534624970 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb9380cfb20 con 0x7fb9441088a0 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.476+0000 7fb943e00700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb92c0a2d30 0x7fb92c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.476+0000 7fb943e00700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb92c0a2d30 0x7fb92c0a51f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb9300098e0 tx=0x7fb930005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.477+0000 7fb941600700 1 -- 172.21.15.103:0/1534624970 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb93809c560 con 0x7fb9441088a0 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.706+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb94402dbc0 con 0x7fb9441088a0 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.706+0000 7fb941600700 1 -- 172.21.15.103:0/1534624970 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb93809be60 con 0x7fb9441088a0 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb92c0a2d30 msgr2=0x7fb92c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb92c0a2d30 0x7fb92c0a51f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb9300098e0 tx=0x7fb930005960 comp rx=0 tx=0).stop 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9441088a0 msgr2=0x7fb94419f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:07.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9441088a0 0x7fb94419f010 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb93800e9a0 tx=0x7fb93800ecb0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 shutdown_connections 2024-05-22T23:59:07.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb94410e890 0x7fb9441a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb92c0a2d30 0x7fb92c0a51f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9441088a0 0x7fb94419f010 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 --2- 172.21.15.103:0/1534624970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb944109260 0x7fb94419f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:07.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.709+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 >> 172.21.15.103:0/1534624970 conn(0x7fb9440780c0 msgr2=0x7fb9440785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:07.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.710+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 shutdown_connections 2024-05-22T23:59:07.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:05.710+0000 7fb94a400700 1 -- 172.21.15.103:0/1534624970 wait complete. 2024-05-22T23:59:07.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:07 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3817854440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-22T23:59:08.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:07 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3817854440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-22T23:59:08.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:07 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3817854440' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-22T23:59:08.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.311+0000 7fb30a600700 1 -- 172.21.15.103:0/2722673962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304108420 msgr2=0x7fb304108800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:08.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.311+0000 7fb30a600700 1 --2- 172.21.15.103:0/2722673962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304108420 0x7fb304108800 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2f40098f0 tx=0x7fb2f4009c00 comp rx=0 tx=0).stop 2024-05-22T23:59:08.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.311+0000 7fb30a600700 1 -- 172.21.15.103:0/2722673962 shutdown_connections 2024-05-22T23:59:08.659 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.311+0000 7fb30a600700 1 --2- 172.21.15.103:0/2722673962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb304102dd0 0x7fb30410f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.311+0000 7fb30a600700 1 --2- 172.21.15.103:0/2722673962 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304108420 0x7fb304108800 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.311+0000 7fb30a600700 1 --2- 172.21.15.103:0/2722673962 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb304102410 0x7fb304102890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.311+0000 7fb30a600700 1 -- 172.21.15.103:0/2722673962 >> 172.21.15.103:0/2722673962 conn(0x7fb3040794c0 msgr2=0x7fb3040798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.311+0000 7fb30a600700 1 -- 172.21.15.103:0/2722673962 shutdown_connections 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.312+0000 7fb30a600700 1 -- 172.21.15.103:0/2722673962 wait complete. 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.313+0000 7fb30a600700 1 Processor -- start 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.313+0000 7fb30a600700 1 -- start start 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb30a600700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304102410 0x7fb30419e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb30a600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb304102dd0 0x7fb30419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb30a600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb304108420 0x7fb304198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:08.660 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb30a600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb3041126a0 con 0x7fb304108420 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb30a600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb304112520 con 0x7fb304102dd0 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb30a600700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb304112820 con 0x7fb304102410 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb303e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb304108420 0x7fb304198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb303400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304102410 0x7fb30419e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb303e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb304108420 0x7fb304198920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46960/0 (socket says 172.21.15.103:46960) 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb303e00700 1 -- 172.21.15.103:0/3817854440 learned_addr learned my addr 172.21.15.103:0/3817854440 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.314+0000 7fb302a00700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb304102dd0 0x7fb30419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb303400700 1 -- 172.21.15.103:0/3817854440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb304102dd0 msgr2=0x7fb30419ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb303400700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb304102dd0 0x7fb30419ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb303400700 1 -- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb304108420 msgr2=0x7fb304198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb303400700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb304108420 0x7fb304198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb303400700 1 -- 172.21.15.103:0/3817854440 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb304199080 con 0x7fb304102410 2024-05-22T23:59:08.661 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb303400700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304102410 0x7fb30419e710 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4000c00 tx=0x7fb2f400ff20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:08.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb2fbe00700 1 -- 172.21.15.103:0/3817854440 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4005410 con 0x7fb304102410 2024-05-22T23:59:08.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb2fbe00700 1 -- 172.21.15.103:0/3817854440 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2f40055b0 con 0x7fb304102410 2024-05-22T23:59:08.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb2fbe00700 1 -- 172.21.15.103:0/3817854440 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4003d50 con 0x7fb304102410 2024-05-22T23:59:08.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb304199310 con 0x7fb304102410 2024-05-22T23:59:08.662 INFO:teuthology.orchestra.run.smithi103.stdout:111669149774 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.315+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb304074590 con 0x7fb304102410 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.316+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb304103ad0 con 0x7fb304102410 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.319+0000 7fb2fbe00700 1 -- 172.21.15.103:0/3817854440 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fb2f400bb60 con 0x7fb304102410 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.320+0000 7fb2fbe00700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb2e80a2cb0 0x7fb2e80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.320+0000 7fb2fbe00700 1 -- 172.21.15.103:0/3817854440 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb2f40cfe40 con 0x7fb304102410 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.320+0000 7fb302a00700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb2e80a2cb0 0x7fb2e80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.321+0000 7fb302a00700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb2e80a2cb0 0x7fb2e80a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009750 tx=0x7fb2f0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.322+0000 7fb2fbe00700 1 -- 172.21.15.103:0/3817854440 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb2f4013030 con 0x7fb304102410 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.552+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb30402dbc0 con 0x7fb304102410 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.553+0000 7fb2fbe00700 1 -- 172.21.15.103:0/3817854440 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fb2f409ca20 con 0x7fb304102410 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb2e80a2cb0 msgr2=0x7fb2e80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:08.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb2e80a2cb0 0x7fb2e80a5170 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0009750 tx=0x7fb2f0009310 comp rx=0 tx=0).stop 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304102410 msgr2=0x7fb30419e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304102410 0x7fb30419e710 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4000c00 tx=0x7fb2f400ff20 comp rx=0 tx=0).stop 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 shutdown_connections 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb2e80a2cb0 0x7fb2e80a5170 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb304108420 0x7fb304198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb304102410 0x7fb30419e710 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 --2- 172.21.15.103:0/3817854440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb304102dd0 0x7fb30419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.556+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 >> 172.21.15.103:0/3817854440 conn(0x7fb3040794c0 msgr2=0x7fb304101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.557+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 shutdown_connections 2024-05-22T23:59:08.664 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:07.557+0000 7fb30a600700 1 -- 172.21.15.103:0/3817854440 wait complete. 2024-05-22T23:59:08.931 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:08 smithi103 ceph-mon[31370]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:09.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:08 smithi105 ceph-mon[36426]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:09.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:08 smithi174 ceph-mon[36179]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 -- 172.21.15.103:0/2359695465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4109350 msgr2=0x7f62d4115860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 --2- 172.21.15.103:0/2359695465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4109350 0x7f62d4115860 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00b0a0 tx=0x7f62cc00b3b0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 -- 172.21.15.103:0/2359695465 shutdown_connections 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 --2- 172.21.15.103:0/2359695465 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4109350 0x7f62d4115860 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 --2- 172.21.15.103:0/2359695465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62d410e990 0x7f62d410ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 --2- 172.21.15.103:0/2359695465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62d4108990 0x7f62d4108e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 -- 172.21.15.103:0/2359695465 >> 172.21.15.103:0/2359695465 conn(0x7f62d4075b30 msgr2=0x7f62d4107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 -- 172.21.15.103:0/2359695465 shutdown_connections 2024-05-22T23:59:09.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.928+0000 7f62da200700 1 -- 172.21.15.103:0/2359695465 wait complete. 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.929+0000 7f62da200700 1 Processor -- start 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.929+0000 7f62da200700 1 -- start start 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62da200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4108990 0x7f62d419f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62da200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62d4109350 0x7f62d419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62da200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62d410e990 0x7f62d41a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62da200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f62d4118d20 con 0x7f62d4108990 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62da200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f62d4118ba0 con 0x7f62d410e990 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62da200700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62d4118ea0 con 0x7f62d4109350 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4108990 0x7f62d419f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4108990 0x7f62d419f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:46974/0 (socket says 172.21.15.103:46974) 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3400700 1 -- 172.21.15.103:0/3296407259 learned_addr learned my addr 172.21.15.103:0/3296407259 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:09.222 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d2a00700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62d4109350 0x7f62d419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3e00700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62d410e990 0x7f62d41a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3400700 1 -- 172.21.15.103:0/3296407259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62d4109350 msgr2=0x7f62d419f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3400700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62d4109350 0x7f62d419f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3400700 1 -- 172.21.15.103:0/3296407259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62d410e990 msgr2=0x7f62d41a3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3400700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62d410e990 0x7f62d41a3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3400700 1 -- 172.21.15.103:0/3296407259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62d41a40c0 con 0x7f62d4108990 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d3e00700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62d410e990 0x7f62d41a3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.930+0000 7f62d2a00700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62d4109350 0x7f62d419f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.931+0000 7f62d3400700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4108990 0x7f62d419f0b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f62c400e9a0 tx=0x7f62c400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.931+0000 7f62cbe00700 1 -- 172.21.15.103:0/3296407259 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c400cbd0 con 0x7f62d4108990 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.931+0000 7f62cbe00700 1 -- 172.21.15.103:0/3296407259 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62c400cd70 con 0x7f62d4108990 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.931+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62d41a43b0 con 0x7f62d4108990 2024-05-22T23:59:09.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.931+0000 7f62cbe00700 1 -- 172.21.15.103:0/3296407259 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4018750 con 0x7f62d4108990 2024-05-22T23:59:09.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.931+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62d41abc30 con 0x7f62d4108990 2024-05-22T23:59:09.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.932+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62d4069050 con 0x7f62d4108990 2024-05-22T23:59:09.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.934+0000 7f62cbe00700 1 -- 172.21.15.103:0/3296407259 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f62c4010430 con 0x7f62d4108990 2024-05-22T23:59:09.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.936+0000 7f62cbe00700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62b80a2d30 0x7f62b80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:09.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.936+0000 7f62cbe00700 1 -- 172.21.15.103:0/3296407259 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f62c40cf780 con 0x7f62d4108990 2024-05-22T23:59:09.224 INFO:teuthology.orchestra.run.smithi103.stdout:171798691909 2024-05-22T23:59:09.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.936+0000 7f62d2a00700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62b80a2d30 0x7f62b80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.937+0000 7f62d2a00700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62b80a2d30 0x7f62b80a51f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f62d4077910 tx=0x7f62c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:08.941+0000 7f62cbe00700 1 -- 172.21.15.103:0/3296407259 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f62c409c1c0 con 0x7f62d4108990 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.168+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f62d41abe60 con 0x7f62d4108990 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.169+0000 7f62cbe00700 1 -- 172.21.15.103:0/3296407259 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f62c409bac0 con 0x7f62d4108990 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62b80a2d30 msgr2=0x7f62b80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62b80a2d30 0x7f62b80a51f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f62d4077910 tx=0x7f62c0009260 comp rx=0 tx=0).stop 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4108990 msgr2=0x7f62d419f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4108990 0x7f62d419f0b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f62c400e9a0 tx=0x7f62c400ecb0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 shutdown_connections 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f62d410e990 0x7f62d41a3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f62b80a2d30 0x7f62b80a51f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f62d4108990 0x7f62d419f0b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.225 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 --2- 172.21.15.103:0/3296407259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62d4109350 0x7f62d419f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:09.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 >> 172.21.15.103:0/3296407259 conn(0x7f62d4075b30 msgr2=0x7f62d4075fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:09.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 shutdown_connections 2024-05-22T23:59:09.226 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:09.172+0000 7f62da200700 1 -- 172.21.15.103:0/3296407259 wait complete. 2024-05-22T23:59:09.969 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:09 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3296407259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-22T23:59:10.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:09 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3296407259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-22T23:59:10.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:09 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3296407259' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-22T23:59:10.154 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888132 got 146028888136 for osd.4 2024-05-22T23:59:10.154 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:10.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.701+0000 7ff873e00700 1 -- 172.21.15.103:0/4288622605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff874109260 msgr2=0x7ff874115700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:10.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.701+0000 7ff873e00700 1 --2- 172.21.15.103:0/4288622605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff874109260 0x7ff874115700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00b0a0 tx=0x7ff86c00b3b0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.701+0000 7ff873e00700 1 -- 172.21.15.103:0/4288622605 shutdown_connections 2024-05-22T23:59:10.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.701+0000 7ff873e00700 1 --2- 172.21.15.103:0/4288622605 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff87410d770 0x7ff87410db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.701+0000 7ff873e00700 1 --2- 172.21.15.103:0/4288622605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff874109260 0x7ff874115700 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.701+0000 7ff873e00700 1 --2- 172.21.15.103:0/4288622605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8741088a0 0x7ff874108d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.701+0000 7ff873e00700 1 -- 172.21.15.103:0/4288622605 >> 172.21.15.103:0/4288622605 conn(0x7ff8740780f0 msgr2=0x7ff874107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:10.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.701+0000 7ff873e00700 1 -- 172.21.15.103:0/4288622605 shutdown_connections 2024-05-22T23:59:10.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.702+0000 7ff873e00700 1 -- 172.21.15.103:0/4288622605 wait complete. 2024-05-22T23:59:10.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.703+0000 7ff873e00700 1 Processor -- start 2024-05-22T23:59:10.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.703+0000 7ff873e00700 1 -- start start 2024-05-22T23:59:10.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.703+0000 7ff873e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8741088a0 0x7ff87419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:10.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.703+0000 7ff873e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff874109260 0x7ff87419f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:10.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.703+0000 7ff873e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff87410d770 0x7ff8741a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:10.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.703+0000 7ff873e00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff874118c10 con 0x7ff8741088a0 2024-05-22T23:59:10.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.703+0000 7ff873e00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff874118a90 con 0x7ff874109260 2024-05-22T23:59:10.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.703+0000 7ff873e00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff874118d90 con 0x7ff87410d770 2024-05-22T23:59:10.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff872a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8741088a0 0x7ff87419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff873400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff87410d770 0x7ff8741a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff872000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff874109260 0x7ff87419f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff872a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8741088a0 0x7ff87419f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47000/0 (socket says 172.21.15.103:47000) 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff872a00700 1 -- 172.21.15.103:0/96539749 learned_addr learned my addr 172.21.15.103:0/96539749 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff873400700 1 -- 172.21.15.103:0/96539749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff874109260 msgr2=0x7ff87419f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff873400700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff874109260 0x7ff87419f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff873400700 1 -- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8741088a0 msgr2=0x7ff87419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff873400700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8741088a0 0x7ff87419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff873400700 1 -- 172.21.15.103:0/96539749 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8741a3fe0 con 0x7ff87410d770 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.704+0000 7ff873400700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff87410d770 0x7ff8741a38a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00b070 tx=0x7ff86c004c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.705+0000 7ff86b400700 1 -- 172.21.15.103:0/96539749 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c021070 con 0x7ff87410d770 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.705+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8741a4270 con 0x7ff87410d770 2024-05-22T23:59:10.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.705+0000 7ff86b400700 1 -- 172.21.15.103:0/96539749 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff86c007b80 con 0x7ff87410d770 2024-05-22T23:59:10.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.705+0000 7ff86b400700 1 -- 172.21.15.103:0/96539749 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86c005040 con 0x7ff87410d770 2024-05-22T23:59:10.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.705+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8741abb50 con 0x7ff87410d770 2024-05-22T23:59:10.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.706+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff874069050 con 0x7ff87410d770 2024-05-22T23:59:10.710 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.708+0000 7ff86b400700 1 -- 172.21.15.103:0/96539749 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7ff86c024070 con 0x7ff87410d770 2024-05-22T23:59:10.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.709+0000 7ff86b400700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff8580a2cb0 0x7ff8580a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:10.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.710+0000 7ff86b400700 1 -- 172.21.15.103:0/96539749 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7ff86c00f020 con 0x7ff87410d770 2024-05-22T23:59:10.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.710+0000 7ff872a00700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff8580a2cb0 0x7ff8580a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:10.713 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.710+0000 7ff872a00700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff8580a2cb0 0x7ff8580a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff864009ca0 tx=0x7ff864009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:10.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.712+0000 7ff86b400700 1 -- 172.21.15.103:0/96539749 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff86c02b020 con 0x7ff87410d770 2024-05-22T23:59:10.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.960+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7ff87402dbc0 con 0x7ff87410d770 2024-05-22T23:59:10.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.961+0000 7ff86b400700 1 -- 172.21.15.103:0/96539749 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ff86c007d30 con 0x7ff87410d770 2024-05-22T23:59:10.964 INFO:teuthology.orchestra.run.smithi103.stdout:317827579943 2024-05-22T23:59:10.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.963+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff8580a2cb0 msgr2=0x7ff8580a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:10.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.963+0000 7ff873e00700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff8580a2cb0 0x7ff8580a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff864009ca0 tx=0x7ff864009260 comp rx=0 tx=0).stop 2024-05-22T23:59:10.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff87410d770 msgr2=0x7ff8741a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:10.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff87410d770 0x7ff8741a38a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00b070 tx=0x7ff86c004c80 comp rx=0 tx=0).stop 2024-05-22T23:59:10.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 shutdown_connections 2024-05-22T23:59:10.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7ff8580a2cb0 0x7ff8580a5170 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff87410d770 0x7ff8741a38a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8741088a0 0x7ff87419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 --2- 172.21.15.103:0/96539749 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff874109260 0x7ff87419f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:10.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 >> 172.21.15.103:0/96539749 conn(0x7ff8740780f0 msgr2=0x7ff8740786b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:10.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 shutdown_connections 2024-05-22T23:59:10.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:10.964+0000 7ff873e00700 1 -- 172.21.15.103:0/96539749 wait complete. 2024-05-22T23:59:11.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:10 smithi105 ceph-mon[36426]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-22T23:59:11.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:10 smithi103 ceph-mon[31370]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-22T23:59:11.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:10 smithi174 ceph-mon[36179]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-22T23:59:11.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149770 got 111669149774 for osd.3 2024-05-22T23:59:11.739 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:12.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:11 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/96539749' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-22T23:59:12.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:11 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3203481937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-22T23:59:12.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:11 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/96539749' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-22T23:59:12.063 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:11 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3203481937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-22T23:59:12.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:11 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/96539749' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-22T23:59:12.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:11 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3203481937' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-22T23:59:12.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 -- 172.21.15.103:0/1977744398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104490 msgr2=0x7fd554104910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:12.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 --2- 172.21.15.103:0/1977744398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104490 0x7fd554104910 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0099c0 tx=0x7fd53c009cd0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 -- 172.21.15.103:0/1977744398 shutdown_connections 2024-05-22T23:59:12.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 --2- 172.21.15.103:0/1977744398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd554104e50 0x7fd554111310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 --2- 172.21.15.103:0/1977744398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd55410a540 0x7fd55410a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 --2- 172.21.15.103:0/1977744398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104490 0x7fd554104910 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 -- 172.21.15.103:0/1977744398 >> 172.21.15.103:0/1977744398 conn(0x7fd5540fff40 msgr2=0x7fd554102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 -- 172.21.15.103:0/1977744398 shutdown_connections 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.142+0000 7fd553200700 1 -- 172.21.15.103:0/1977744398 wait complete. 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.143+0000 7fd553200700 1 Processor -- start 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.143+0000 7fd553200700 1 -- start start 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd553200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd554104490 0x7fd5541a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd553200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104e50 0x7fd5541a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd553200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd55410a540 0x7fd55419aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd553200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd554114860 con 0x7fd554104e50 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd553200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd5541146e0 con 0x7fd55410a540 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd553200700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd5541149e0 con 0x7fd554104490 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd551400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104e50 0x7fd5541a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:12.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd551400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104e50 0x7fd5541a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:47018/0 (socket says 172.21.15.103:47018) 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd551400700 1 -- 172.21.15.103:0/3203481937 learned_addr learned my addr 172.21.15.103:0/3203481937 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.144+0000 7fd552800700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd55410a540 0x7fd55419aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd551400700 1 -- 172.21.15.103:0/3203481937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd554104490 msgr2=0x7fd5541a0880 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd551400700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd554104490 0x7fd5541a0880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd551400700 1 -- 172.21.15.103:0/3203481937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd55410a540 msgr2=0x7fd55419aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd551400700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd55410a540 0x7fd55419aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd551400700 1 -- 172.21.15.103:0/3203481937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55419b1a0 con 0x7fd554104e50 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd551400700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104e50 0x7fd5541a0dc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0098e0 tx=0x7fd53c003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd54aa00700 1 -- 172.21.15.103:0/3203481937 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c009e60 con 0x7fd554104e50 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd54aa00700 1 -- 172.21.15.103:0/3203481937 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd53c00f460 con 0x7fd554104e50 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd54aa00700 1 -- 172.21.15.103:0/3203481937 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c017650 con 0x7fd554104e50 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd553200700 1 -- 172.21.15.103:0/3203481937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd55419b430 con 0x7fd554104e50 2024-05-22T23:59:12.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.145+0000 7fd553200700 1 -- 172.21.15.103:0/3203481937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5541a7750 con 0x7fd554104e50 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.147+0000 7fd543e00700 1 -- 172.21.15.103:0/3203481937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd554069050 con 0x7fd554104e50 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.148+0000 7fd54aa00700 1 -- 172.21.15.103:0/3203481937 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fd53c00f610 con 0x7fd554104e50 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.150+0000 7fd54aa00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5380a2d30 0x7fd5380a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.150+0000 7fd54aa00700 1 -- 172.21.15.103:0/3203481937 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fd53c0d02a0 con 0x7fd554104e50 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.150+0000 7fd551e00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5380a2d30 0x7fd5380a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.151+0000 7fd551e00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5380a2d30 0x7fd5380a51f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd55419bb80 tx=0x7fd544006c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.154+0000 7fd54aa00700 1 -- 172.21.15.103:0/3203481937 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd53c013070 con 0x7fd554104e50 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.377+0000 7fd543e00700 1 -- 172.21.15.103:0/3203481937 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd554062f70 con 0x7fd554104e50 2024-05-22T23:59:12.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.377+0000 7fd54aa00700 1 -- 172.21.15.103:0/3203481937 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd53c09cd70 con 0x7fd554104e50 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stdout:55834574938 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.380+0000 7fd543e00700 1 -- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5380a2d30 msgr2=0x7fd5380a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.380+0000 7fd543e00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5380a2d30 0x7fd5380a51f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd55419bb80 tx=0x7fd544006c60 comp rx=0 tx=0).stop 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.380+0000 7fd543e00700 1 -- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104e50 msgr2=0x7fd5541a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.380+0000 7fd543e00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104e50 0x7fd5541a0dc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd53c0098e0 tx=0x7fd53c003dd0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.380+0000 7fd543e00700 1 -- 172.21.15.103:0/3203481937 shutdown_connections 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.381+0000 7fd543e00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd55410a540 0x7fd55419aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.381+0000 7fd543e00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd5380a2d30 0x7fd5380a51f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.381+0000 7fd543e00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd554104490 0x7fd5541a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.381+0000 7fd543e00700 1 --2- 172.21.15.103:0/3203481937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd554104e50 0x7fd5541a0dc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:12.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.381+0000 7fd543e00700 1 -- 172.21.15.103:0/3203481937 >> 172.21.15.103:0/3203481937 conn(0x7fd5540fff40 msgr2=0x7fd554101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:12.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.381+0000 7fd543e00700 1 -- 172.21.15.103:0/3203481937 shutdown_connections 2024-05-22T23:59:12.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:11.381+0000 7fd543e00700 1 -- 172.21.15.103:0/3203481937 wait complete. 2024-05-22T23:59:13.014 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:12 smithi103 ceph-mon[31370]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:13.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:12 smithi174 ceph-mon[36179]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:13.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:12 smithi105 ceph-mon[36426]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:13.657 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:13.663 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:13.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574932 got 55834574938 for osd.1 2024-05-22T23:59:13.808 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:14.515 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:14 smithi103 ceph-mon[31370]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:14.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:14 smithi105 ceph-mon[36426]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:14.573 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691905 got 171798691909 for osd.5 2024-05-22T23:59:14.573 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:14.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:14 smithi174 ceph-mon[36179]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:14.679 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:14.681 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:14.684 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:14.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.972+0000 7fded5000700 1 -- 172.21.15.103:0/4102500022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded006a070 msgr2=0x7fded006a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:14.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.972+0000 7fded5000700 1 --2- 172.21.15.103:0/4102500022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded006a070 0x7fded006a4d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdebc009990 tx=0x7fdebc009ca0 comp rx=0 tx=0).stop 2024-05-22T23:59:14.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.973+0000 7fded5000700 1 -- 172.21.15.103:0/4102500022 shutdown_connections 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.973+0000 7fded5000700 1 --2- 172.21.15.103:0/4102500022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded006aa10 0x7fded010f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.973+0000 7fded5000700 1 --2- 172.21.15.103:0/4102500022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fded01083e0 0x7fded01087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.973+0000 7fded5000700 1 --2- 172.21.15.103:0/4102500022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded006a070 0x7fded006a4d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.973+0000 7fded5000700 1 -- 172.21.15.103:0/4102500022 >> 172.21.15.103:0/4102500022 conn(0x7fded0078210 msgr2=0x7fded0078620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.973+0000 7fded5000700 1 -- 172.21.15.103:0/4102500022 shutdown_connections 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.973+0000 7fded5000700 1 -- 172.21.15.103:0/4102500022 wait complete. 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.974+0000 7fded5000700 1 Processor -- start 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.975+0000 7fded5000700 1 -- start start 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.975+0000 7fded5000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fded006a070 0x7fded019ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.975+0000 7fded5000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded006aa10 0x7fded019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:14.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.975+0000 7fded5000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded01083e0 0x7fded019f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.975+0000 7fded5000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fded0112640 con 0x7fded01083e0 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.975+0000 7fded5000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fded01124c0 con 0x7fded006aa10 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.975+0000 7fded5000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fded01127c0 con 0x7fded006a070 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecde00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fded006a070 0x7fded019ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecde00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fded006a070 0x7fded019ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:52444/0 (socket says 172.21.15.103:52444) 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecde00700 1 -- 172.21.15.103:0/1897512960 learned_addr learned my addr 172.21.15.103:0/1897512960 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecd400700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded006aa10 0x7fded019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdece800700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded01083e0 0x7fded019f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecd400700 1 -- 172.21.15.103:0/1897512960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fded006a070 msgr2=0x7fded019ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecd400700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fded006a070 0x7fded019ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecd400700 1 -- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded01083e0 msgr2=0x7fded019f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecd400700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded01083e0 0x7fded019f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:14.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.976+0000 7fdecd400700 1 -- 172.21.15.103:0/1897512960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded019fd80 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.977+0000 7fdecd400700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded006aa10 0x7fded019b2b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdebc0059d0 tx=0x7fdebc00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.977+0000 7fdec6a00700 1 -- 172.21.15.103:0/1897512960 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc027070 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.977+0000 7fdec6a00700 1 -- 172.21.15.103:0/1897512960 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdebc003d70 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.977+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded01a0010 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.977+0000 7fdec6a00700 1 -- 172.21.15.103:0/1897512960 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdebc00f460 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.977+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fded01a78f0 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.978+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded0103a90 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.980+0000 7fdec6a00700 1 -- 172.21.15.103:0/1897512960 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fdebc02a070 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.982+0000 7fdec6a00700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdeb40a2d30 0x7fdeb40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.982+0000 7fdec6a00700 1 -- 172.21.15.103:0/1897512960 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fdebc0d0490 con 0x7fded006aa10 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.982+0000 7fdecde00700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdeb40a2d30 0x7fdeb40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.983+0000 7fdecde00700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdeb40a2d30 0x7fdeb40a51f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009ce0 tx=0x7fdec0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:14.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:14.986+0000 7fdec6a00700 1 -- 172.21.15.103:0/1897512960 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdebc05e850 con 0x7fded006aa10 2024-05-22T23:59:15.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 -- 172.21.15.103:0/1911171320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 msgr2=0x7f7a2c115920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:15.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 --2- 172.21.15.103:0/1911171320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 0x7f7a2c115920 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7a24009990 tx=0x7f7a24009ca0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 -- 172.21.15.103:0/1911171320 shutdown_connections 2024-05-22T23:59:15.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 --2- 172.21.15.103:0/1911171320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a2c074590 0x7f7a2c074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 --2- 172.21.15.103:0/1911171320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 0x7f7a2c115920 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 --2- 172.21.15.103:0/1911171320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a2c074eb0 0x7f7a2c112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 -- 172.21.15.103:0/1911171320 >> 172.21.15.103:0/1911171320 conn(0x7f7a2c0794c0 msgr2=0x7f7a2c0798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 -- 172.21.15.103:0/1911171320 shutdown_connections 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.055+0000 7f7a30a00700 1 -- 172.21.15.103:0/1911171320 wait complete. 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.056+0000 7f7a30a00700 1 Processor -- start 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.056+0000 7f7a30a00700 1 -- start start 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a30a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a2c074590 0x7f7a2c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a30a00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a2c074eb0 0x7f7a2c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a30a00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 0x7f7a2c1a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a30a00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c118ca0 con 0x7f7a2c113530 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a30a00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c118b20 con 0x7f7a2c074590 2024-05-22T23:59:15.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a30a00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2c118e20 con 0x7f7a2c074eb0 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a2a800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 0x7f7a2c1a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a29e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a2c074590 0x7f7a2c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a2a800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 0x7f7a2c1a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53296/0 (socket says 172.21.15.103:53296) 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a29e00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a2c074590 0x7f7a2c19f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:35024/0 (socket says 172.21.15.103:35024) 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a2a800700 1 -- 172.21.15.103:0/3358371542 learned_addr learned my addr 172.21.15.103:0/3358371542 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.057+0000 7f7a21400700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a2c074eb0 0x7f7a2c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.058+0000 7f7a21400700 1 -- 172.21.15.103:0/3358371542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a2c074590 msgr2=0x7f7a2c19f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.058+0000 7f7a21400700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a2c074590 0x7f7a2c19f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.058+0000 7f7a21400700 1 -- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 msgr2=0x7f7a2c1a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.058+0000 7f7a21400700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 0x7f7a2c1a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.058+0000 7f7a21400700 1 -- 172.21.15.103:0/3358371542 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2c1a40f0 con 0x7f7a2c074eb0 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.058+0000 7f7a29e00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a2c074590 0x7f7a2c19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:15.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.058+0000 7f7a21400700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a2c074eb0 0x7f7a2c19f670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7a1400e9f0 tx=0x7f7a1400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.058+0000 7f7a23400700 1 -- 172.21.15.103:0/3358371542 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1400cb70 con 0x7f7a2c074eb0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.059+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2c1a43e0 con 0x7f7a2c074eb0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.059+0000 7f7a23400700 1 -- 172.21.15.103:0/3358371542 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1400ee90 con 0x7f7a2c074eb0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.059+0000 7f7a23400700 1 -- 172.21.15.103:0/3358371542 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a14003dc0 con 0x7f7a2c074eb0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.059+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a2c1abc60 con 0x7f7a2c074eb0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.060+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a2c069050 con 0x7f7a2c074eb0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.062+0000 7f7a23400700 1 -- 172.21.15.103:0/3358371542 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f7a14003f60 con 0x7f7a2c074eb0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.064+0000 7f7a23400700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7a100a2d30 0x7f7a100a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.065+0000 7f7a23400700 1 -- 172.21.15.103:0/3358371542 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f7a140cf600 con 0x7f7a2c074eb0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.065+0000 7f7a29e00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7a100a2d30 0x7f7a100a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.065+0000 7f7a29e00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7a100a2d30 0x7f7a100a51f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009480 tx=0x7f7a1c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:15.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.068+0000 7f7a23400700 1 -- 172.21.15.103:0/3358371542 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7a1409c040 con 0x7f7a2c074eb0 2024-05-22T23:59:15.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.234+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fded01a7ac0 con 0x7fded006aa10 2024-05-22T23:59:15.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.235+0000 7fdec6a00700 1 -- 172.21.15.103:0/1897512960 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdebc013070 con 0x7fded006aa10 2024-05-22T23:59:15.237 INFO:teuthology.orchestra.run.smithi103.stdout:227633266746 2024-05-22T23:59:15.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdeb40a2d30 msgr2=0x7fdeb40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:15.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdeb40a2d30 0x7fdeb40a51f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009ce0 tx=0x7fdec0009260 comp rx=0 tx=0).stop 2024-05-22T23:59:15.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded006aa10 msgr2=0x7fded019b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:15.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded006aa10 0x7fded019b2b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fdebc0059d0 tx=0x7fdebc00bab0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 shutdown_connections 2024-05-22T23:59:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdeb40a2d30 0x7fdeb40a51f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fded01083e0 0x7fded019f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fded006a070 0x7fded019ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 --2- 172.21.15.103:0/1897512960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fded006aa10 0x7fded019b2b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 >> 172.21.15.103:0/1897512960 conn(0x7fded0078210 msgr2=0x7fded010d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 shutdown_connections 2024-05-22T23:59:15.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.238+0000 7fded5000700 1 -- 172.21.15.103:0/1897512960 wait complete. 2024-05-22T23:59:15.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.316+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f7a2c1abef0 con 0x7f7a2c074eb0 2024-05-22T23:59:15.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:15 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1897512960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-22T23:59:15.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.317+0000 7f7a23400700 1 -- 172.21.15.103:0/3358371542 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7a1409b940 con 0x7f7a2c074eb0 2024-05-22T23:59:15.320 INFO:teuthology.orchestra.run.smithi103.stdout:352187318306 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7a100a2d30 msgr2=0x7f7a100a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7a100a2d30 0x7f7a100a51f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c009480 tx=0x7f7a1c00b040 comp rx=0 tx=0).stop 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a2c074eb0 msgr2=0x7f7a2c19f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a2c074eb0 0x7f7a2c19f670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7a1400e9f0 tx=0x7f7a1400ed00 comp rx=0 tx=0).stop 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 shutdown_connections 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a2c074590 0x7f7a2c19f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7a100a2d30 0x7f7a100a51f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a2c113530 0x7f7a2c1a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 --2- 172.21.15.103:0/3358371542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a2c074eb0 0x7f7a2c19f670 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 >> 172.21.15.103:0/3358371542 conn(0x7f7a2c0794c0 msgr2=0x7f7a2c112290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 shutdown_connections 2024-05-22T23:59:15.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:15.320+0000 7f7a30a00700 1 -- 172.21.15.103:0/3358371542 wait complete. 2024-05-22T23:59:15.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:15 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1897512960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-22T23:59:15.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:15 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1897512960' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-22T23:59:16.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:16 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3358371542' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-22T23:59:16.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:16 smithi105 ceph-mon[36426]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:16.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:16 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3358371542' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-22T23:59:16.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:16 smithi103 ceph-mon[31370]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:16.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:16 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3358371542' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-22T23:59:16.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:16 smithi174 ceph-mon[36179]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:18.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:18 smithi105 ceph-mon[36426]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:18.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:18 smithi103 ceph-mon[31370]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:18.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:18 smithi174 ceph-mon[36179]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:20.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:20 smithi103 ceph-mon[31370]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:20.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:20 smithi105 ceph-mon[36426]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:20.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:20 smithi174 ceph-mon[36179]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:20.720 INFO:tasks.cephadm.ceph_manager.ceph:need seq 317827579936 got 317827579943 for osd.10 2024-05-22T23:59:20.720 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:21.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 352187318300 got 352187318306 for osd.11 2024-05-22T23:59:21.080 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:22.472 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:22 smithi103 ceph-mon[31370]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:22.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:22 smithi105 ceph-mon[36426]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:22.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:22 smithi174 ceph-mon[36179]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.289+0000 7f1358000700 1 -- 172.21.15.103:0/2056930883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1350074eb0 msgr2=0x7f135010e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.289+0000 7f1358000700 1 --2- 172.21.15.103:0/2056930883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1350074eb0 0x7f135010e880 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f133c009990 tx=0x7f133c009ca0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.289+0000 7f1358000700 1 -- 172.21.15.103:0/2056930883 shutdown_connections 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.289+0000 7f1358000700 1 --2- 172.21.15.103:0/2056930883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1350074590 0x7f1350074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.289+0000 7f1358000700 1 --2- 172.21.15.103:0/2056930883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f135010edc0 0x7f13501111b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.289+0000 7f1358000700 1 --2- 172.21.15.103:0/2056930883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1350074eb0 0x7f135010e880 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.289+0000 7f1358000700 1 -- 172.21.15.103:0/2056930883 >> 172.21.15.103:0/2056930883 conn(0x7f13500794c0 msgr2=0x7f13500798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.290+0000 7f1358000700 1 -- 172.21.15.103:0/2056930883 shutdown_connections 2024-05-22T23:59:23.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.290+0000 7f1358000700 1 -- 172.21.15.103:0/2056930883 wait complete. 2024-05-22T23:59:23.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.291+0000 7f1358000700 1 Processor -- start 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.291+0000 7f1358000700 1 -- start start 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.291+0000 7f1358000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1350074590 0x7f13501a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.291+0000 7f1358000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1350074eb0 0x7f13501a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.291+0000 7f1358000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f135010edc0 0x7f135019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.291+0000 7f1358000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1350114920 con 0x7f1350074590 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.291+0000 7f1358000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f13501147a0 con 0x7f1350074eb0 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.291+0000 7f1358000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1350114aa0 con 0x7f135010edc0 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1355600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1350074590 0x7f13501a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1355600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1350074590 0x7f13501a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53312/0 (socket says 172.21.15.103:53312) 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1355600700 1 -- 172.21.15.103:0/2299780079 learned_addr learned my addr 172.21.15.103:0/2299780079 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1354c00700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1350074eb0 0x7f13501a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1356000700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f135010edc0 0x7f135019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1355600700 1 -- 172.21.15.103:0/2299780079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f135010edc0 msgr2=0x7f135019ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1355600700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f135010edc0 0x7f135019ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1355600700 1 -- 172.21.15.103:0/2299780079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1350074eb0 msgr2=0x7f13501a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1355600700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1350074eb0 0x7f13501a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.292+0000 7f1355600700 1 -- 172.21.15.103:0/2299780079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f135019b400 con 0x7f1350074590 2024-05-22T23:59:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.293+0000 7f1355600700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1350074590 0x7f13501a0a50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f134400e9c0 tx=0x7f134400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.293+0000 7f134e000700 1 -- 172.21.15.103:0/2299780079 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f134400cbd0 con 0x7f1350074590 2024-05-22T23:59:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.293+0000 7f134e000700 1 -- 172.21.15.103:0/2299780079 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f134400cd70 con 0x7f1350074590 2024-05-22T23:59:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.293+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f135019b6f0 con 0x7f1350074590 2024-05-22T23:59:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.293+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13501a79d0 con 0x7f1350074590 2024-05-22T23:59:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.294+0000 7f134e000700 1 -- 172.21.15.103:0/2299780079 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1344010430 con 0x7f1350074590 2024-05-22T23:59:23.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.294+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1350075c50 con 0x7f1350074590 2024-05-22T23:59:23.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.297+0000 7f134e000700 1 -- 172.21.15.103:0/2299780079 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f1344004750 con 0x7f1350074590 2024-05-22T23:59:23.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.298+0000 7f134e000700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13380a2cb0 0x7f13380a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:23.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.298+0000 7f134e000700 1 -- 172.21.15.103:0/2299780079 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f13440cfaa0 con 0x7f1350074590 2024-05-22T23:59:23.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.298+0000 7f1354c00700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13380a2cb0 0x7f13380a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:23.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.299+0000 7f1354c00700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13380a2cb0 0x7f13380a5170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f133c00b2e0 tx=0x7f133c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:23.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.305+0000 7f134e000700 1 -- 172.21.15.103:0/2299780079 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1344053850 con 0x7f1350074590 2024-05-22T23:59:23.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.529+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f135019bfe0 con 0x7f1350074590 2024-05-22T23:59:23.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.529+0000 7f134e000700 1 -- 172.21.15.103:0/2299780079 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f134409c4e0 con 0x7f1350074590 2024-05-22T23:59:23.533 INFO:teuthology.orchestra.run.smithi103.stdout:296352743471 2024-05-22T23:59:23.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.532+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13380a2cb0 msgr2=0x7f13380a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:23.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.532+0000 7f1358000700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13380a2cb0 0x7f13380a5170 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f133c00b2e0 tx=0x7f133c01a040 comp rx=0 tx=0).stop 2024-05-22T23:59:23.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.532+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1350074590 msgr2=0x7f13501a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:23.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.532+0000 7f1358000700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1350074590 0x7f13501a0a50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f134400e9c0 tx=0x7f134400ecd0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.533+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 shutdown_connections 2024-05-22T23:59:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.533+0000 7f1358000700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13380a2cb0 0x7f13380a5170 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.533+0000 7f1358000700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f135010edc0 0x7f135019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.533+0000 7f1358000700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1350074590 0x7f13501a0a50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.533+0000 7f1358000700 1 --2- 172.21.15.103:0/2299780079 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1350074eb0 0x7f13501a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.533+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 >> 172.21.15.103:0/2299780079 conn(0x7f13500794c0 msgr2=0x7f13500ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.533+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 shutdown_connections 2024-05-22T23:59:23.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.533+0000 7f1358000700 1 -- 172.21.15.103:0/2299780079 wait complete. 2024-05-22T23:59:23.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:23 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2299780079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-22T23:59:23.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:23 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2299780079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-22T23:59:24.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:23 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2299780079' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-22T23:59:24.301 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266741 got 227633266746 for osd.7 2024-05-22T23:59:24.301 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 -- 172.21.15.103:0/2707413369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b0103e30 msgr2=0x7fc2b010c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 --2- 172.21.15.103:0/2707413369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b0103e30 0x7fc2b010c320 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0098f0 tx=0x7fc29c009c00 comp rx=0 tx=0).stop 2024-05-22T23:59:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 -- 172.21.15.103:0/2707413369 shutdown_connections 2024-05-22T23:59:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 --2- 172.21.15.103:0/2707413369 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2b010c8f0 0x7fc2b010ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 --2- 172.21.15.103:0/2707413369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2b0103510 0x7fc2b01038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 --2- 172.21.15.103:0/2707413369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b0103e30 0x7fc2b010c320 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 -- 172.21.15.103:0/2707413369 >> 172.21.15.103:0/2707413369 conn(0x7fc2b00fcc80 msgr2=0x7fc2b00ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:24.478 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 -- 172.21.15.103:0/2707413369 shutdown_connections 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.375+0000 7fc2b5200700 1 -- 172.21.15.103:0/2707413369 wait complete. 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.376+0000 7fc2b5200700 1 Processor -- start 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.376+0000 7fc2b5200700 1 -- start start 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.376+0000 7fc2b5200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b0103510 0x7fc2b019ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.376+0000 7fc2b5200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2b0103e30 0x7fc2b019b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2b5200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2b010c8f0 0x7fc2b019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2b5200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b0112610 con 0x7fc2b0103e30 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2b5200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b0112490 con 0x7fc2b010c8f0 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2b5200700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b0112790 con 0x7fc2b0103510 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2aea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2b0103e30 0x7fc2b019b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:24.479 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2aea00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2b0103e30 0x7fc2b019b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53322/0 (socket says 172.21.15.103:53322) 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2aea00700 1 -- 172.21.15.103:0/3574068971 learned_addr learned my addr 172.21.15.103:0/3574068971 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2afe00700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2b010c8f0 0x7fc2b019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2af400700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b0103510 0x7fc2b019ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2aea00700 1 -- 172.21.15.103:0/3574068971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b0103510 msgr2=0x7fc2b019ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2aea00700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b0103510 0x7fc2b019ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2aea00700 1 -- 172.21.15.103:0/3574068971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2b010c8f0 msgr2=0x7fc2b019f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2aea00700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2b010c8f0 0x7fc2b019f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2aea00700 1 -- 172.21.15.103:0/3574068971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b019fd40 con 0x7fc2b0103e30 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.377+0000 7fc2afe00700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2b010c8f0 0x7fc2b019f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.378+0000 7fc2aea00700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2b0103e30 0x7fc2b019b270 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0094f0 tx=0x7fc29c00fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.378+0000 7fc2a7e00700 1 -- 172.21.15.103:0/3574068971 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c027070 con 0x7fc2b0103e30 2024-05-22T23:59:24.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.378+0000 7fc2a7e00700 1 -- 172.21.15.103:0/3574068971 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc29c0055a0 con 0x7fc2b0103e30 2024-05-22T23:59:24.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.378+0000 7fc2a7e00700 1 -- 172.21.15.103:0/3574068971 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c004930 con 0x7fc2b0103e30 2024-05-22T23:59:24.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.378+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b019ffd0 con 0x7fc2b0103e30 2024-05-22T23:59:24.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.378+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b01a78b0 con 0x7fc2b0103e30 2024-05-22T23:59:24.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.379+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b0069050 con 0x7fc2b0103e30 2024-05-22T23:59:24.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.383+0000 7fc2a7e00700 1 -- 172.21.15.103:0/3574068971 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fc29c02a070 con 0x7fc2b0103e30 2024-05-22T23:59:24.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.384+0000 7fc2a7e00700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc2940a2d30 0x7fc2940a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:24.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.384+0000 7fc2a7e00700 1 -- 172.21.15.103:0/3574068971 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fc29c0d02b0 con 0x7fc2b0103e30 2024-05-22T23:59:24.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.384+0000 7fc2af400700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc2940a2d30 0x7fc2940a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:24.482 INFO:teuthology.orchestra.run.smithi103.stdout:206158430271 2024-05-22T23:59:24.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.385+0000 7fc2af400700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc2940a2d30 0x7fc2940a51f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009750 tx=0x7fc2a0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:24.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.387+0000 7fc2a7e00700 1 -- 172.21.15.103:0/3574068971 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc29c031020 con 0x7fc2b0103e30 2024-05-22T23:59:24.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.637+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc2b002dbc0 con 0x7fc2b0103e30 2024-05-22T23:59:24.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.637+0000 7fc2a7e00700 1 -- 172.21.15.103:0/3574068971 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc29c013030 con 0x7fc2b0103e30 2024-05-22T23:59:24.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc2940a2d30 msgr2=0x7fc2940a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc2940a2d30 0x7fc2940a51f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009750 tx=0x7fc2a0009310 comp rx=0 tx=0).stop 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2b0103e30 msgr2=0x7fc2b019b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2b0103e30 0x7fc2b019b270 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0094f0 tx=0x7fc29c00fa00 comp rx=0 tx=0).stop 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 shutdown_connections 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2b010c8f0 0x7fc2b019f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fc2940a2d30 0x7fc2940a51f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b0103510 0x7fc2b019ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 --2- 172.21.15.103:0/3574068971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc2b0103e30 0x7fc2b019b270 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 >> 172.21.15.103:0/3574068971 conn(0x7fc2b00fcc80 msgr2=0x7fc2b00fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 shutdown_connections 2024-05-22T23:59:24.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:23.640+0000 7fc2b5200700 1 -- 172.21.15.103:0/3574068971 wait complete. 2024-05-22T23:59:24.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:24 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3574068971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-22T23:59:24.850 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:24 smithi103 ceph-mon[31370]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:24.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:24 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3574068971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-22T23:59:24.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:24 smithi174 ceph-mon[36179]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:25.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:24 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3574068971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-22T23:59:25.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:24 smithi105 ceph-mon[36426]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:25.782 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430266 got 206158430271 for osd.6 2024-05-22T23:59:25.782 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:25.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 -- 172.21.15.103:0/2447819987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce04108990 msgr2=0x7fce04108e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:25.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 --2- 172.21.15.103:0/2447819987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce04108990 0x7fce04108e10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009990 tx=0x7fcdf0009ca0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 -- 172.21.15.103:0/2447819987 shutdown_connections 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 --2- 172.21.15.103:0/2447819987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce04109350 0x7fce04115860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 --2- 172.21.15.103:0/2447819987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce0410e990 0x7fce0410ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 --2- 172.21.15.103:0/2447819987 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce04108990 0x7fce04108e10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 -- 172.21.15.103:0/2447819987 >> 172.21.15.103:0/2447819987 conn(0x7fce04075b30 msgr2=0x7fce04107180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 -- 172.21.15.103:0/2447819987 shutdown_connections 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.656+0000 7fce09400700 1 -- 172.21.15.103:0/2447819987 wait complete. 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.657+0000 7fce09400700 1 Processor -- start 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.657+0000 7fce09400700 1 -- start start 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce09400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce04108990 0x7fce0419f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:25.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce09400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce04109350 0x7fce0419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce09400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce0410e990 0x7fce041a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce09400700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fce04118d20 con 0x7fce0410e990 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce09400700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fce04118ba0 con 0x7fce04108990 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce09400700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fce04118ea0 con 0x7fce04109350 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce03200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce0410e990 0x7fce041a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce03200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce0410e990 0x7fce041a3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37310/0 (socket says 172.21.15.103:37310) 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce03200700 1 -- 172.21.15.103:0/3475696304 learned_addr learned my addr 172.21.15.103:0/3475696304 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce01e00700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce04109350 0x7fce0419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.658+0000 7fce02800700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce04108990 0x7fce0419f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce03200700 1 -- 172.21.15.103:0/3475696304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce04109350 msgr2=0x7fce0419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:25.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce03200700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce04109350 0x7fce0419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce03200700 1 -- 172.21.15.103:0/3475696304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce04108990 msgr2=0x7fce0419f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce03200700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce04108990 0x7fce0419f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce03200700 1 -- 172.21.15.103:0/3475696304 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce041a4200 con 0x7fce0410e990 2024-05-22T23:59:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce02800700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce04108990 0x7fce0419f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:59:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce01e00700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce04109350 0x7fce0419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce03200700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce0410e990 0x7fce041a3ac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00e940 tx=0x7fcdfc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fcdfb400700 1 -- 172.21.15.103:0/3475696304 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc00cbd0 con 0x7fce0410e990 2024-05-22T23:59:25.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fcdfb400700 1 -- 172.21.15.103:0/3475696304 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdfc00ee90 con 0x7fce0410e990 2024-05-22T23:59:25.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce041a44f0 con 0x7fce0410e990 2024-05-22T23:59:25.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fcdfb400700 1 -- 172.21.15.103:0/3475696304 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0056c0 con 0x7fce0410e990 2024-05-22T23:59:25.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.659+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce041abd70 con 0x7fce0410e990 2024-05-22T23:59:25.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.660+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce04069050 con 0x7fce0410e990 2024-05-22T23:59:25.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.662+0000 7fcdfb400700 1 -- 172.21.15.103:0/3475696304 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7fcdfc004750 con 0x7fce0410e990 2024-05-22T23:59:25.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.664+0000 7fcdfb400700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcde80a2cb0 0x7fcde80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:25.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.664+0000 7fcdfb400700 1 -- 172.21.15.103:0/3475696304 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fcdfc0cf7e0 con 0x7fce0410e990 2024-05-22T23:59:25.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.664+0000 7fce02800700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcde80a2cb0 0x7fcde80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:25.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.665+0000 7fce02800700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcde80a2cb0 0x7fcde80a5170 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40093a0 tx=0x7fcdf400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:25.904 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.668+0000 7fcdfb400700 1 -- 172.21.15.103:0/3475696304 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcdfc09c220 con 0x7fce0410e990 2024-05-22T23:59:25.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.896+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fce041a0550 con 0x7fce0410e990 2024-05-22T23:59:25.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.897+0000 7fcdfb400700 1 -- 172.21.15.103:0/3475696304 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fcdfc09bb20 con 0x7fce0410e990 2024-05-22T23:59:25.905 INFO:teuthology.orchestra.run.smithi103.stdout:261993005109 2024-05-22T23:59:25.905 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.900+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcde80a2cb0 msgr2=0x7fcde80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.900+0000 7fce09400700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcde80a2cb0 0x7fcde80a5170 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcdf40093a0 tx=0x7fcdf400b040 comp rx=0 tx=0).stop 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce0410e990 msgr2=0x7fce041a3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce0410e990 0x7fce041a3ac0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00e940 tx=0x7fcdfc00ec50 comp rx=0 tx=0).stop 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 shutdown_connections 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fce04108990 0x7fce0419f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcde80a2cb0 0x7fcde80a5170 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fce0410e990 0x7fce041a3ac0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 --2- 172.21.15.103:0/3475696304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce04109350 0x7fce0419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:25.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 >> 172.21.15.103:0/3475696304 conn(0x7fce04075b30 msgr2=0x7fce040760f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:25.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 shutdown_connections 2024-05-22T23:59:25.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:25.901+0000 7fce09400700 1 -- 172.21.15.103:0/3475696304 wait complete. 2024-05-22T23:59:26.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:25 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3475696304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-22T23:59:26.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:25 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3475696304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-22T23:59:26.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:25 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3475696304' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-22T23:59:26.398 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.394+0000 7f6603200700 1 -- 172.21.15.103:0/869199377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc10a560 msgr2=0x7f65fc10a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:26.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.394+0000 7f6603200700 1 --2- 172.21.15.103:0/869199377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc10a560 0x7f65fc10a940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f65f00099c0 tx=0x7f65f0009cd0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.394+0000 7f6603200700 1 -- 172.21.15.103:0/869199377 shutdown_connections 2024-05-22T23:59:26.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.394+0000 7f6603200700 1 --2- 172.21.15.103:0/869199377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65fc104e70 0x7f65fc111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.394+0000 7f6603200700 1 --2- 172.21.15.103:0/869199377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc10a560 0x7f65fc10a940 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.394+0000 7f6603200700 1 --2- 172.21.15.103:0/869199377 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65fc1044b0 0x7f65fc104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.395+0000 7f6603200700 1 -- 172.21.15.103:0/869199377 >> 172.21.15.103:0/869199377 conn(0x7f65fc0fff40 msgr2=0x7f65fc102360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:26.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.395+0000 7f6603200700 1 -- 172.21.15.103:0/869199377 shutdown_connections 2024-05-22T23:59:26.399 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.395+0000 7f6603200700 1 -- 172.21.15.103:0/869199377 wait complete. 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6603200700 1 Processor -- start 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6603200700 1 -- start start 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6603200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65fc1044b0 0x7f65fc1a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6603200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65fc104e70 0x7f65fc1a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6603200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc10a560 0x7f65fc19a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6603200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc114810 con 0x7f65fc10a560 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6603200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc114690 con 0x7f65fc104e70 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6603200700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc114990 con 0x7f65fc1044b0 2024-05-22T23:59:26.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6601400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65fc104e70 0x7f65fc1a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6602800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc10a560 0x7f65fc19a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6601e00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65fc1044b0 0x7f65fc1a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.396+0000 7f6601400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65fc104e70 0x7f65fc1a0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49204/0 (socket says 172.21.15.103:49204) 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.397+0000 7f6601400700 1 -- 172.21.15.103:0/1570524896 learned_addr learned my addr 172.21.15.103:0/1570524896 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.397+0000 7f6601400700 1 -- 172.21.15.103:0/1570524896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65fc1044b0 msgr2=0x7f65fc1a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.397+0000 7f6601400700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65fc1044b0 0x7f65fc1a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.397+0000 7f6601400700 1 -- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc10a560 msgr2=0x7f65fc19a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.397+0000 7f6601400700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc10a560 0x7f65fc19a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.397+0000 7f6601400700 1 -- 172.21.15.103:0/1570524896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc19b110 con 0x7f65fc104e70 2024-05-22T23:59:26.402 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.397+0000 7f6601400700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65fc104e70 0x7f65fc1a0d30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00e9a0 tx=0x7f65ec00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:26.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.398+0000 7f65f6a00700 1 -- 172.21.15.103:0/1570524896 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec00ca10 con 0x7f65fc104e70 2024-05-22T23:59:26.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.398+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc19b400 con 0x7f65fc104e70 2024-05-22T23:59:26.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.398+0000 7f65f6a00700 1 -- 172.21.15.103:0/1570524896 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65ec00cbb0 con 0x7f65fc104e70 2024-05-22T23:59:26.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.398+0000 7f65f6a00700 1 -- 172.21.15.103:0/1570524896 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec018680 con 0x7f65fc104e70 2024-05-22T23:59:26.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.398+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65fc1a76c0 con 0x7f65fc104e70 2024-05-22T23:59:26.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.399+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65fc069050 con 0x7f65fc104e70 2024-05-22T23:59:26.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.402+0000 7f65f6a00700 1 -- 172.21.15.103:0/1570524896 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f65ec010430 con 0x7f65fc104e70 2024-05-22T23:59:26.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.404+0000 7f65f6a00700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65e40a2d30 0x7f65e40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:26.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.404+0000 7f65f6a00700 1 -- 172.21.15.103:0/1570524896 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f65ec0cfbb0 con 0x7f65fc104e70 2024-05-22T23:59:26.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.404+0000 7f6601e00700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65e40a2d30 0x7f65e40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:26.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.405+0000 7f65f6a00700 1 -- 172.21.15.103:0/1570524896 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f65ec09c5f0 con 0x7f65fc104e70 2024-05-22T23:59:26.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.405+0000 7f6601e00700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65e40a2d30 0x7f65e40a51f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f65f0000c00 tx=0x7f65f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:26.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.628+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f65fc1a78f0 con 0x7f65fc104e70 2024-05-22T23:59:26.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.629+0000 7f65f6a00700 1 -- 172.21.15.103:0/1570524896 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f65ec09bef0 con 0x7f65fc104e70 2024-05-22T23:59:26.632 INFO:teuthology.orchestra.run.smithi103.stdout:34359738465 2024-05-22T23:59:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65e40a2d30 msgr2=0x7f65e40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65e40a2d30 0x7f65e40a51f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f65f0000c00 tx=0x7f65f0005fb0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65fc104e70 msgr2=0x7f65fc1a0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65fc104e70 0x7f65fc1a0d30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00e9a0 tx=0x7f65ec00ecb0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 shutdown_connections 2024-05-22T23:59:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f65e40a2d30 0x7f65e40a51f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc10a560 0x7f65fc19a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65fc1044b0 0x7f65fc1a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 --2- 172.21.15.103:0/1570524896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f65fc104e70 0x7f65fc1a0d30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.631+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 >> 172.21.15.103:0/1570524896 conn(0x7f65fc0fff40 msgr2=0x7f65fc101400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.632+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 shutdown_connections 2024-05-22T23:59:26.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:26.632+0000 7f6603200700 1 -- 172.21.15.103:0/1570524896 wait complete. 2024-05-22T23:59:27.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:26 smithi105 ceph-mon[36426]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:27.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:26 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1570524896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-22T23:59:27.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:26 smithi103 ceph-mon[31370]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:27.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:26 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1570524896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-22T23:59:27.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:26 smithi174 ceph-mon[36179]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:27.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:26 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1570524896' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-22T23:59:27.485 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738459 got 34359738465 for osd.0 2024-05-22T23:59:27.485 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:28.343 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743464 got 296352743471 for osd.9 2024-05-22T23:59:28.343 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:28.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 -- 172.21.15.103:0/3822471186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a4010edc0 msgr2=0x7f1a401111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:28.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 --2- 172.21.15.103:0/3822471186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a4010edc0 0x7f1a401111b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009f90 tx=0x7f1a3800b330 comp rx=0 tx=0).stop 2024-05-22T23:59:28.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 -- 172.21.15.103:0/3822471186 shutdown_connections 2024-05-22T23:59:28.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 --2- 172.21.15.103:0/3822471186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a4010edc0 0x7f1a401111b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 --2- 172.21.15.103:0/3822471186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a40074590 0x7f1a40074970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 --2- 172.21.15.103:0/3822471186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a40074eb0 0x7f1a4010e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 -- 172.21.15.103:0/3822471186 >> 172.21.15.103:0/3822471186 conn(0x7f1a400794c0 msgr2=0x7f1a400798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 -- 172.21.15.103:0/3822471186 shutdown_connections 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.568+0000 7f1a46000700 1 -- 172.21.15.103:0/3822471186 wait complete. 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.569+0000 7f1a46000700 1 Processor -- start 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a46000700 1 -- start start 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a46000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a40074590 0x7f1a401024e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a46000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a40074eb0 0x7f1a40102a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a46000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a4010edc0 0x7f1a40102f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a46000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40114900 con 0x7f1a40074590 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a46000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40114780 con 0x7f1a40074eb0 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a46000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a40114a80 con 0x7f1a4010edc0 2024-05-22T23:59:28.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a3f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a40074590 0x7f1a401024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a3fe00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a4010edc0 0x7f1a40102f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a3f400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a40074590 0x7f1a401024e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37344/0 (socket says 172.21.15.103:37344) 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a3f400700 1 -- 172.21.15.103:0/3746673818 learned_addr learned my addr 172.21.15.103:0/3746673818 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.570+0000 7f1a3ea00700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a40074eb0 0x7f1a40102a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a3fe00700 1 -- 172.21.15.103:0/3746673818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a40074eb0 msgr2=0x7f1a40102a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a3fe00700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a40074eb0 0x7f1a40102a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a3fe00700 1 -- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a40074590 msgr2=0x7f1a401024e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a3fe00700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a40074590 0x7f1a401024e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a3fe00700 1 -- 172.21.15.103:0/3746673818 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a401ab3e0 con 0x7f1a4010edc0 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a3f400700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a40074590 0x7f1a401024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a3fe00700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a4010edc0 0x7f1a40102f60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009f90 tx=0x7f1a3800cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:28.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a37e00700 1 -- 172.21.15.103:0/3746673818 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38021020 con 0x7f1a4010edc0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a37e00700 1 -- 172.21.15.103:0/3746673818 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a38007ac0 con 0x7f1a4010edc0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a401ab670 con 0x7f1a4010edc0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.571+0000 7f1a37e00700 1 -- 172.21.15.103:0/3746673818 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38005040 con 0x7f1a4010edc0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.572+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a401abad0 con 0x7f1a4010edc0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.573+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a40075c70 con 0x7f1a4010edc0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.575+0000 7f1a37e00700 1 -- 172.21.15.103:0/3746673818 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 17) ==== 137885+0+0 (secure 0 0 0) 0x7f1a38024020 con 0x7f1a4010edc0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.576+0000 7f1a37e00700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a240a2cb0 0x7f1a240a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.576+0000 7f1a37e00700 1 -- 172.21.15.103:0/3746673818 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1a380d04e0 con 0x7f1a4010edc0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.576+0000 7f1a3f400700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a240a2cb0 0x7f1a240a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.577+0000 7f1a3f400700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a240a2cb0 0x7f1a240a5170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1a30011550 tx=0x7f1a30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:28.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.580+0000 7f1a37e00700 1 -- 172.21.15.103:0/3746673818 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1a3802b020 con 0x7f1a4010edc0 2024-05-22T23:59:28.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.809+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1a401abca0 con 0x7f1a4010edc0 2024-05-22T23:59:28.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.809+0000 7f1a37e00700 1 -- 172.21.15.103:0/3746673818 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1a38007c70 con 0x7f1a4010edc0 2024-05-22T23:59:28.813 INFO:teuthology.orchestra.run.smithi103.stdout:77309411416 2024-05-22T23:59:28.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.812+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a240a2cb0 msgr2=0x7f1a240a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.812+0000 7f1a46000700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a240a2cb0 0x7f1a240a5170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1a30011550 tx=0x7f1a30009260 comp rx=0 tx=0).stop 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a4010edc0 msgr2=0x7f1a40102f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a4010edc0 0x7f1a40102f60 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009f90 tx=0x7f1a3800cef0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 shutdown_connections 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a240a2cb0 0x7f1a240a5170 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a4010edc0 0x7f1a40102f60 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a40074590 0x7f1a401024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 --2- 172.21.15.103:0/3746673818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a40074eb0 0x7f1a40102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 >> 172.21.15.103:0/3746673818 conn(0x7f1a400794c0 msgr2=0x7f1a401005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 shutdown_connections 2024-05-22T23:59:28.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:28.813+0000 7f1a46000700 1 -- 172.21.15.103:0/3746673818 wait complete. 2024-05-22T23:59:29.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:29.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:29 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:29.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:29 smithi103 ceph-mon[31370]: pgmap v382: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:29.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:29 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3746673818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-22T23:59:29.226 INFO:tasks.cephadm.ceph_manager.ceph:need seq 261993005103 got 261993005109 for osd.8 2024-05-22T23:59:29.226 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:29.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:29.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:29 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:29.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:29 smithi105 ceph-mon[36426]: pgmap v382: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:29.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:29 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3746673818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-22T23:59:29.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:29.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:29 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:29.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:29 smithi174 ceph-mon[36179]: pgmap v382: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:29.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:29 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3746673818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-22T23:59:29.739 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411407 got 77309411416 for osd.2 2024-05-22T23:59:29.739 DEBUG:teuthology.parallel:result is None 2024-05-22T23:59:29.739 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-22T23:59:29.740 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph pg dump --format=json 2024-05-22T23:59:30.198 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:30.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:59:30.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:59:30.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:30 smithi105 ceph-mon[36426]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:59:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:59:30.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:30 smithi103 ceph-mon[31370]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:30.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-22T23:59:30.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-22T23:59:30.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:30 smithi174 ceph-mon[36179]: from='mgr.14152 172.21.15.103:0/2994270729' entity='mgr.x' 2024-05-22T23:59:31.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:31 smithi103 ceph-mon[31370]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:31.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:31 smithi174 ceph-mon[36179]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:31.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:31 smithi105 ceph-mon[36426]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:32.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:32 smithi103 ceph-mon[31370]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-22T23:59:32.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:32 smithi174 ceph-mon[36179]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-22T23:59:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:32 smithi105 ceph-mon[36426]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-05-22T23:59:32.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 -- 172.21.15.103:0/3653175109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05941045a0 msgr2=0x7f0594104a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:32.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 --2- 172.21.15.103:0/3653175109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05941045a0 0x7f0594104a20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0580009990 tx=0x7f0580009ca0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 -- 172.21.15.103:0/3653175109 shutdown_connections 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 --2- 172.21.15.103:0/3653175109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f059410a650 0x7f059410aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 --2- 172.21.15.103:0/3653175109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0594104f60 0x7f0594111420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 --2- 172.21.15.103:0/3653175109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05941045a0 0x7f0594104a20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 -- 172.21.15.103:0/3653175109 >> 172.21.15.103:0/3653175109 conn(0x7f0594100070 msgr2=0x7f0594102490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 -- 172.21.15.103:0/3653175109 shutdown_connections 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.566+0000 7f0599c00700 1 -- 172.21.15.103:0/3653175109 wait complete. 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.567+0000 7f0599c00700 1 Processor -- start 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.568+0000 7f0599c00700 1 -- start start 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.568+0000 7f0599c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05941045a0 0x7f05941a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.568+0000 7f0599c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0594104f60 0x7f05941a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.568+0000 7f0599c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f059410a650 0x7f059419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:32.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.568+0000 7f0599c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0594114900 con 0x7f0594104f60 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.568+0000 7f0599c00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0594114780 con 0x7f05941045a0 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.568+0000 7f0599c00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0594114a80 con 0x7f059410a650 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.568+0000 7f0592000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0594104f60 0x7f05941a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05941045a0 0x7f05941a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0594104f60 0x7f05941a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37354/0 (socket says 172.21.15.103:37354) 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592000700 1 -- 172.21.15.103:0/2584618476 learned_addr learned my addr 172.21.15.103:0/2584618476 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592a00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05941045a0 0x7f05941a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49236/0 (socket says 172.21.15.103:49236) 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0593400700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f059410a650 0x7f059419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592a00700 1 -- 172.21.15.103:0/2584618476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f059410a650 msgr2=0x7f059419aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592a00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f059410a650 0x7f059419aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592a00700 1 -- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0594104f60 msgr2=0x7f05941a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:32.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592a00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0594104f60 0x7f05941a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592a00700 1 -- 172.21.15.103:0/2584618476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f059419b1c0 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f0592a00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05941045a0 0x7f05941a08a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f058400e9f0 tx=0x7f058400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.569+0000 7f058b400700 1 -- 172.21.15.103:0/2584618476 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584004220 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.570+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f059419b4b0 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.570+0000 7f058b400700 1 -- 172.21.15.103:0/2584618476 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0584004d10 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.570+0000 7f058b400700 1 -- 172.21.15.103:0/2584618476 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584005020 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.570+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05941a7770 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.571+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0594069050 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.573+0000 7f058b400700 1 -- 172.21.15.103:0/2584618476 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f0584042ae0 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.574+0000 7f058b400700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f05780a2ca0 0x7f05780a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.574+0000 7f058b400700 1 -- 172.21.15.103:0/2584618476 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f05840cf210 con 0x7f05941045a0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.574+0000 7f0592000700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f05780a2ca0 0x7f05780a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:32.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.575+0000 7f0592000700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f05780a2ca0 0x7f05780a5160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0594109be0 tx=0x7f0580009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:32.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.576+0000 7f058b400700 1 -- 172.21.15.103:0/2584618476 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f058401f3a0 con 0x7f05941045a0 2024-05-22T23:59:32.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.797+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f05940627a0 con 0x7f05780a2ca0 2024-05-22T23:59:32.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.816+0000 7f058b400700 1 -- 172.21.15.103:0/2584618476 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340794 (secure 0 0 0) 0x7f05940627a0 con 0x7f05780a2ca0 2024-05-22T23:59:32.826 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:32.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.821+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f05780a2ca0 msgr2=0x7f05780a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:32.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f05780a2ca0 0x7f05780a5160 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0594109be0 tx=0x7f0580009590 comp rx=0 tx=0).stop 2024-05-22T23:59:32.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05941045a0 msgr2=0x7f05941a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:32.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05941045a0 0x7f05941a08a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f058400e9f0 tx=0x7f058400ed00 comp rx=0 tx=0).stop 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 shutdown_connections 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f05941045a0 0x7f05941a08a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f05780a2ca0 0x7f05780a5160 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f059410a650 0x7f059419aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 --2- 172.21.15.103:0/2584618476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0594104f60 0x7f05941a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 >> 172.21.15.103:0/2584618476 conn(0x7f0594100070 msgr2=0x7f0594101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 shutdown_connections 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:32.822+0000 7f0599c00700 1 -- 172.21.15.103:0/2584618476 wait complete. 2024-05-22T23:59:32.829 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-05-22T23:59:33.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:33 smithi103 ceph-mon[31370]: pgmap v384: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-22T23:59:33.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:33 smithi174 ceph-mon[36179]: pgmap v384: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-22T23:59:33.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:33 smithi105 ceph-mon[36426]: pgmap v384: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-22T23:59:33.617 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-05-22T23:59:32.156600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657323,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":661,"num_read_kb":451,"num_write":349,"num_write_kb":3058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1181748,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":570,"ondisk_log_size":570,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357424,"kb_used_data":12392,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584400,"statfs":{"total":1151940427776,"available":1151574425600,"internally_reserved":0,"allocated":12689408,"data_stored":8048846,"data_compressed":359403,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":60,"num_read_kb":30,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005015"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.119831+0000","last_change":"2024-05-22T23:57:33.328075+0000","last_active":"2024-05-22T23:59:30.119831+0000","last_peered":"2024-05-22T23:59:30.119831+0000","last_clean":"2024-05-22T23:59:30.119831+0000","last_became_active":"2024-05-22T23:57:33.327704+0000","last_became_peered":"2024-05-22T23:57:33.327704+0000","last_unstale":"2024-05-22T23:59:30.119831+0000","last_undegraded":"2024-05-22T23:59:30.119831+0000","last_fullsized":"2024-05-22T23:59:30.119831+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:42:32.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101195+0000","last_change":"2024-05-22T23:57:33.332043+0000","last_active":"2024-05-22T23:58:31.101195+0000","last_peered":"2024-05-22T23:58:31.101195+0000","last_clean":"2024-05-22T23:58:31.101195+0000","last_became_active":"2024-05-22T23:57:33.322269+0000","last_became_peered":"2024-05-22T23:57:33.322269+0000","last_unstale":"2024-05-22T23:58:31.101195+0000","last_undegraded":"2024-05-22T23:58:31.101195+0000","last_fullsized":"2024-05-22T23:58:31.101195+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:40:23.542938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100766+0000","last_change":"2024-05-22T23:57:33.334993+0000","last_active":"2024-05-22T23:58:31.100766+0000","last_peered":"2024-05-22T23:58:31.100766+0000","last_clean":"2024-05-22T23:58:31.100766+0000","last_became_active":"2024-05-22T23:57:33.326446+0000","last_became_peered":"2024-05-22T23:57:33.326446+0000","last_unstale":"2024-05-22T23:58:31.100766+0000","last_undegraded":"2024-05-22T23:58:31.100766+0000","last_fullsized":"2024-05-22T23:58:31.100766+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:44:45.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097155+0000","last_change":"2024-05-22T23:57:33.320181+0000","last_active":"2024-05-22T23:58:31.097155+0000","last_peered":"2024-05-22T23:58:31.097155+0000","last_clean":"2024-05-22T23:58:31.097155+0000","last_became_active":"2024-05-22T23:57:33.319630+0000","last_became_peered":"2024-05-22T23:57:33.319630+0000","last_unstale":"2024-05-22T23:58:31.097155+0000","last_undegraded":"2024-05-22T23:58:31.097155+0000","last_fullsized":"2024-05-22T23:58:31.097155+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:38:51.836241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107815+0000","last_change":"2024-05-22T23:57:33.332168+0000","last_active":"2024-05-22T23:58:31.107815+0000","last_peered":"2024-05-22T23:58:31.107815+0000","last_clean":"2024-05-22T23:58:31.107815+0000","last_became_active":"2024-05-22T23:57:33.331755+0000","last_became_peered":"2024-05-22T23:57:33.331755+0000","last_unstale":"2024-05-22T23:58:31.107815+0000","last_undegraded":"2024-05-22T23:58:31.107815+0000","last_fullsized":"2024-05-22T23:58:31.107815+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:28.013365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.105537+0000","last_change":"2024-05-22T23:57:33.322449+0000","last_active":"2024-05-22T23:58:31.105537+0000","last_peered":"2024-05-22T23:58:31.105537+0000","last_clean":"2024-05-22T23:58:31.105537+0000","last_became_active":"2024-05-22T23:57:33.322099+0000","last_became_peered":"2024-05-22T23:57:33.322099+0000","last_unstale":"2024-05-22T23:58:31.105537+0000","last_undegraded":"2024-05-22T23:58:31.105537+0000","last_fullsized":"2024-05-22T23:58:31.105537+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:38.427434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108181+0000","last_change":"2024-05-22T23:57:33.322391+0000","last_active":"2024-05-22T23:59:30.108181+0000","last_peered":"2024-05-22T23:59:30.108181+0000","last_clean":"2024-05-22T23:59:30.108181+0000","last_became_active":"2024-05-22T23:57:33.322076+0000","last_became_peered":"2024-05-22T23:57:33.322076+0000","last_unstale":"2024-05-22T23:59:30.108181+0000","last_undegraded":"2024-05-22T23:59:30.108181+0000","last_fullsized":"2024-05-22T23:59:30.108181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:45:44.857586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.115118+0000","last_change":"2024-05-22T23:57:33.333005+0000","last_active":"2024-05-22T23:59:30.115118+0000","last_peered":"2024-05-22T23:59:30.115118+0000","last_clean":"2024-05-22T23:59:30.115118+0000","last_became_active":"2024-05-22T23:57:33.332669+0000","last_became_peered":"2024-05-22T23:57:33.332669+0000","last_unstale":"2024-05-22T23:59:30.115118+0000","last_undegraded":"2024-05-22T23:59:30.115118+0000","last_fullsized":"2024-05-22T23:59:30.115118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:47:25.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.105987+0000","last_change":"2024-05-22T23:58:31.104194+0000","last_active":"2024-05-22T23:59:30.105987+0000","last_peered":"2024-05-22T23:59:30.105987+0000","last_clean":"2024-05-22T23:59:30.105987+0000","last_became_active":"2024-05-22T23:58:31.103788+0000","last_became_peered":"2024-05-22T23:58:31.103788+0000","last_unstale":"2024-05-22T23:59:30.105987+0000","last_undegraded":"2024-05-22T23:59:30.105987+0000","last_fullsized":"2024-05-22T23:59:30.105987+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:41:09.215669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.105338+0000","last_change":"2024-05-22T23:57:33.324445+0000","last_active":"2024-05-22T23:58:31.105338+0000","last_peered":"2024-05-22T23:58:31.105338+0000","last_clean":"2024-05-22T23:58:31.105338+0000","last_became_active":"2024-05-22T23:57:33.324085+0000","last_became_peered":"2024-05-22T23:57:33.324085+0000","last_unstale":"2024-05-22T23:58:31.105338+0000","last_undegraded":"2024-05-22T23:58:31.105338+0000","last_fullsized":"2024-05-22T23:58:31.105338+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:30:51.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.104148+0000","last_change":"2024-05-22T23:57:33.321481+0000","last_active":"2024-05-22T23:59:30.104148+0000","last_peered":"2024-05-22T23:59:30.104148+0000","last_clean":"2024-05-22T23:59:30.104148+0000","last_became_active":"2024-05-22T23:57:33.320911+0000","last_became_peered":"2024-05-22T23:57:33.320911+0000","last_unstale":"2024-05-22T23:59:30.104148+0000","last_undegraded":"2024-05-22T23:59:30.104148+0000","last_fullsized":"2024-05-22T23:59:30.104148+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:24:24.041980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097223+0000","last_change":"2024-05-22T23:57:33.318819+0000","last_active":"2024-05-22T23:58:31.097223+0000","last_peered":"2024-05-22T23:58:31.097223+0000","last_clean":"2024-05-22T23:58:31.097223+0000","last_became_active":"2024-05-22T23:57:33.318284+0000","last_became_peered":"2024-05-22T23:57:33.318284+0000","last_unstale":"2024-05-22T23:58:31.097223+0000","last_undegraded":"2024-05-22T23:58:31.097223+0000","last_fullsized":"2024-05-22T23:58:31.097223+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:56.043446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.112808+0000","last_change":"2024-05-22T23:57:33.321268+0000","last_active":"2024-05-22T23:58:31.112808+0000","last_peered":"2024-05-22T23:58:31.112808+0000","last_clean":"2024-05-22T23:58:31.112808+0000","last_became_active":"2024-05-22T23:57:33.320902+0000","last_became_peered":"2024-05-22T23:57:33.320902+0000","last_unstale":"2024-05-22T23:58:31.112808+0000","last_undegraded":"2024-05-22T23:58:31.112808+0000","last_fullsized":"2024-05-22T23:58:31.112808+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:51:39.716541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.099567+0000","last_change":"2024-05-22T23:57:33.322638+0000","last_active":"2024-05-22T23:58:31.099567+0000","last_peered":"2024-05-22T23:58:31.099567+0000","last_clean":"2024-05-22T23:58:31.099567+0000","last_became_active":"2024-05-22T23:57:33.322318+0000","last_became_peered":"2024-05-22T23:57:33.322318+0000","last_unstale":"2024-05-22T23:58:31.099567+0000","last_undegraded":"2024-05-22T23:58:31.099567+0000","last_fullsized":"2024-05-22T23:58:31.099567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:44:25.487618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098935+0000","last_change":"2024-05-22T23:57:33.325832+0000","last_active":"2024-05-22T23:58:31.098935+0000","last_peered":"2024-05-22T23:58:31.098935+0000","last_clean":"2024-05-22T23:58:31.098935+0000","last_became_active":"2024-05-22T23:57:33.325062+0000","last_became_peered":"2024-05-22T23:57:33.325062+0000","last_unstale":"2024-05-22T23:58:31.098935+0000","last_undegraded":"2024-05-22T23:58:31.098935+0000","last_fullsized":"2024-05-22T23:58:31.098935+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:33:04.161351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.119980+0000","last_change":"2024-05-22T23:57:33.324864+0000","last_active":"2024-05-22T23:59:30.119980+0000","last_peered":"2024-05-22T23:59:30.119980+0000","last_clean":"2024-05-22T23:59:30.119980+0000","last_became_active":"2024-05-22T23:57:33.324558+0000","last_became_peered":"2024-05-22T23:57:33.324558+0000","last_unstale":"2024-05-22T23:59:30.119980+0000","last_undegraded":"2024-05-22T23:59:30.119980+0000","last_fullsized":"2024-05-22T23:59:30.119980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:48:49.722371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.119946+0000","last_change":"2024-05-22T23:57:33.337985+0000","last_active":"2024-05-22T23:59:30.119946+0000","last_peered":"2024-05-22T23:59:30.119946+0000","last_clean":"2024-05-22T23:59:30.119946+0000","last_became_active":"2024-05-22T23:57:33.337719+0000","last_became_peered":"2024-05-22T23:57:33.337719+0000","last_unstale":"2024-05-22T23:59:30.119946+0000","last_undegraded":"2024-05-22T23:59:30.119946+0000","last_fullsized":"2024-05-22T23:59:30.119946+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:50:44.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.103774+0000","last_change":"2024-05-22T23:57:33.325709+0000","last_active":"2024-05-22T23:59:30.103774+0000","last_peered":"2024-05-22T23:59:30.103774+0000","last_clean":"2024-05-22T23:59:30.103774+0000","last_became_active":"2024-05-22T23:57:33.325063+0000","last_became_peered":"2024-05-22T23:57:33.325063+0000","last_unstale":"2024-05-22T23:59:30.103774+0000","last_undegraded":"2024-05-22T23:59:30.103774+0000","last_fullsized":"2024-05-22T23:59:30.103774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:39.145217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108004+0000","last_change":"2024-05-22T23:57:33.333251+0000","last_active":"2024-05-22T23:59:30.108004+0000","last_peered":"2024-05-22T23:59:30.108004+0000","last_clean":"2024-05-22T23:59:30.108004+0000","last_became_active":"2024-05-22T23:57:33.332894+0000","last_became_peered":"2024-05-22T23:57:33.332894+0000","last_unstale":"2024-05-22T23:59:30.108004+0000","last_undegraded":"2024-05-22T23:59:30.108004+0000","last_fullsized":"2024-05-22T23:59:30.108004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:13:12.082781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098311+0000","last_change":"2024-05-22T23:57:37.788060+0000","last_active":"2024-05-22T23:58:31.098311+0000","last_peered":"2024-05-22T23:58:31.098311+0000","last_clean":"2024-05-22T23:58:31.098311+0000","last_became_active":"2024-05-22T23:57:37.787656+0000","last_became_peered":"2024-05-22T23:57:37.787656+0000","last_unstale":"2024-05-22T23:58:31.098311+0000","last_undegraded":"2024-05-22T23:58:31.098311+0000","last_fullsized":"2024-05-22T23:58:31.098311+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:39:13.718817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.109737+0000","last_change":"2024-05-22T23:57:32.323434+0000","last_active":"2024-05-22T23:58:31.109737+0000","last_peered":"2024-05-22T23:58:31.109737+0000","last_clean":"2024-05-22T23:58:31.109737+0000","last_became_active":"2024-05-22T23:57:32.323033+0000","last_became_peered":"2024-05-22T23:57:32.323033+0000","last_unstale":"2024-05-22T23:58:31.109737+0000","last_undegraded":"2024-05-22T23:58:31.109737+0000","last_fullsized":"2024-05-22T23:58:31.109737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:15:27.676027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.110459+0000","last_change":"2024-05-22T23:57:33.323336+0000","last_active":"2024-05-22T23:59:30.110459+0000","last_peered":"2024-05-22T23:59:30.110459+0000","last_clean":"2024-05-22T23:59:30.110459+0000","last_became_active":"2024-05-22T23:57:33.323026+0000","last_became_peered":"2024-05-22T23:57:33.323026+0000","last_unstale":"2024-05-22T23:59:30.110459+0000","last_undegraded":"2024-05-22T23:59:30.110459+0000","last_fullsized":"2024-05-22T23:59:30.110459+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:02:20.142760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096471+0000","last_change":"2024-05-22T23:57:37.785137+0000","last_active":"2024-05-22T23:58:31.096471+0000","last_peered":"2024-05-22T23:58:31.096471+0000","last_clean":"2024-05-22T23:58:31.096471+0000","last_became_active":"2024-05-22T23:57:37.784791+0000","last_became_peered":"2024-05-22T23:57:37.784791+0000","last_unstale":"2024-05-22T23:58:31.096471+0000","last_undegraded":"2024-05-22T23:58:31.096471+0000","last_fullsized":"2024-05-22T23:58:31.096471+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:54:51.036334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.099658+0000","last_change":"2024-05-22T23:57:32.332969+0000","last_active":"2024-05-22T23:58:31.099658+0000","last_peered":"2024-05-22T23:58:31.099658+0000","last_clean":"2024-05-22T23:58:31.099658+0000","last_became_active":"2024-05-22T23:57:32.332611+0000","last_became_peered":"2024-05-22T23:57:32.332611+0000","last_unstale":"2024-05-22T23:58:31.099658+0000","last_undegraded":"2024-05-22T23:58:31.099658+0000","last_fullsized":"2024-05-22T23:58:31.099658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:00:22.006355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101481+0000","last_change":"2024-05-22T23:57:33.332151+0000","last_active":"2024-05-22T23:58:31.101481+0000","last_peered":"2024-05-22T23:58:31.101481+0000","last_clean":"2024-05-22T23:58:31.101481+0000","last_became_active":"2024-05-22T23:57:33.329518+0000","last_became_peered":"2024-05-22T23:57:33.329518+0000","last_unstale":"2024-05-22T23:58:31.101481+0000","last_undegraded":"2024-05-22T23:58:31.101481+0000","last_fullsized":"2024-05-22T23:58:31.101481+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:10:12.083766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096570+0000","last_change":"2024-05-22T23:57:37.787524+0000","last_active":"2024-05-22T23:58:31.096570+0000","last_peered":"2024-05-22T23:58:31.096570+0000","last_clean":"2024-05-22T23:58:31.096570+0000","last_became_active":"2024-05-22T23:57:37.787103+0000","last_became_peered":"2024-05-22T23:57:37.787103+0000","last_unstale":"2024-05-22T23:58:31.096570+0000","last_undegraded":"2024-05-22T23:58:31.096570+0000","last_fullsized":"2024-05-22T23:58:31.096570+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:19:16.963159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108461+0000","last_change":"2024-05-22T23:57:32.318469+0000","last_active":"2024-05-22T23:58:31.108461+0000","last_peered":"2024-05-22T23:58:31.108461+0000","last_clean":"2024-05-22T23:58:31.108461+0000","last_became_active":"2024-05-22T23:57:32.317640+0000","last_became_peered":"2024-05-22T23:57:32.317640+0000","last_unstale":"2024-05-22T23:58:31.108461+0000","last_undegraded":"2024-05-22T23:58:31.108461+0000","last_fullsized":"2024-05-22T23:58:31.108461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:01:55.710019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.109225+0000","last_change":"2024-05-22T23:57:33.318604+0000","last_active":"2024-05-22T23:59:30.109225+0000","last_peered":"2024-05-22T23:59:30.109225+0000","last_clean":"2024-05-22T23:59:30.109225+0000","last_became_active":"2024-05-22T23:57:33.318184+0000","last_became_peered":"2024-05-22T23:57:33.318184+0000","last_unstale":"2024-05-22T23:59:30.109225+0000","last_undegraded":"2024-05-22T23:59:30.109225+0000","last_fullsized":"2024-05-22T23:59:30.109225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:16.140799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096467+0000","last_change":"2024-05-22T23:57:37.796085+0000","last_active":"2024-05-22T23:58:31.096467+0000","last_peered":"2024-05-22T23:58:31.096467+0000","last_clean":"2024-05-22T23:58:31.096467+0000","last_became_active":"2024-05-22T23:57:37.795265+0000","last_became_peered":"2024-05-22T23:57:37.795265+0000","last_unstale":"2024-05-22T23:58:31.096467+0000","last_undegraded":"2024-05-22T23:58:31.096467+0000","last_fullsized":"2024-05-22T23:58:31.096467+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:43:55.577072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108195+0000","last_change":"2024-05-22T23:57:32.316346+0000","last_active":"2024-05-22T23:58:31.108195+0000","last_peered":"2024-05-22T23:58:31.108195+0000","last_clean":"2024-05-22T23:58:31.108195+0000","last_became_active":"2024-05-22T23:57:32.316038+0000","last_became_peered":"2024-05-22T23:57:32.316038+0000","last_unstale":"2024-05-22T23:58:31.108195+0000","last_undegraded":"2024-05-22T23:58:31.108195+0000","last_fullsized":"2024-05-22T23:58:31.108195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:57:41.987250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102462+0000","last_change":"2024-05-22T23:57:33.329341+0000","last_active":"2024-05-22T23:58:31.102462+0000","last_peered":"2024-05-22T23:58:31.102462+0000","last_clean":"2024-05-22T23:58:31.102462+0000","last_became_active":"2024-05-22T23:57:33.329009+0000","last_became_peered":"2024-05-22T23:57:33.329009+0000","last_unstale":"2024-05-22T23:58:31.102462+0000","last_undegraded":"2024-05-22T23:58:31.102462+0000","last_fullsized":"2024-05-22T23:58:31.102462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:53:07.591497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098490+0000","last_change":"2024-05-22T23:57:37.779097+0000","last_active":"2024-05-22T23:58:31.098490+0000","last_peered":"2024-05-22T23:58:31.098490+0000","last_clean":"2024-05-22T23:58:31.098490+0000","last_became_active":"2024-05-22T23:57:37.778564+0000","last_became_peered":"2024-05-22T23:57:37.778564+0000","last_unstale":"2024-05-22T23:58:31.098490+0000","last_undegraded":"2024-05-22T23:58:31.098490+0000","last_fullsized":"2024-05-22T23:58:31.098490+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:09:16.351947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'6","reported_seq":132,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:28.193072+0000","last_change":"2024-05-22T23:58:31.109013+0000","last_active":"2024-05-22T23:59:28.193072+0000","last_peered":"2024-05-22T23:59:28.193072+0000","last_clean":"2024-05-22T23:59:28.193072+0000","last_became_active":"2024-05-22T23:58:31.103858+0000","last_became_peered":"2024-05-22T23:58:31.103858+0000","last_unstale":"2024-05-22T23:59:28.193072+0000","last_undegraded":"2024-05-22T23:59:28.193072+0000","last_fullsized":"2024-05-22T23:59:28.193072+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:20:08.783189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1012,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107759+0000","last_change":"2024-05-22T23:57:33.324587+0000","last_active":"2024-05-22T23:58:31.107759+0000","last_peered":"2024-05-22T23:58:31.107759+0000","last_clean":"2024-05-22T23:58:31.107759+0000","last_became_active":"2024-05-22T23:57:33.323972+0000","last_became_peered":"2024-05-22T23:57:33.323972+0000","last_unstale":"2024-05-22T23:58:31.107759+0000","last_undegraded":"2024-05-22T23:58:31.107759+0000","last_fullsized":"2024-05-22T23:58:31.107759+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:23:52.551699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098921+0000","last_change":"2024-05-22T23:57:37.791726+0000","last_active":"2024-05-22T23:58:31.098921+0000","last_peered":"2024-05-22T23:58:31.098921+0000","last_clean":"2024-05-22T23:58:31.098921+0000","last_became_active":"2024-05-22T23:57:37.791185+0000","last_became_peered":"2024-05-22T23:57:37.791185+0000","last_unstale":"2024-05-22T23:58:31.098921+0000","last_undegraded":"2024-05-22T23:58:31.098921+0000","last_fullsized":"2024-05-22T23:58:31.098921+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:55:30.994272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.116594+0000","last_change":"2024-05-22T23:58:31.116594+0000","last_active":"2024-05-22T23:58:31.116594+0000","last_peered":"2024-05-22T23:58:31.116594+0000","last_clean":"2024-05-22T23:58:31.116594+0000","last_became_active":"2024-05-22T23:58:31.113117+0000","last_became_peered":"2024-05-22T23:58:31.113117+0000","last_unstale":"2024-05-22T23:58:31.116594+0000","last_undegraded":"2024-05-22T23:58:31.116594+0000","last_fullsized":"2024-05-22T23:58:31.116594+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:05:30.758638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.116203+0000","last_change":"2024-05-22T23:58:31.116203+0000","last_active":"2024-05-22T23:58:31.116203+0000","last_peered":"2024-05-22T23:58:31.116203+0000","last_clean":"2024-05-22T23:58:31.116203+0000","last_became_active":"2024-05-22T23:58:31.115881+0000","last_became_peered":"2024-05-22T23:58:31.115881+0000","last_unstale":"2024-05-22T23:58:31.116203+0000","last_undegraded":"2024-05-22T23:58:31.116203+0000","last_fullsized":"2024-05-22T23:58:31.116203+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:39:31.247426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101119+0000","last_change":"2024-05-22T23:57:37.787910+0000","last_active":"2024-05-22T23:58:31.101119+0000","last_peered":"2024-05-22T23:58:31.101119+0000","last_clean":"2024-05-22T23:58:31.101119+0000","last_became_active":"2024-05-22T23:57:37.787465+0000","last_became_peered":"2024-05-22T23:57:37.787465+0000","last_unstale":"2024-05-22T23:58:31.101119+0000","last_undegraded":"2024-05-22T23:58:31.101119+0000","last_fullsized":"2024-05-22T23:58:31.101119+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:54:59.202855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107375+0000","last_change":"2024-05-22T23:57:32.329365+0000","last_active":"2024-05-22T23:58:31.107375+0000","last_peered":"2024-05-22T23:58:31.107375+0000","last_clean":"2024-05-22T23:58:31.107375+0000","last_became_active":"2024-05-22T23:57:32.327864+0000","last_became_peered":"2024-05-22T23:57:32.327864+0000","last_unstale":"2024-05-22T23:58:31.107375+0000","last_undegraded":"2024-05-22T23:58:31.107375+0000","last_fullsized":"2024-05-22T23:58:31.107375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:26.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.113454+0000","last_change":"2024-05-22T23:57:33.313930+0000","last_active":"2024-05-22T23:59:30.113454+0000","last_peered":"2024-05-22T23:59:30.113454+0000","last_clean":"2024-05-22T23:59:30.113454+0000","last_became_active":"2024-05-22T23:57:33.313612+0000","last_became_peered":"2024-05-22T23:57:33.313612+0000","last_unstale":"2024-05-22T23:59:30.113454+0000","last_undegraded":"2024-05-22T23:59:30.113454+0000","last_fullsized":"2024-05-22T23:59:30.113454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:45:25.800338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100765+0000","last_change":"2024-05-22T23:57:37.791537+0000","last_active":"2024-05-22T23:58:31.100765+0000","last_peered":"2024-05-22T23:58:31.100765+0000","last_clean":"2024-05-22T23:58:31.100765+0000","last_became_active":"2024-05-22T23:57:37.791052+0000","last_became_peered":"2024-05-22T23:57:37.791052+0000","last_unstale":"2024-05-22T23:58:31.100765+0000","last_undegraded":"2024-05-22T23:58:31.100765+0000","last_fullsized":"2024-05-22T23:58:31.100765+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:18:11.759840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100298+0000","last_change":"2024-05-22T23:57:32.315907+0000","last_active":"2024-05-22T23:58:31.100298+0000","last_peered":"2024-05-22T23:58:31.100298+0000","last_clean":"2024-05-22T23:58:31.100298+0000","last_became_active":"2024-05-22T23:57:32.315576+0000","last_became_peered":"2024-05-22T23:57:32.315576+0000","last_unstale":"2024-05-22T23:58:31.100298+0000","last_undegraded":"2024-05-22T23:58:31.100298+0000","last_fullsized":"2024-05-22T23:58:31.100298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:45:48.998377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.110282+0000","last_change":"2024-05-22T23:57:33.338681+0000","last_active":"2024-05-22T23:59:30.110282+0000","last_peered":"2024-05-22T23:59:30.110282+0000","last_clean":"2024-05-22T23:59:30.110282+0000","last_became_active":"2024-05-22T23:57:33.334705+0000","last_became_peered":"2024-05-22T23:57:33.334705+0000","last_unstale":"2024-05-22T23:59:30.110282+0000","last_undegraded":"2024-05-22T23:59:30.110282+0000","last_fullsized":"2024-05-22T23:59:30.110282+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:22.135137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100227+0000","last_change":"2024-05-22T23:57:37.780470+0000","last_active":"2024-05-22T23:58:31.100227+0000","last_peered":"2024-05-22T23:58:31.100227+0000","last_clean":"2024-05-22T23:58:31.100227+0000","last_became_active":"2024-05-22T23:57:37.779110+0000","last_became_peered":"2024-05-22T23:57:37.779110+0000","last_unstale":"2024-05-22T23:58:31.100227+0000","last_undegraded":"2024-05-22T23:58:31.100227+0000","last_fullsized":"2024-05-22T23:58:31.100227+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:55:45.486862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'3","reported_seq":112,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:26.189052+0000","last_change":"2024-05-22T23:57:32.322489+0000","last_active":"2024-05-22T23:59:26.189052+0000","last_peered":"2024-05-22T23:59:26.189052+0000","last_clean":"2024-05-22T23:59:26.189052+0000","last_became_active":"2024-05-22T23:57:32.322125+0000","last_became_peered":"2024-05-22T23:57:32.322125+0000","last_unstale":"2024-05-22T23:59:26.189052+0000","last_undegraded":"2024-05-22T23:59:26.189052+0000","last_fullsized":"2024-05-22T23:59:26.189052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:11:47.707506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108893+0000","last_change":"2024-05-22T23:57:33.322417+0000","last_active":"2024-05-22T23:59:30.108893+0000","last_peered":"2024-05-22T23:59:30.108893+0000","last_clean":"2024-05-22T23:59:30.108893+0000","last_became_active":"2024-05-22T23:57:33.322102+0000","last_became_peered":"2024-05-22T23:57:33.322102+0000","last_unstale":"2024-05-22T23:59:30.108893+0000","last_undegraded":"2024-05-22T23:59:30.108893+0000","last_fullsized":"2024-05-22T23:59:30.108893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:26:43.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108560+0000","last_change":"2024-05-22T23:57:37.784424+0000","last_active":"2024-05-22T23:58:31.108560+0000","last_peered":"2024-05-22T23:58:31.108560+0000","last_clean":"2024-05-22T23:58:31.108560+0000","last_became_active":"2024-05-22T23:57:37.783949+0000","last_became_peered":"2024-05-22T23:57:37.783949+0000","last_unstale":"2024-05-22T23:58:31.108560+0000","last_undegraded":"2024-05-22T23:58:31.108560+0000","last_fullsized":"2024-05-22T23:58:31.108560+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:52:23.763273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110764+0000","last_change":"2024-05-22T23:58:31.110764+0000","last_active":"2024-05-22T23:58:31.110764+0000","last_peered":"2024-05-22T23:58:31.110764+0000","last_clean":"2024-05-22T23:58:31.110764+0000","last_became_active":"2024-05-22T23:58:31.103962+0000","last_became_peered":"2024-05-22T23:58:31.103962+0000","last_unstale":"2024-05-22T23:58:31.110764+0000","last_undegraded":"2024-05-22T23:58:31.110764+0000","last_fullsized":"2024-05-22T23:58:31.110764+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:14:24.248179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102138+0000","last_change":"2024-05-22T23:57:33.324237+0000","last_active":"2024-05-22T23:58:31.102138+0000","last_peered":"2024-05-22T23:58:31.102138+0000","last_clean":"2024-05-22T23:58:31.102138+0000","last_became_active":"2024-05-22T23:57:33.323870+0000","last_became_peered":"2024-05-22T23:57:33.323870+0000","last_unstale":"2024-05-22T23:58:31.102138+0000","last_undegraded":"2024-05-22T23:58:31.102138+0000","last_fullsized":"2024-05-22T23:58:31.102138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:41:29.328104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096679+0000","last_change":"2024-05-22T23:57:37.787795+0000","last_active":"2024-05-22T23:58:31.096679+0000","last_peered":"2024-05-22T23:58:31.096679+0000","last_clean":"2024-05-22T23:58:31.096679+0000","last_became_active":"2024-05-22T23:57:37.787210+0000","last_became_peered":"2024-05-22T23:57:37.787210+0000","last_unstale":"2024-05-22T23:58:31.096679+0000","last_undegraded":"2024-05-22T23:58:31.096679+0000","last_fullsized":"2024-05-22T23:58:31.096679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:19:04.616758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102339+0000","last_change":"2024-05-22T23:57:32.321415+0000","last_active":"2024-05-22T23:58:31.102339+0000","last_peered":"2024-05-22T23:58:31.102339+0000","last_clean":"2024-05-22T23:58:31.102339+0000","last_became_active":"2024-05-22T23:57:32.321055+0000","last_became_peered":"2024-05-22T23:57:32.321055+0000","last_unstale":"2024-05-22T23:58:31.102339+0000","last_undegraded":"2024-05-22T23:58:31.102339+0000","last_fullsized":"2024-05-22T23:58:31.102339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:28:58.577548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.109609+0000","last_change":"2024-05-22T23:58:31.116991+0000","last_active":"2024-05-22T23:59:30.109609+0000","last_peered":"2024-05-22T23:59:30.109609+0000","last_clean":"2024-05-22T23:59:30.109609+0000","last_became_active":"2024-05-22T23:58:31.116526+0000","last_became_peered":"2024-05-22T23:58:31.116526+0000","last_unstale":"2024-05-22T23:59:30.109609+0000","last_undegraded":"2024-05-22T23:59:30.109609+0000","last_fullsized":"2024-05-22T23:59:30.109609+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:51:23.350871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.109785+0000","last_change":"2024-05-22T23:57:37.781367+0000","last_active":"2024-05-22T23:58:31.109785+0000","last_peered":"2024-05-22T23:58:31.109785+0000","last_clean":"2024-05-22T23:58:31.109785+0000","last_became_active":"2024-05-22T23:57:37.780955+0000","last_became_peered":"2024-05-22T23:57:37.780955+0000","last_unstale":"2024-05-22T23:58:31.109785+0000","last_undegraded":"2024-05-22T23:58:31.109785+0000","last_fullsized":"2024-05-22T23:58:31.109785+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:36:18.962940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'34","reported_seq":233,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:27.190767+0000","last_change":"2024-05-22T23:57:32.313873+0000","last_active":"2024-05-22T23:59:27.190767+0000","last_peered":"2024-05-22T23:59:27.190767+0000","last_clean":"2024-05-22T23:59:27.190767+0000","last_became_active":"2024-05-22T23:57:32.313388+0000","last_became_peered":"2024-05-22T23:57:32.313388+0000","last_unstale":"2024-05-22T23:59:27.190767+0000","last_undegraded":"2024-05-22T23:59:27.190767+0000","last_fullsized":"2024-05-22T23:59:27.190767+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:16.383384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62331,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":172,"num_read_kb":146,"num_write":35,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108117+0000","last_change":"2024-05-22T23:57:33.318751+0000","last_active":"2024-05-22T23:58:31.108117+0000","last_peered":"2024-05-22T23:58:31.108117+0000","last_clean":"2024-05-22T23:58:31.108117+0000","last_became_active":"2024-05-22T23:57:33.317710+0000","last_became_peered":"2024-05-22T23:57:33.317710+0000","last_unstale":"2024-05-22T23:58:31.108117+0000","last_undegraded":"2024-05-22T23:58:31.108117+0000","last_fullsized":"2024-05-22T23:58:31.108117+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:57:50.852615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102879+0000","last_change":"2024-05-22T23:57:37.790073+0000","last_active":"2024-05-22T23:58:31.102879+0000","last_peered":"2024-05-22T23:58:31.102879+0000","last_clean":"2024-05-22T23:58:31.102879+0000","last_became_active":"2024-05-22T23:57:37.789477+0000","last_became_peered":"2024-05-22T23:57:37.789477+0000","last_unstale":"2024-05-22T23:58:31.102879+0000","last_undegraded":"2024-05-22T23:58:31.102879+0000","last_fullsized":"2024-05-22T23:58:31.102879+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:11.003201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098794+0000","last_change":"2024-05-22T23:57:32.329846+0000","last_active":"2024-05-22T23:58:31.098794+0000","last_peered":"2024-05-22T23:58:31.098794+0000","last_clean":"2024-05-22T23:58:31.098794+0000","last_became_active":"2024-05-22T23:57:32.329414+0000","last_became_peered":"2024-05-22T23:57:32.329414+0000","last_unstale":"2024-05-22T23:58:31.098794+0000","last_undegraded":"2024-05-22T23:58:31.098794+0000","last_fullsized":"2024-05-22T23:58:31.098794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:15:38.176790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108431+0000","last_change":"2024-05-22T23:57:33.333298+0000","last_active":"2024-05-22T23:58:31.108431+0000","last_peered":"2024-05-22T23:58:31.108431+0000","last_clean":"2024-05-22T23:58:31.108431+0000","last_became_active":"2024-05-22T23:57:33.332934+0000","last_became_peered":"2024-05-22T23:57:33.332934+0000","last_unstale":"2024-05-22T23:58:31.108431+0000","last_undegraded":"2024-05-22T23:58:31.108431+0000","last_fullsized":"2024-05-22T23:58:31.108431+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:06:00.427627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100781+0000","last_change":"2024-05-22T23:57:37.795628+0000","last_active":"2024-05-22T23:58:31.100781+0000","last_peered":"2024-05-22T23:58:31.100781+0000","last_clean":"2024-05-22T23:58:31.100781+0000","last_became_active":"2024-05-22T23:57:37.795208+0000","last_became_peered":"2024-05-22T23:57:37.795208+0000","last_unstale":"2024-05-22T23:58:31.100781+0000","last_undegraded":"2024-05-22T23:58:31.100781+0000","last_fullsized":"2024-05-22T23:58:31.100781+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:03:12.684474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102064+0000","last_change":"2024-05-22T23:57:32.334039+0000","last_active":"2024-05-22T23:58:31.102064+0000","last_peered":"2024-05-22T23:58:31.102064+0000","last_clean":"2024-05-22T23:58:31.102064+0000","last_became_active":"2024-05-22T23:57:32.333709+0000","last_became_peered":"2024-05-22T23:57:32.333709+0000","last_unstale":"2024-05-22T23:58:31.102064+0000","last_undegraded":"2024-05-22T23:58:31.102064+0000","last_fullsized":"2024-05-22T23:58:31.102064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:41:43.319094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.118518+0000","last_change":"2024-05-22T23:57:33.328056+0000","last_active":"2024-05-22T23:59:30.118518+0000","last_peered":"2024-05-22T23:59:30.118518+0000","last_clean":"2024-05-22T23:59:30.118518+0000","last_became_active":"2024-05-22T23:57:33.326559+0000","last_became_peered":"2024-05-22T23:57:33.326559+0000","last_unstale":"2024-05-22T23:59:30.118518+0000","last_undegraded":"2024-05-22T23:59:30.118518+0000","last_fullsized":"2024-05-22T23:59:30.118518+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:06.463837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110580+0000","last_change":"2024-05-22T23:57:37.795892+0000","last_active":"2024-05-22T23:58:31.110580+0000","last_peered":"2024-05-22T23:58:31.110580+0000","last_clean":"2024-05-22T23:58:31.110580+0000","last_became_active":"2024-05-22T23:57:37.792246+0000","last_became_peered":"2024-05-22T23:57:37.792246+0000","last_unstale":"2024-05-22T23:58:31.110580+0000","last_undegraded":"2024-05-22T23:58:31.110580+0000","last_fullsized":"2024-05-22T23:58:31.110580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:18.462267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":118,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:27.416988+0000","last_change":"2024-05-22T23:57:32.319379+0000","last_active":"2024-05-22T23:59:27.416988+0000","last_peered":"2024-05-22T23:59:27.416988+0000","last_clean":"2024-05-22T23:59:27.416988+0000","last_became_active":"2024-05-22T23:57:32.318803+0000","last_became_peered":"2024-05-22T23:57:32.318803+0000","last_unstale":"2024-05-22T23:59:27.416988+0000","last_undegraded":"2024-05-22T23:59:27.416988+0000","last_fullsized":"2024-05-22T23:59:27.416988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:59:15.689119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108216+0000","last_change":"2024-05-22T23:57:33.325294+0000","last_active":"2024-05-22T23:59:30.108216+0000","last_peered":"2024-05-22T23:59:30.108216+0000","last_clean":"2024-05-22T23:59:30.108216+0000","last_became_active":"2024-05-22T23:57:33.324339+0000","last_became_peered":"2024-05-22T23:57:33.324339+0000","last_unstale":"2024-05-22T23:59:30.108216+0000","last_undegraded":"2024-05-22T23:59:30.108216+0000","last_fullsized":"2024-05-22T23:59:30.108216+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:29:47.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107019+0000","last_change":"2024-05-22T23:57:37.797961+0000","last_active":"2024-05-22T23:58:31.107019+0000","last_peered":"2024-05-22T23:58:31.107019+0000","last_clean":"2024-05-22T23:58:31.107019+0000","last_became_active":"2024-05-22T23:57:37.797532+0000","last_became_peered":"2024-05-22T23:57:37.797532+0000","last_unstale":"2024-05-22T23:58:31.107019+0000","last_undegraded":"2024-05-22T23:58:31.107019+0000","last_fullsized":"2024-05-22T23:58:31.107019+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:51:12.997674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107627+0000","last_change":"2024-05-22T23:57:32.317434+0000","last_active":"2024-05-22T23:58:31.107627+0000","last_peered":"2024-05-22T23:58:31.107627+0000","last_clean":"2024-05-22T23:58:31.107627+0000","last_became_active":"2024-05-22T23:57:32.316987+0000","last_became_peered":"2024-05-22T23:57:32.316987+0000","last_unstale":"2024-05-22T23:58:31.107627+0000","last_undegraded":"2024-05-22T23:58:31.107627+0000","last_fullsized":"2024-05-22T23:58:31.107627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:46:50.306756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101403+0000","last_change":"2024-05-22T23:57:33.328670+0000","last_active":"2024-05-22T23:58:31.101403+0000","last_peered":"2024-05-22T23:58:31.101403+0000","last_clean":"2024-05-22T23:58:31.101403+0000","last_became_active":"2024-05-22T23:57:33.328294+0000","last_became_peered":"2024-05-22T23:57:33.328294+0000","last_unstale":"2024-05-22T23:58:31.101403+0000","last_undegraded":"2024-05-22T23:58:31.101403+0000","last_fullsized":"2024-05-22T23:58:31.101403+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:09:10.807844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101376+0000","last_change":"2024-05-22T23:57:37.788885+0000","last_active":"2024-05-22T23:58:31.101376+0000","last_peered":"2024-05-22T23:58:31.101376+0000","last_clean":"2024-05-22T23:58:31.101376+0000","last_became_active":"2024-05-22T23:57:37.788145+0000","last_became_peered":"2024-05-22T23:57:37.788145+0000","last_unstale":"2024-05-22T23:58:31.101376+0000","last_undegraded":"2024-05-22T23:58:31.101376+0000","last_fullsized":"2024-05-22T23:58:31.101376+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:14:50.027785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108211+0000","last_change":"2024-05-22T23:57:32.317855+0000","last_active":"2024-05-22T23:58:31.108211+0000","last_peered":"2024-05-22T23:58:31.108211+0000","last_clean":"2024-05-22T23:58:31.108211+0000","last_became_active":"2024-05-22T23:57:32.317087+0000","last_became_peered":"2024-05-22T23:57:32.317087+0000","last_unstale":"2024-05-22T23:58:31.108211+0000","last_undegraded":"2024-05-22T23:58:31.108211+0000","last_fullsized":"2024-05-22T23:58:31.108211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:49:40.486222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.093428+0000","last_change":"2024-05-22T23:57:33.323897+0000","last_active":"2024-05-22T23:59:30.093428+0000","last_peered":"2024-05-22T23:59:30.093428+0000","last_clean":"2024-05-22T23:59:30.093428+0000","last_became_active":"2024-05-22T23:57:33.323567+0000","last_became_peered":"2024-05-22T23:57:33.323567+0000","last_unstale":"2024-05-22T23:59:30.093428+0000","last_undegraded":"2024-05-22T23:59:30.093428+0000","last_fullsized":"2024-05-22T23:59:30.093428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:17:48.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107731+0000","last_change":"2024-05-22T23:57:38.218313+0000","last_active":"2024-05-22T23:58:31.107731+0000","last_peered":"2024-05-22T23:58:31.107731+0000","last_clean":"2024-05-22T23:58:31.107731+0000","last_became_active":"2024-05-22T23:57:38.217756+0000","last_became_peered":"2024-05-22T23:57:38.217756+0000","last_unstale":"2024-05-22T23:58:31.107731+0000","last_undegraded":"2024-05-22T23:58:31.107731+0000","last_fullsized":"2024-05-22T23:58:31.107731+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:07:55.122260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110929+0000","last_change":"2024-05-22T23:57:32.325729+0000","last_active":"2024-05-22T23:58:31.110929+0000","last_peered":"2024-05-22T23:58:31.110929+0000","last_clean":"2024-05-22T23:58:31.110929+0000","last_became_active":"2024-05-22T23:57:32.325383+0000","last_became_peered":"2024-05-22T23:57:32.325383+0000","last_unstale":"2024-05-22T23:58:31.110929+0000","last_undegraded":"2024-05-22T23:58:31.110929+0000","last_fullsized":"2024-05-22T23:58:31.110929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:57.978806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.105836+0000","last_change":"2024-05-22T23:57:33.337499+0000","last_active":"2024-05-22T23:58:31.105836+0000","last_peered":"2024-05-22T23:58:31.105836+0000","last_clean":"2024-05-22T23:58:31.105836+0000","last_became_active":"2024-05-22T23:57:33.337271+0000","last_became_peered":"2024-05-22T23:57:33.337271+0000","last_unstale":"2024-05-22T23:58:31.105836+0000","last_undegraded":"2024-05-22T23:58:31.105836+0000","last_fullsized":"2024-05-22T23:58:31.105836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:32:39.133984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.111255+0000","last_change":"2024-05-22T23:57:37.792707+0000","last_active":"2024-05-22T23:58:31.111255+0000","last_peered":"2024-05-22T23:58:31.111255+0000","last_clean":"2024-05-22T23:58:31.111255+0000","last_became_active":"2024-05-22T23:57:37.792209+0000","last_became_peered":"2024-05-22T23:57:37.792209+0000","last_unstale":"2024-05-22T23:58:31.111255+0000","last_undegraded":"2024-05-22T23:58:31.111255+0000","last_fullsized":"2024-05-22T23:58:31.111255+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:36:34.089424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.112612+0000","last_change":"2024-05-22T23:58:31.112612+0000","last_active":"2024-05-22T23:58:31.112612+0000","last_peered":"2024-05-22T23:58:31.112612+0000","last_clean":"2024-05-22T23:58:31.112612+0000","last_became_active":"2024-05-22T23:58:31.112181+0000","last_became_peered":"2024-05-22T23:58:31.112181+0000","last_unstale":"2024-05-22T23:58:31.112612+0000","last_undegraded":"2024-05-22T23:58:31.112612+0000","last_fullsized":"2024-05-22T23:58:31.112612+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:25:18.717294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.104949+0000","last_change":"2024-05-22T23:57:33.328129+0000","last_active":"2024-05-22T23:58:31.104949+0000","last_peered":"2024-05-22T23:58:31.104949+0000","last_clean":"2024-05-22T23:58:31.104949+0000","last_became_active":"2024-05-22T23:57:33.327732+0000","last_became_peered":"2024-05-22T23:57:33.327732+0000","last_unstale":"2024-05-22T23:58:31.104949+0000","last_undegraded":"2024-05-22T23:58:31.104949+0000","last_fullsized":"2024-05-22T23:58:31.104949+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:25:14.105235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107157+0000","last_change":"2024-05-22T23:57:38.220884+0000","last_active":"2024-05-22T23:58:31.107157+0000","last_peered":"2024-05-22T23:58:31.107157+0000","last_clean":"2024-05-22T23:58:31.107157+0000","last_became_active":"2024-05-22T23:57:38.219365+0000","last_became_peered":"2024-05-22T23:57:38.219365+0000","last_unstale":"2024-05-22T23:58:31.107157+0000","last_undegraded":"2024-05-22T23:58:31.107157+0000","last_fullsized":"2024-05-22T23:58:31.107157+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:54:56.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100581+0000","last_change":"2024-05-22T23:57:33.328272+0000","last_active":"2024-05-22T23:58:31.100581+0000","last_peered":"2024-05-22T23:58:31.100581+0000","last_clean":"2024-05-22T23:58:31.100581+0000","last_became_active":"2024-05-22T23:57:33.327915+0000","last_became_peered":"2024-05-22T23:57:33.327915+0000","last_unstale":"2024-05-22T23:58:31.100581+0000","last_undegraded":"2024-05-22T23:58:31.100581+0000","last_fullsized":"2024-05-22T23:58:31.100581+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:59:09.085649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'4","reported_seq":115,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:27.416440+0000","last_change":"2024-05-22T23:57:32.331979+0000","last_active":"2024-05-22T23:59:27.416440+0000","last_peered":"2024-05-22T23:59:27.416440+0000","last_clean":"2024-05-22T23:59:27.416440+0000","last_became_active":"2024-05-22T23:57:32.330983+0000","last_became_peered":"2024-05-22T23:57:32.330983+0000","last_unstale":"2024-05-22T23:59:27.416440+0000","last_undegraded":"2024-05-22T23:59:27.416440+0000","last_fullsized":"2024-05-22T23:59:27.416440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:07:19.137652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101829+0000","last_change":"2024-05-22T23:57:37.787935+0000","last_active":"2024-05-22T23:58:31.101829+0000","last_peered":"2024-05-22T23:58:31.101829+0000","last_clean":"2024-05-22T23:58:31.101829+0000","last_became_active":"2024-05-22T23:57:37.787495+0000","last_became_peered":"2024-05-22T23:57:37.787495+0000","last_unstale":"2024-05-22T23:58:31.101829+0000","last_undegraded":"2024-05-22T23:58:31.101829+0000","last_fullsized":"2024-05-22T23:58:31.101829+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:37.682711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.111144+0000","last_change":"2024-05-22T23:57:33.329451+0000","last_active":"2024-05-22T23:59:30.111144+0000","last_peered":"2024-05-22T23:59:30.111144+0000","last_clean":"2024-05-22T23:59:30.111144+0000","last_became_active":"2024-05-22T23:57:33.329107+0000","last_became_peered":"2024-05-22T23:57:33.329107+0000","last_unstale":"2024-05-22T23:59:30.111144+0000","last_undegraded":"2024-05-22T23:59:30.111144+0000","last_fullsized":"2024-05-22T23:59:30.111144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:44:50.198040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096649+0000","last_change":"2024-05-22T23:57:32.321873+0000","last_active":"2024-05-22T23:58:31.096649+0000","last_peered":"2024-05-22T23:58:31.096649+0000","last_clean":"2024-05-22T23:58:31.096649+0000","last_became_active":"2024-05-22T23:57:32.321550+0000","last_became_peered":"2024-05-22T23:57:32.321550+0000","last_unstale":"2024-05-22T23:58:31.096649+0000","last_undegraded":"2024-05-22T23:58:31.096649+0000","last_fullsized":"2024-05-22T23:58:31.096649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:30:17.479238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101701+0000","last_change":"2024-05-22T23:56:11.327788+0000","last_active":"2024-05-22T23:58:31.101701+0000","last_peered":"2024-05-22T23:58:31.101701+0000","last_clean":"2024-05-22T23:58:31.101701+0000","last_became_active":"2024-05-22T23:56:11.327232+0000","last_became_peered":"2024-05-22T23:56:11.327232+0000","last_unstale":"2024-05-22T23:58:31.101701+0000","last_undegraded":"2024-05-22T23:58:31.101701+0000","last_fullsized":"2024-05-22T23:58:31.101701+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_clean_scrub_stamp":"2024-05-22T23:52:16.719643+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:26:13.077470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102591+0000","last_change":"2024-05-22T23:57:38.219326+0000","last_active":"2024-05-22T23:58:31.102591+0000","last_peered":"2024-05-22T23:58:31.102591+0000","last_clean":"2024-05-22T23:58:31.102591+0000","last_became_active":"2024-05-22T23:57:38.218835+0000","last_became_peered":"2024-05-22T23:57:38.218835+0000","last_unstale":"2024-05-22T23:58:31.102591+0000","last_undegraded":"2024-05-22T23:58:31.102591+0000","last_fullsized":"2024-05-22T23:58:31.102591+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:29.722862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.106697+0000","last_change":"2024-05-22T23:57:33.316760+0000","last_active":"2024-05-22T23:58:31.106697+0000","last_peered":"2024-05-22T23:58:31.106697+0000","last_clean":"2024-05-22T23:58:31.106697+0000","last_became_active":"2024-05-22T23:57:33.316328+0000","last_became_peered":"2024-05-22T23:57:33.316328+0000","last_unstale":"2024-05-22T23:58:31.106697+0000","last_undegraded":"2024-05-22T23:58:31.106697+0000","last_fullsized":"2024-05-22T23:58:31.106697+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:53:11.592134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102301+0000","last_change":"2024-05-22T23:57:32.319154+0000","last_active":"2024-05-22T23:58:31.102301+0000","last_peered":"2024-05-22T23:58:31.102301+0000","last_clean":"2024-05-22T23:58:31.102301+0000","last_became_active":"2024-05-22T23:57:32.318808+0000","last_became_peered":"2024-05-22T23:57:32.318808+0000","last_unstale":"2024-05-22T23:58:31.102301+0000","last_undegraded":"2024-05-22T23:58:31.102301+0000","last_fullsized":"2024-05-22T23:58:31.102301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:37:47.271767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101063+0000","last_change":"2024-05-22T23:57:37.791464+0000","last_active":"2024-05-22T23:58:31.101063+0000","last_peered":"2024-05-22T23:58:31.101063+0000","last_clean":"2024-05-22T23:58:31.101063+0000","last_became_active":"2024-05-22T23:57:37.791044+0000","last_became_peered":"2024-05-22T23:57:37.791044+0000","last_unstale":"2024-05-22T23:58:31.101063+0000","last_undegraded":"2024-05-22T23:58:31.101063+0000","last_fullsized":"2024-05-22T23:58:31.101063+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:47:00.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108206+0000","last_change":"2024-05-22T23:57:32.322740+0000","last_active":"2024-05-22T23:58:31.108206+0000","last_peered":"2024-05-22T23:58:31.108206+0000","last_clean":"2024-05-22T23:58:31.108206+0000","last_became_active":"2024-05-22T23:57:32.322425+0000","last_became_peered":"2024-05-22T23:57:32.322425+0000","last_unstale":"2024-05-22T23:58:31.108206+0000","last_undegraded":"2024-05-22T23:58:31.108206+0000","last_fullsized":"2024-05-22T23:58:31.108206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:05:10.522462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110858+0000","last_change":"2024-05-22T23:57:33.324667+0000","last_active":"2024-05-22T23:58:31.110858+0000","last_peered":"2024-05-22T23:58:31.110858+0000","last_clean":"2024-05-22T23:58:31.110858+0000","last_became_active":"2024-05-22T23:57:33.324251+0000","last_became_peered":"2024-05-22T23:57:33.324251+0000","last_unstale":"2024-05-22T23:58:31.110858+0000","last_undegraded":"2024-05-22T23:58:31.110858+0000","last_fullsized":"2024-05-22T23:58:31.110858+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:49:57.203996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097700+0000","last_change":"2024-05-22T23:57:38.222346+0000","last_active":"2024-05-22T23:58:31.097700+0000","last_peered":"2024-05-22T23:58:31.097700+0000","last_clean":"2024-05-22T23:58:31.097700+0000","last_became_active":"2024-05-22T23:57:38.220248+0000","last_became_peered":"2024-05-22T23:57:38.220248+0000","last_unstale":"2024-05-22T23:58:31.097700+0000","last_undegraded":"2024-05-22T23:58:31.097700+0000","last_fullsized":"2024-05-22T23:58:31.097700+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:26:44.026641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'9","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.126975+0000","last_change":"2024-05-22T23:58:31.126056+0000","last_active":"2024-05-22T23:58:31.126975+0000","last_peered":"2024-05-22T23:58:31.126975+0000","last_clean":"2024-05-22T23:58:31.126975+0000","last_became_active":"2024-05-22T23:58:31.117186+0000","last_became_peered":"2024-05-22T23:58:31.117186+0000","last_unstale":"2024-05-22T23:58:31.126975+0000","last_undegraded":"2024-05-22T23:58:31.126975+0000","last_fullsized":"2024-05-22T23:58:31.126975+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:21:34.156655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.097685+0000","last_change":"2024-05-22T23:58:31.116649+0000","last_active":"2024-05-22T23:59:30.097685+0000","last_peered":"2024-05-22T23:59:30.097685+0000","last_clean":"2024-05-22T23:59:30.097685+0000","last_became_active":"2024-05-22T23:58:31.116009+0000","last_became_peered":"2024-05-22T23:58:31.116009+0000","last_unstale":"2024-05-22T23:59:30.097685+0000","last_undegraded":"2024-05-22T23:59:30.097685+0000","last_fullsized":"2024-05-22T23:59:30.097685+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:00:29.964565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098180+0000","last_change":"2024-05-22T23:57:37.778524+0000","last_active":"2024-05-22T23:58:31.098180+0000","last_peered":"2024-05-22T23:58:31.098180+0000","last_clean":"2024-05-22T23:58:31.098180+0000","last_became_active":"2024-05-22T23:57:37.777827+0000","last_became_peered":"2024-05-22T23:57:37.777827+0000","last_unstale":"2024-05-22T23:58:31.098180+0000","last_undegraded":"2024-05-22T23:58:31.098180+0000","last_fullsized":"2024-05-22T23:58:31.098180+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:30:59.154921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097105+0000","last_change":"2024-05-22T23:57:32.317649+0000","last_active":"2024-05-22T23:58:31.097105+0000","last_peered":"2024-05-22T23:58:31.097105+0000","last_clean":"2024-05-22T23:58:31.097105+0000","last_became_active":"2024-05-22T23:57:32.317280+0000","last_became_peered":"2024-05-22T23:57:32.317280+0000","last_unstale":"2024-05-22T23:58:31.097105+0000","last_undegraded":"2024-05-22T23:58:31.097105+0000","last_fullsized":"2024-05-22T23:58:31.097105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:01:16.917079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101548+0000","last_change":"2024-05-22T23:57:33.339296+0000","last_active":"2024-05-22T23:58:31.101548+0000","last_peered":"2024-05-22T23:58:31.101548+0000","last_clean":"2024-05-22T23:58:31.101548+0000","last_became_active":"2024-05-22T23:57:33.338457+0000","last_became_peered":"2024-05-22T23:57:33.338457+0000","last_unstale":"2024-05-22T23:58:31.101548+0000","last_undegraded":"2024-05-22T23:58:31.101548+0000","last_fullsized":"2024-05-22T23:58:31.101548+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:13:05.927893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097166+0000","last_change":"2024-05-22T23:57:38.218649+0000","last_active":"2024-05-22T23:58:31.097166+0000","last_peered":"2024-05-22T23:58:31.097166+0000","last_clean":"2024-05-22T23:58:31.097166+0000","last_became_active":"2024-05-22T23:57:38.218189+0000","last_became_peered":"2024-05-22T23:57:38.218189+0000","last_unstale":"2024-05-22T23:58:31.097166+0000","last_undegraded":"2024-05-22T23:58:31.097166+0000","last_fullsized":"2024-05-22T23:58:31.097166+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:30.747293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098608+0000","last_change":"2024-05-22T23:57:32.309379+0000","last_active":"2024-05-22T23:58:31.098608+0000","last_peered":"2024-05-22T23:58:31.098608+0000","last_clean":"2024-05-22T23:58:31.098608+0000","last_became_active":"2024-05-22T23:57:32.309068+0000","last_became_peered":"2024-05-22T23:57:32.309068+0000","last_unstale":"2024-05-22T23:58:31.098608+0000","last_undegraded":"2024-05-22T23:58:31.098608+0000","last_fullsized":"2024-05-22T23:58:31.098608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:43:54.787996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107824+0000","last_change":"2024-05-22T23:57:33.320253+0000","last_active":"2024-05-22T23:58:31.107824+0000","last_peered":"2024-05-22T23:58:31.107824+0000","last_clean":"2024-05-22T23:58:31.107824+0000","last_became_active":"2024-05-22T23:57:33.319604+0000","last_became_peered":"2024-05-22T23:57:33.319604+0000","last_unstale":"2024-05-22T23:58:31.107824+0000","last_undegraded":"2024-05-22T23:58:31.107824+0000","last_fullsized":"2024-05-22T23:58:31.107824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:23:51.745040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110063+0000","last_change":"2024-05-22T23:57:37.795368+0000","last_active":"2024-05-22T23:58:31.110063+0000","last_peered":"2024-05-22T23:58:31.110063+0000","last_clean":"2024-05-22T23:58:31.110063+0000","last_became_active":"2024-05-22T23:57:37.792848+0000","last_became_peered":"2024-05-22T23:57:37.792848+0000","last_unstale":"2024-05-22T23:58:31.110063+0000","last_undegraded":"2024-05-22T23:58:31.110063+0000","last_fullsized":"2024-05-22T23:58:31.110063+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:45.859867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098291+0000","last_change":"2024-05-22T23:57:32.329147+0000","last_active":"2024-05-22T23:58:31.098291+0000","last_peered":"2024-05-22T23:58:31.098291+0000","last_clean":"2024-05-22T23:58:31.098291+0000","last_became_active":"2024-05-22T23:57:32.327254+0000","last_became_peered":"2024-05-22T23:57:32.327254+0000","last_unstale":"2024-05-22T23:58:31.098291+0000","last_undegraded":"2024-05-22T23:58:31.098291+0000","last_fullsized":"2024-05-22T23:58:31.098291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:55:24.915817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.119795+0000","last_change":"2024-05-22T23:57:33.332843+0000","last_active":"2024-05-22T23:59:30.119795+0000","last_peered":"2024-05-22T23:59:30.119795+0000","last_clean":"2024-05-22T23:59:30.119795+0000","last_became_active":"2024-05-22T23:57:33.332497+0000","last_became_peered":"2024-05-22T23:57:33.332497+0000","last_unstale":"2024-05-22T23:59:30.119795+0000","last_undegraded":"2024-05-22T23:59:30.119795+0000","last_fullsized":"2024-05-22T23:59:30.119795+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:55:16.109676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096973+0000","last_change":"2024-05-22T23:57:37.787609+0000","last_active":"2024-05-22T23:58:31.096973+0000","last_peered":"2024-05-22T23:58:31.096973+0000","last_clean":"2024-05-22T23:58:31.096973+0000","last_became_active":"2024-05-22T23:57:37.787153+0000","last_became_peered":"2024-05-22T23:57:37.787153+0000","last_unstale":"2024-05-22T23:58:31.096973+0000","last_undegraded":"2024-05-22T23:58:31.096973+0000","last_fullsized":"2024-05-22T23:58:31.096973+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:57:51.640037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098316+0000","last_change":"2024-05-22T23:57:32.329784+0000","last_active":"2024-05-22T23:58:31.098316+0000","last_peered":"2024-05-22T23:58:31.098316+0000","last_clean":"2024-05-22T23:58:31.098316+0000","last_became_active":"2024-05-22T23:57:32.329393+0000","last_became_peered":"2024-05-22T23:57:32.329393+0000","last_unstale":"2024-05-22T23:58:31.098316+0000","last_undegraded":"2024-05-22T23:58:31.098316+0000","last_fullsized":"2024-05-22T23:58:31.098316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:17:54.811962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.122622+0000","last_change":"2024-05-22T23:57:33.318803+0000","last_active":"2024-05-22T23:59:30.122622+0000","last_peered":"2024-05-22T23:59:30.122622+0000","last_clean":"2024-05-22T23:59:30.122622+0000","last_became_active":"2024-05-22T23:57:33.318170+0000","last_became_peered":"2024-05-22T23:57:33.318170+0000","last_unstale":"2024-05-22T23:59:30.122622+0000","last_undegraded":"2024-05-22T23:59:30.122622+0000","last_fullsized":"2024-05-22T23:59:30.122622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:56:56.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097015+0000","last_change":"2024-05-22T23:57:38.218583+0000","last_active":"2024-05-22T23:58:31.097015+0000","last_peered":"2024-05-22T23:58:31.097015+0000","last_clean":"2024-05-22T23:58:31.097015+0000","last_became_active":"2024-05-22T23:57:38.218048+0000","last_became_peered":"2024-05-22T23:57:38.218048+0000","last_unstale":"2024-05-22T23:58:31.097015+0000","last_undegraded":"2024-05-22T23:58:31.097015+0000","last_fullsized":"2024-05-22T23:58:31.097015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:10:57.579656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107852+0000","last_change":"2024-05-22T23:57:32.317908+0000","last_active":"2024-05-22T23:58:31.107852+0000","last_peered":"2024-05-22T23:58:31.107852+0000","last_clean":"2024-05-22T23:58:31.107852+0000","last_became_active":"2024-05-22T23:57:32.317580+0000","last_became_peered":"2024-05-22T23:57:32.317580+0000","last_unstale":"2024-05-22T23:58:31.107852+0000","last_undegraded":"2024-05-22T23:58:31.107852+0000","last_fullsized":"2024-05-22T23:58:31.107852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:50:02.044817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.109257+0000","last_change":"2024-05-22T23:57:33.321184+0000","last_active":"2024-05-22T23:59:30.109257+0000","last_peered":"2024-05-22T23:59:30.109257+0000","last_clean":"2024-05-22T23:59:30.109257+0000","last_became_active":"2024-05-22T23:57:33.320418+0000","last_became_peered":"2024-05-22T23:57:33.320418+0000","last_unstale":"2024-05-22T23:59:30.109257+0000","last_undegraded":"2024-05-22T23:59:30.109257+0000","last_fullsized":"2024-05-22T23:59:30.109257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:22:43.019809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098941+0000","last_change":"2024-05-22T23:57:37.795718+0000","last_active":"2024-05-22T23:58:31.098941+0000","last_peered":"2024-05-22T23:58:31.098941+0000","last_clean":"2024-05-22T23:58:31.098941+0000","last_became_active":"2024-05-22T23:57:37.795297+0000","last_became_peered":"2024-05-22T23:57:37.795297+0000","last_unstale":"2024-05-22T23:58:31.098941+0000","last_undegraded":"2024-05-22T23:58:31.098941+0000","last_fullsized":"2024-05-22T23:58:31.098941+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:56:47.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.099085+0000","last_change":"2024-05-22T23:57:32.311921+0000","last_active":"2024-05-22T23:58:31.099085+0000","last_peered":"2024-05-22T23:58:31.099085+0000","last_clean":"2024-05-22T23:58:31.099085+0000","last_became_active":"2024-05-22T23:57:32.311600+0000","last_became_peered":"2024-05-22T23:57:32.311600+0000","last_unstale":"2024-05-22T23:58:31.099085+0000","last_undegraded":"2024-05-22T23:58:31.099085+0000","last_fullsized":"2024-05-22T23:58:31.099085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:35:21.882773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.104314+0000","last_change":"2024-05-22T23:57:33.321289+0000","last_active":"2024-05-22T23:59:30.104314+0000","last_peered":"2024-05-22T23:59:30.104314+0000","last_clean":"2024-05-22T23:59:30.104314+0000","last_became_active":"2024-05-22T23:57:33.320458+0000","last_became_peered":"2024-05-22T23:57:33.320458+0000","last_unstale":"2024-05-22T23:59:30.104314+0000","last_undegraded":"2024-05-22T23:59:30.104314+0000","last_fullsized":"2024-05-22T23:59:30.104314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:02:05.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.099889+0000","last_change":"2024-05-22T23:57:37.786062+0000","last_active":"2024-05-22T23:58:31.099889+0000","last_peered":"2024-05-22T23:58:31.099889+0000","last_clean":"2024-05-22T23:58:31.099889+0000","last_became_active":"2024-05-22T23:57:37.785624+0000","last_became_peered":"2024-05-22T23:57:37.785624+0000","last_unstale":"2024-05-22T23:58:31.099889+0000","last_undegraded":"2024-05-22T23:58:31.099889+0000","last_fullsized":"2024-05-22T23:58:31.099889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:38:56.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102580+0000","last_change":"2024-05-22T23:57:32.333909+0000","last_active":"2024-05-22T23:58:31.102580+0000","last_peered":"2024-05-22T23:58:31.102580+0000","last_clean":"2024-05-22T23:58:31.102580+0000","last_became_active":"2024-05-22T23:57:32.333531+0000","last_became_peered":"2024-05-22T23:57:32.333531+0000","last_unstale":"2024-05-22T23:58:31.102580+0000","last_undegraded":"2024-05-22T23:58:31.102580+0000","last_fullsized":"2024-05-22T23:58:31.102580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:53:21.843150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108530+0000","last_change":"2024-05-22T23:57:33.334366+0000","last_active":"2024-05-22T23:59:30.108530+0000","last_peered":"2024-05-22T23:59:30.108530+0000","last_clean":"2024-05-22T23:59:30.108530+0000","last_became_active":"2024-05-22T23:57:33.334037+0000","last_became_peered":"2024-05-22T23:57:33.334037+0000","last_unstale":"2024-05-22T23:59:30.108530+0000","last_undegraded":"2024-05-22T23:59:30.108530+0000","last_fullsized":"2024-05-22T23:59:30.108530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:15:10.631087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100135+0000","last_change":"2024-05-22T23:57:37.780241+0000","last_active":"2024-05-22T23:58:31.100135+0000","last_peered":"2024-05-22T23:58:31.100135+0000","last_clean":"2024-05-22T23:58:31.100135+0000","last_became_active":"2024-05-22T23:57:37.779816+0000","last_became_peered":"2024-05-22T23:57:37.779816+0000","last_unstale":"2024-05-22T23:58:31.100135+0000","last_undegraded":"2024-05-22T23:58:31.100135+0000","last_fullsized":"2024-05-22T23:58:31.100135+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:11.397681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.106923+0000","last_change":"2024-05-22T23:57:32.334829+0000","last_active":"2024-05-22T23:58:31.106923+0000","last_peered":"2024-05-22T23:58:31.106923+0000","last_clean":"2024-05-22T23:58:31.106923+0000","last_became_active":"2024-05-22T23:57:32.334368+0000","last_became_peered":"2024-05-22T23:57:32.334368+0000","last_unstale":"2024-05-22T23:58:31.106923+0000","last_undegraded":"2024-05-22T23:58:31.106923+0000","last_fullsized":"2024-05-22T23:58:31.106923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:17:42.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.105274+0000","last_change":"2024-05-22T23:57:33.318740+0000","last_active":"2024-05-22T23:59:30.105274+0000","last_peered":"2024-05-22T23:59:30.105274+0000","last_clean":"2024-05-22T23:59:30.105274+0000","last_became_active":"2024-05-22T23:57:33.318363+0000","last_became_peered":"2024-05-22T23:57:33.318363+0000","last_unstale":"2024-05-22T23:59:30.105274+0000","last_undegraded":"2024-05-22T23:59:30.105274+0000","last_fullsized":"2024-05-22T23:59:30.105274+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:51:42.319667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.096705+0000","last_change":"2024-05-22T23:57:33.325582+0000","last_active":"2024-05-22T23:59:30.096705+0000","last_peered":"2024-05-22T23:59:30.096705+0000","last_clean":"2024-05-22T23:59:30.096705+0000","last_became_active":"2024-05-22T23:57:33.325268+0000","last_became_peered":"2024-05-22T23:57:33.325268+0000","last_unstale":"2024-05-22T23:59:30.096705+0000","last_undegraded":"2024-05-22T23:59:30.096705+0000","last_fullsized":"2024-05-22T23:59:30.096705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:02:18.968634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.102477+0000","last_change":"2024-05-22T23:58:31.109626+0000","last_active":"2024-05-22T23:59:30.102477+0000","last_peered":"2024-05-22T23:59:30.102477+0000","last_clean":"2024-05-22T23:59:30.102477+0000","last_became_active":"2024-05-22T23:58:31.109179+0000","last_became_peered":"2024-05-22T23:58:31.109179+0000","last_unstale":"2024-05-22T23:59:30.102477+0000","last_undegraded":"2024-05-22T23:59:30.102477+0000","last_fullsized":"2024-05-22T23:59:30.102477+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:36.653833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.123926+0000","last_change":"2024-05-22T23:57:33.332935+0000","last_active":"2024-05-22T23:59:30.123926+0000","last_peered":"2024-05-22T23:59:30.123926+0000","last_clean":"2024-05-22T23:59:30.123926+0000","last_became_active":"2024-05-22T23:57:33.332607+0000","last_became_peered":"2024-05-22T23:57:33.332607+0000","last_unstale":"2024-05-22T23:59:30.123926+0000","last_undegraded":"2024-05-22T23:59:30.123926+0000","last_fullsized":"2024-05-22T23:59:30.123926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:09.589896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100640+0000","last_change":"2024-05-22T23:57:33.333269+0000","last_active":"2024-05-22T23:58:31.100640+0000","last_peered":"2024-05-22T23:58:31.100640+0000","last_clean":"2024-05-22T23:58:31.100640+0000","last_became_active":"2024-05-22T23:57:33.329017+0000","last_became_peered":"2024-05-22T23:57:33.329017+0000","last_unstale":"2024-05-22T23:58:31.100640+0000","last_undegraded":"2024-05-22T23:58:31.100640+0000","last_fullsized":"2024-05-22T23:58:31.100640+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:35:42.156434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110613+0000","last_change":"2024-05-22T23:57:33.328721+0000","last_active":"2024-05-22T23:58:31.110613+0000","last_peered":"2024-05-22T23:58:31.110613+0000","last_clean":"2024-05-22T23:58:31.110613+0000","last_became_active":"2024-05-22T23:57:33.328318+0000","last_became_peered":"2024-05-22T23:57:33.328318+0000","last_unstale":"2024-05-22T23:58:31.110613+0000","last_undegraded":"2024-05-22T23:58:31.110613+0000","last_fullsized":"2024-05-22T23:58:31.110613+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:14.772688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.092872+0000","last_change":"2024-05-22T23:57:33.324777+0000","last_active":"2024-05-22T23:59:30.092872+0000","last_peered":"2024-05-22T23:59:30.092872+0000","last_clean":"2024-05-22T23:59:30.092872+0000","last_became_active":"2024-05-22T23:57:33.324446+0000","last_became_peered":"2024-05-22T23:57:33.324446+0000","last_unstale":"2024-05-22T23:59:30.092872+0000","last_undegraded":"2024-05-22T23:59:30.092872+0000","last_fullsized":"2024-05-22T23:59:30.092872+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:20:42.327094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.099266+0000","last_change":"2024-05-22T23:57:33.318783+0000","last_active":"2024-05-22T23:59:30.099266+0000","last_peered":"2024-05-22T23:59:30.099266+0000","last_clean":"2024-05-22T23:59:30.099266+0000","last_became_active":"2024-05-22T23:57:33.318481+0000","last_became_peered":"2024-05-22T23:57:33.318481+0000","last_unstale":"2024-05-22T23:59:30.099266+0000","last_undegraded":"2024-05-22T23:59:30.099266+0000","last_fullsized":"2024-05-22T23:59:30.099266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:14:35.745683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.106040+0000","last_change":"2024-05-22T23:57:33.319939+0000","last_active":"2024-05-22T23:59:30.106040+0000","last_peered":"2024-05-22T23:59:30.106040+0000","last_clean":"2024-05-22T23:59:30.106040+0000","last_became_active":"2024-05-22T23:57:33.316029+0000","last_became_peered":"2024-05-22T23:57:33.316029+0000","last_unstale":"2024-05-22T23:59:30.106040+0000","last_undegraded":"2024-05-22T23:59:30.106040+0000","last_fullsized":"2024-05-22T23:59:30.106040+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:46:34.383953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100370+0000","last_change":"2024-05-22T23:57:33.334625+0000","last_active":"2024-05-22T23:58:31.100370+0000","last_peered":"2024-05-22T23:58:31.100370+0000","last_clean":"2024-05-22T23:58:31.100370+0000","last_became_active":"2024-05-22T23:57:33.334341+0000","last_became_peered":"2024-05-22T23:57:33.334341+0000","last_unstale":"2024-05-22T23:58:31.100370+0000","last_undegraded":"2024-05-22T23:58:31.100370+0000","last_fullsized":"2024-05-22T23:58:31.100370+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:38.319721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.107679+0000","last_change":"2024-05-22T23:57:33.330958+0000","last_active":"2024-05-22T23:59:30.107679+0000","last_peered":"2024-05-22T23:59:30.107679+0000","last_clean":"2024-05-22T23:59:30.107679+0000","last_became_active":"2024-05-22T23:57:33.317355+0000","last_became_peered":"2024-05-22T23:57:33.317355+0000","last_unstale":"2024-05-22T23:59:30.107679+0000","last_undegraded":"2024-05-22T23:59:30.107679+0000","last_fullsized":"2024-05-22T23:59:30.107679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:49:43.784780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108012+0000","last_change":"2024-05-22T23:57:33.319530+0000","last_active":"2024-05-22T23:59:30.108012+0000","last_peered":"2024-05-22T23:59:30.108012+0000","last_clean":"2024-05-22T23:59:30.108012+0000","last_became_active":"2024-05-22T23:57:33.319122+0000","last_became_peered":"2024-05-22T23:57:33.319122+0000","last_unstale":"2024-05-22T23:59:30.108012+0000","last_undegraded":"2024-05-22T23:59:30.108012+0000","last_fullsized":"2024-05-22T23:59:30.108012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:31:26.937804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.113151+0000","last_change":"2024-05-22T23:57:33.323722+0000","last_active":"2024-05-22T23:58:31.113151+0000","last_peered":"2024-05-22T23:58:31.113151+0000","last_clean":"2024-05-22T23:58:31.113151+0000","last_became_active":"2024-05-22T23:57:33.323343+0000","last_became_peered":"2024-05-22T23:57:33.323343+0000","last_unstale":"2024-05-22T23:58:31.113151+0000","last_undegraded":"2024-05-22T23:58:31.113151+0000","last_fullsized":"2024-05-22T23:58:31.113151+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:24:59.575536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.090145+0000","last_change":"2024-05-22T23:57:33.320925+0000","last_active":"2024-05-22T23:59:30.090145+0000","last_peered":"2024-05-22T23:59:30.090145+0000","last_clean":"2024-05-22T23:59:30.090145+0000","last_became_active":"2024-05-22T23:57:33.320570+0000","last_became_peered":"2024-05-22T23:57:33.320570+0000","last_unstale":"2024-05-22T23:59:30.090145+0000","last_undegraded":"2024-05-22T23:59:30.090145+0000","last_fullsized":"2024-05-22T23:59:30.090145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:38:27.659025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":288,"ondisk_log_size":288,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244946,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66721,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":525,"num_read_kb":332,"num_write":166,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1012,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":200943,"data_compressed":5079,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318310,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":508368,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579947,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1159168,"data_stored":1158357,"data_compressed":57992,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743473,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":512700,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005110,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":508524,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266750,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1105104,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430273,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":503780,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":566270,"data_compressed":22296,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":505168,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":566749,"data_compressed":22296,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":503974,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888142,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":509832,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691914,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":1100020,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":62580,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63308,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64299,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-22T23:59:33.623 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph pg dump --format=json 2024-05-22T23:59:34.065 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:34.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:34 smithi103 ceph-mon[31370]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:59:34.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:34 smithi174 ceph-mon[36179]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:59:34.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:34 smithi105 ceph-mon[36426]: from='client.25043 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:59:35.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:35 smithi174 ceph-mon[36179]: pgmap v385: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-22T23:59:35.471 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:35 smithi103 ceph-mon[31370]: pgmap v385: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-22T23:59:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:35 smithi105 ceph-mon[36426]: pgmap v385: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-22T23:59:36.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.346+0000 7f7b01800700 1 -- 172.21.15.103:0/4013062278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 msgr2=0x7f7afc074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.346+0000 7f7b01800700 1 --2- 172.21.15.103:0/4013062278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 0x7f7afc074970 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009f90 tx=0x7f7aec00b330 comp rx=0 tx=0).stop 2024-05-22T23:59:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.347+0000 7f7b01800700 1 -- 172.21.15.103:0/4013062278 shutdown_connections 2024-05-22T23:59:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.347+0000 7f7b01800700 1 --2- 172.21.15.103:0/4013062278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 0x7f7afc074970 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.347+0000 7f7b01800700 1 --2- 172.21.15.103:0/4013062278 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7afc113530 0x7f7afc115920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.347+0000 7f7b01800700 1 --2- 172.21.15.103:0/4013062278 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7afc074eb0 0x7f7afc112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.347+0000 7f7b01800700 1 -- 172.21.15.103:0/4013062278 >> 172.21.15.103:0/4013062278 conn(0x7f7afc0794c0 msgr2=0x7f7afc0798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.347+0000 7f7b01800700 1 -- 172.21.15.103:0/4013062278 shutdown_connections 2024-05-22T23:59:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.347+0000 7f7b01800700 1 -- 172.21.15.103:0/4013062278 wait complete. 2024-05-22T23:59:36.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.348+0000 7f7b01800700 1 Processor -- start 2024-05-22T23:59:36.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.348+0000 7f7b01800700 1 -- start start 2024-05-22T23:59:36.352 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.348+0000 7f7b01800700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 0x7f7afc19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.348+0000 7f7b01800700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7afc074eb0 0x7f7afc19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.348+0000 7f7b01800700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7afc113530 0x7f7afc1a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.348+0000 7f7b01800700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc118c90 con 0x7f7afc074eb0 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.348+0000 7f7b01800700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc118b10 con 0x7f7afc074590 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.348+0000 7f7b01800700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc118e10 con 0x7f7afc113530 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7af2000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7afc074eb0 0x7f7afc19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7af2000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7afc074eb0 0x7f7afc19f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48620/0 (socket says 172.21.15.103:48620) 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7af2000700 1 -- 172.21.15.103:0/623002273 learned_addr learned my addr 172.21.15.103:0/623002273 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7afaa00700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 0x7f7afc19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7afb400700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7afc113530 0x7f7afc1a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7af2000700 1 -- 172.21.15.103:0/623002273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7afc113530 msgr2=0x7f7afc1a39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7af2000700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7afc113530 0x7f7afc1a39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7af2000700 1 -- 172.21.15.103:0/623002273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 msgr2=0x7f7afc19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7af2000700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 0x7f7afc19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7af2000700 1 -- 172.21.15.103:0/623002273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7afc1a4130 con 0x7f7afc074eb0 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.349+0000 7f7afaa00700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 0x7f7afc19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.350+0000 7f7af2000700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7afc074eb0 0x7f7afc19f660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400e9c0 tx=0x7f7ae400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.350+0000 7f7af3e00700 1 -- 172.21.15.103:0/623002273 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400cbd0 con 0x7f7afc074eb0 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.350+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7afc1a4420 con 0x7f7afc074eb0 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.350+0000 7f7af3e00700 1 -- 172.21.15.103:0/623002273 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ae400ee50 con 0x7f7afc074eb0 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.350+0000 7f7af3e00700 1 -- 172.21.15.103:0/623002273 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4005680 con 0x7f7afc074eb0 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.350+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7afc1abca0 con 0x7f7afc074eb0 2024-05-22T23:59:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.351+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7afc069050 con 0x7f7afc074eb0 2024-05-22T23:59:36.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.353+0000 7f7af3e00700 1 -- 172.21.15.103:0/623002273 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f7ae4004750 con 0x7f7afc074eb0 2024-05-22T23:59:36.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.356+0000 7f7af3e00700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ae00a2ca0 0x7f7ae00a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:36.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.356+0000 7f7af3e00700 1 -- 172.21.15.103:0/623002273 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f7ae40cf7e0 con 0x7f7afc074eb0 2024-05-22T23:59:36.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.356+0000 7f7afaa00700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ae00a2ca0 0x7f7ae00a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:36.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.357+0000 7f7afaa00700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ae00a2ca0 0x7f7ae00a5160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009790 tx=0x7f7aec009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:36.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.359+0000 7f7af3e00700 1 -- 172.21.15.103:0/623002273 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7ae4053850 con 0x7f7afc074eb0 2024-05-22T23:59:36.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:36 smithi105 ceph-mon[36426]: pgmap v386: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-22T23:59:36.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:36 smithi103 ceph-mon[31370]: pgmap v386: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-22T23:59:36.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.579+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7afc112a70 con 0x7f7ae00a2ca0 2024-05-22T23:59:36.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.590+0000 7f7af3e00700 1 -- 172.21.15.103:0/623002273 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340898 (secure 0 0 0) 0x7f7afc112a70 con 0x7f7ae00a2ca0 2024-05-22T23:59:36.600 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ae00a2ca0 msgr2=0x7f7ae00a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ae00a2ca0 0x7f7ae00a5160 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009790 tx=0x7f7aec009720 comp rx=0 tx=0).stop 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7afc074eb0 msgr2=0x7f7afc19f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7afc074eb0 0x7f7afc19f660 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7ae400e9c0 tx=0x7f7ae400ecd0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 shutdown_connections 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7afc074590 0x7f7afc19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7ae00a2ca0 0x7f7ae00a5160 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7afc113530 0x7f7afc1a39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 --2- 172.21.15.103:0/623002273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7afc074eb0 0x7f7afc19f660 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:36.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 >> 172.21.15.103:0/623002273 conn(0x7f7afc0794c0 msgr2=0x7f7afc112360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:36.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 shutdown_connections 2024-05-22T23:59:36.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:36.597+0000 7f7b01800700 1 -- 172.21.15.103:0/623002273 wait complete. 2024-05-22T23:59:36.604 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-05-22T23:59:36.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:36 smithi174 ceph-mon[36179]: pgmap v386: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-22T23:59:37.424 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-05-22T23:59:36.158558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657854,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":694,"num_read_kb":469,"num_write":356,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1181748,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357384,"kb_used_data":12392,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584440,"statfs":{"total":1151940427776,"available":1151574466560,"internally_reserved":0,"allocated":12689408,"data_stored":8049908,"data_compressed":359403,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":33,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005466"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.119831+0000","last_change":"2024-05-22T23:57:33.328075+0000","last_active":"2024-05-22T23:59:30.119831+0000","last_peered":"2024-05-22T23:59:30.119831+0000","last_clean":"2024-05-22T23:59:30.119831+0000","last_became_active":"2024-05-22T23:57:33.327704+0000","last_became_peered":"2024-05-22T23:57:33.327704+0000","last_unstale":"2024-05-22T23:59:30.119831+0000","last_undegraded":"2024-05-22T23:59:30.119831+0000","last_fullsized":"2024-05-22T23:59:30.119831+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:42:32.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.125165+0000","last_change":"2024-05-22T23:57:33.332043+0000","last_active":"2024-05-22T23:59:30.125165+0000","last_peered":"2024-05-22T23:59:30.125165+0000","last_clean":"2024-05-22T23:59:30.125165+0000","last_became_active":"2024-05-22T23:57:33.322269+0000","last_became_peered":"2024-05-22T23:57:33.322269+0000","last_unstale":"2024-05-22T23:59:30.125165+0000","last_undegraded":"2024-05-22T23:59:30.125165+0000","last_fullsized":"2024-05-22T23:59:30.125165+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:40:23.542938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.124670+0000","last_change":"2024-05-22T23:57:33.334993+0000","last_active":"2024-05-22T23:59:30.124670+0000","last_peered":"2024-05-22T23:59:30.124670+0000","last_clean":"2024-05-22T23:59:30.124670+0000","last_became_active":"2024-05-22T23:57:33.326446+0000","last_became_peered":"2024-05-22T23:57:33.326446+0000","last_unstale":"2024-05-22T23:59:30.124670+0000","last_undegraded":"2024-05-22T23:59:30.124670+0000","last_fullsized":"2024-05-22T23:59:30.124670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:44:45.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.099804+0000","last_change":"2024-05-22T23:57:33.320181+0000","last_active":"2024-05-22T23:59:30.099804+0000","last_peered":"2024-05-22T23:59:30.099804+0000","last_clean":"2024-05-22T23:59:30.099804+0000","last_became_active":"2024-05-22T23:57:33.319630+0000","last_became_peered":"2024-05-22T23:57:33.319630+0000","last_unstale":"2024-05-22T23:59:30.099804+0000","last_undegraded":"2024-05-22T23:59:30.099804+0000","last_fullsized":"2024-05-22T23:59:30.099804+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:38:51.836241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.109469+0000","last_change":"2024-05-22T23:57:33.332168+0000","last_active":"2024-05-22T23:59:30.109469+0000","last_peered":"2024-05-22T23:59:30.109469+0000","last_clean":"2024-05-22T23:59:30.109469+0000","last_became_active":"2024-05-22T23:57:33.331755+0000","last_became_peered":"2024-05-22T23:57:33.331755+0000","last_unstale":"2024-05-22T23:59:30.109469+0000","last_undegraded":"2024-05-22T23:59:30.109469+0000","last_fullsized":"2024-05-22T23:59:30.109469+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:28.013365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.097818+0000","last_change":"2024-05-22T23:57:33.322449+0000","last_active":"2024-05-22T23:59:30.097818+0000","last_peered":"2024-05-22T23:59:30.097818+0000","last_clean":"2024-05-22T23:59:30.097818+0000","last_became_active":"2024-05-22T23:57:33.322099+0000","last_became_peered":"2024-05-22T23:57:33.322099+0000","last_unstale":"2024-05-22T23:59:30.097818+0000","last_undegraded":"2024-05-22T23:59:30.097818+0000","last_fullsized":"2024-05-22T23:59:30.097818+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:38.427434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108181+0000","last_change":"2024-05-22T23:57:33.322391+0000","last_active":"2024-05-22T23:59:30.108181+0000","last_peered":"2024-05-22T23:59:30.108181+0000","last_clean":"2024-05-22T23:59:30.108181+0000","last_became_active":"2024-05-22T23:57:33.322076+0000","last_became_peered":"2024-05-22T23:57:33.322076+0000","last_unstale":"2024-05-22T23:59:30.108181+0000","last_undegraded":"2024-05-22T23:59:30.108181+0000","last_fullsized":"2024-05-22T23:59:30.108181+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:45:44.857586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.115118+0000","last_change":"2024-05-22T23:57:33.333005+0000","last_active":"2024-05-22T23:59:30.115118+0000","last_peered":"2024-05-22T23:59:30.115118+0000","last_clean":"2024-05-22T23:59:30.115118+0000","last_became_active":"2024-05-22T23:57:33.332669+0000","last_became_peered":"2024-05-22T23:57:33.332669+0000","last_unstale":"2024-05-22T23:59:30.115118+0000","last_undegraded":"2024-05-22T23:59:30.115118+0000","last_fullsized":"2024-05-22T23:59:30.115118+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:47:25.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.105987+0000","last_change":"2024-05-22T23:58:31.104194+0000","last_active":"2024-05-22T23:59:30.105987+0000","last_peered":"2024-05-22T23:59:30.105987+0000","last_clean":"2024-05-22T23:59:30.105987+0000","last_became_active":"2024-05-22T23:58:31.103788+0000","last_became_peered":"2024-05-22T23:58:31.103788+0000","last_unstale":"2024-05-22T23:59:30.105987+0000","last_undegraded":"2024-05-22T23:59:30.105987+0000","last_fullsized":"2024-05-22T23:59:30.105987+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:41:09.215669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.091881+0000","last_change":"2024-05-22T23:57:33.324445+0000","last_active":"2024-05-22T23:59:30.091881+0000","last_peered":"2024-05-22T23:59:30.091881+0000","last_clean":"2024-05-22T23:59:30.091881+0000","last_became_active":"2024-05-22T23:57:33.324085+0000","last_became_peered":"2024-05-22T23:57:33.324085+0000","last_unstale":"2024-05-22T23:59:30.091881+0000","last_undegraded":"2024-05-22T23:59:30.091881+0000","last_fullsized":"2024-05-22T23:59:30.091881+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:30:51.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.104148+0000","last_change":"2024-05-22T23:57:33.321481+0000","last_active":"2024-05-22T23:59:30.104148+0000","last_peered":"2024-05-22T23:59:30.104148+0000","last_clean":"2024-05-22T23:59:30.104148+0000","last_became_active":"2024-05-22T23:57:33.320911+0000","last_became_peered":"2024-05-22T23:57:33.320911+0000","last_unstale":"2024-05-22T23:59:30.104148+0000","last_undegraded":"2024-05-22T23:59:30.104148+0000","last_fullsized":"2024-05-22T23:59:30.104148+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:24:24.041980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.087645+0000","last_change":"2024-05-22T23:57:33.318819+0000","last_active":"2024-05-22T23:59:30.087645+0000","last_peered":"2024-05-22T23:59:30.087645+0000","last_clean":"2024-05-22T23:59:30.087645+0000","last_became_active":"2024-05-22T23:57:33.318284+0000","last_became_peered":"2024-05-22T23:57:33.318284+0000","last_unstale":"2024-05-22T23:59:30.087645+0000","last_undegraded":"2024-05-22T23:59:30.087645+0000","last_fullsized":"2024-05-22T23:59:30.087645+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:56.043446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.111936+0000","last_change":"2024-05-22T23:57:33.321268+0000","last_active":"2024-05-22T23:59:30.111936+0000","last_peered":"2024-05-22T23:59:30.111936+0000","last_clean":"2024-05-22T23:59:30.111936+0000","last_became_active":"2024-05-22T23:57:33.320902+0000","last_became_peered":"2024-05-22T23:57:33.320902+0000","last_unstale":"2024-05-22T23:59:30.111936+0000","last_undegraded":"2024-05-22T23:59:30.111936+0000","last_fullsized":"2024-05-22T23:59:30.111936+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:51:39.716541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.123962+0000","last_change":"2024-05-22T23:57:33.322638+0000","last_active":"2024-05-22T23:59:30.123962+0000","last_peered":"2024-05-22T23:59:30.123962+0000","last_clean":"2024-05-22T23:59:30.123962+0000","last_became_active":"2024-05-22T23:57:33.322318+0000","last_became_peered":"2024-05-22T23:57:33.322318+0000","last_unstale":"2024-05-22T23:59:30.123962+0000","last_undegraded":"2024-05-22T23:59:30.123962+0000","last_fullsized":"2024-05-22T23:59:30.123962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:44:25.487618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.121725+0000","last_change":"2024-05-22T23:57:33.325832+0000","last_active":"2024-05-22T23:59:30.121725+0000","last_peered":"2024-05-22T23:59:30.121725+0000","last_clean":"2024-05-22T23:59:30.121725+0000","last_became_active":"2024-05-22T23:57:33.325062+0000","last_became_peered":"2024-05-22T23:57:33.325062+0000","last_unstale":"2024-05-22T23:59:30.121725+0000","last_undegraded":"2024-05-22T23:59:30.121725+0000","last_fullsized":"2024-05-22T23:59:30.121725+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:33:04.161351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.119980+0000","last_change":"2024-05-22T23:57:33.324864+0000","last_active":"2024-05-22T23:59:30.119980+0000","last_peered":"2024-05-22T23:59:30.119980+0000","last_clean":"2024-05-22T23:59:30.119980+0000","last_became_active":"2024-05-22T23:57:33.324558+0000","last_became_peered":"2024-05-22T23:57:33.324558+0000","last_unstale":"2024-05-22T23:59:30.119980+0000","last_undegraded":"2024-05-22T23:59:30.119980+0000","last_fullsized":"2024-05-22T23:59:30.119980+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:48:49.722371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.119946+0000","last_change":"2024-05-22T23:57:33.337985+0000","last_active":"2024-05-22T23:59:30.119946+0000","last_peered":"2024-05-22T23:59:30.119946+0000","last_clean":"2024-05-22T23:59:30.119946+0000","last_became_active":"2024-05-22T23:57:33.337719+0000","last_became_peered":"2024-05-22T23:57:33.337719+0000","last_unstale":"2024-05-22T23:59:30.119946+0000","last_undegraded":"2024-05-22T23:59:30.119946+0000","last_fullsized":"2024-05-22T23:59:30.119946+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:50:44.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.103774+0000","last_change":"2024-05-22T23:57:33.325709+0000","last_active":"2024-05-22T23:59:30.103774+0000","last_peered":"2024-05-22T23:59:30.103774+0000","last_clean":"2024-05-22T23:59:30.103774+0000","last_became_active":"2024-05-22T23:57:33.325063+0000","last_became_peered":"2024-05-22T23:57:33.325063+0000","last_unstale":"2024-05-22T23:59:30.103774+0000","last_undegraded":"2024-05-22T23:59:30.103774+0000","last_fullsized":"2024-05-22T23:59:30.103774+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:39.145217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108004+0000","last_change":"2024-05-22T23:57:33.333251+0000","last_active":"2024-05-22T23:59:30.108004+0000","last_peered":"2024-05-22T23:59:30.108004+0000","last_clean":"2024-05-22T23:59:30.108004+0000","last_became_active":"2024-05-22T23:57:33.332894+0000","last_became_peered":"2024-05-22T23:57:33.332894+0000","last_unstale":"2024-05-22T23:59:30.108004+0000","last_undegraded":"2024-05-22T23:59:30.108004+0000","last_fullsized":"2024-05-22T23:59:30.108004+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:13:12.082781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098311+0000","last_change":"2024-05-22T23:57:37.788060+0000","last_active":"2024-05-22T23:58:31.098311+0000","last_peered":"2024-05-22T23:58:31.098311+0000","last_clean":"2024-05-22T23:58:31.098311+0000","last_became_active":"2024-05-22T23:57:37.787656+0000","last_became_peered":"2024-05-22T23:57:37.787656+0000","last_unstale":"2024-05-22T23:58:31.098311+0000","last_undegraded":"2024-05-22T23:58:31.098311+0000","last_fullsized":"2024-05-22T23:58:31.098311+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:39:13.718817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.109737+0000","last_change":"2024-05-22T23:57:32.323434+0000","last_active":"2024-05-22T23:58:31.109737+0000","last_peered":"2024-05-22T23:58:31.109737+0000","last_clean":"2024-05-22T23:58:31.109737+0000","last_became_active":"2024-05-22T23:57:32.323033+0000","last_became_peered":"2024-05-22T23:57:32.323033+0000","last_unstale":"2024-05-22T23:58:31.109737+0000","last_undegraded":"2024-05-22T23:58:31.109737+0000","last_fullsized":"2024-05-22T23:58:31.109737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:15:27.676027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.110459+0000","last_change":"2024-05-22T23:57:33.323336+0000","last_active":"2024-05-22T23:59:30.110459+0000","last_peered":"2024-05-22T23:59:30.110459+0000","last_clean":"2024-05-22T23:59:30.110459+0000","last_became_active":"2024-05-22T23:57:33.323026+0000","last_became_peered":"2024-05-22T23:57:33.323026+0000","last_unstale":"2024-05-22T23:59:30.110459+0000","last_undegraded":"2024-05-22T23:59:30.110459+0000","last_fullsized":"2024-05-22T23:59:30.110459+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:02:20.142760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096471+0000","last_change":"2024-05-22T23:57:37.785137+0000","last_active":"2024-05-22T23:58:31.096471+0000","last_peered":"2024-05-22T23:58:31.096471+0000","last_clean":"2024-05-22T23:58:31.096471+0000","last_became_active":"2024-05-22T23:57:37.784791+0000","last_became_peered":"2024-05-22T23:57:37.784791+0000","last_unstale":"2024-05-22T23:58:31.096471+0000","last_undegraded":"2024-05-22T23:58:31.096471+0000","last_fullsized":"2024-05-22T23:58:31.096471+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:54:51.036334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.099658+0000","last_change":"2024-05-22T23:57:32.332969+0000","last_active":"2024-05-22T23:58:31.099658+0000","last_peered":"2024-05-22T23:58:31.099658+0000","last_clean":"2024-05-22T23:58:31.099658+0000","last_became_active":"2024-05-22T23:57:32.332611+0000","last_became_peered":"2024-05-22T23:57:32.332611+0000","last_unstale":"2024-05-22T23:58:31.099658+0000","last_undegraded":"2024-05-22T23:58:31.099658+0000","last_fullsized":"2024-05-22T23:58:31.099658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:00:22.006355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.124206+0000","last_change":"2024-05-22T23:57:33.332151+0000","last_active":"2024-05-22T23:59:30.124206+0000","last_peered":"2024-05-22T23:59:30.124206+0000","last_clean":"2024-05-22T23:59:30.124206+0000","last_became_active":"2024-05-22T23:57:33.329518+0000","last_became_peered":"2024-05-22T23:57:33.329518+0000","last_unstale":"2024-05-22T23:59:30.124206+0000","last_undegraded":"2024-05-22T23:59:30.124206+0000","last_fullsized":"2024-05-22T23:59:30.124206+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:10:12.083766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096570+0000","last_change":"2024-05-22T23:57:37.787524+0000","last_active":"2024-05-22T23:58:31.096570+0000","last_peered":"2024-05-22T23:58:31.096570+0000","last_clean":"2024-05-22T23:58:31.096570+0000","last_became_active":"2024-05-22T23:57:37.787103+0000","last_became_peered":"2024-05-22T23:57:37.787103+0000","last_unstale":"2024-05-22T23:58:31.096570+0000","last_undegraded":"2024-05-22T23:58:31.096570+0000","last_fullsized":"2024-05-22T23:58:31.096570+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:19:16.963159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108461+0000","last_change":"2024-05-22T23:57:32.318469+0000","last_active":"2024-05-22T23:58:31.108461+0000","last_peered":"2024-05-22T23:58:31.108461+0000","last_clean":"2024-05-22T23:58:31.108461+0000","last_became_active":"2024-05-22T23:57:32.317640+0000","last_became_peered":"2024-05-22T23:57:32.317640+0000","last_unstale":"2024-05-22T23:58:31.108461+0000","last_undegraded":"2024-05-22T23:58:31.108461+0000","last_fullsized":"2024-05-22T23:58:31.108461+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:01:55.710019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.109225+0000","last_change":"2024-05-22T23:57:33.318604+0000","last_active":"2024-05-22T23:59:30.109225+0000","last_peered":"2024-05-22T23:59:30.109225+0000","last_clean":"2024-05-22T23:59:30.109225+0000","last_became_active":"2024-05-22T23:57:33.318184+0000","last_became_peered":"2024-05-22T23:57:33.318184+0000","last_unstale":"2024-05-22T23:59:30.109225+0000","last_undegraded":"2024-05-22T23:59:30.109225+0000","last_fullsized":"2024-05-22T23:59:30.109225+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:16.140799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096467+0000","last_change":"2024-05-22T23:57:37.796085+0000","last_active":"2024-05-22T23:58:31.096467+0000","last_peered":"2024-05-22T23:58:31.096467+0000","last_clean":"2024-05-22T23:58:31.096467+0000","last_became_active":"2024-05-22T23:57:37.795265+0000","last_became_peered":"2024-05-22T23:57:37.795265+0000","last_unstale":"2024-05-22T23:58:31.096467+0000","last_undegraded":"2024-05-22T23:58:31.096467+0000","last_fullsized":"2024-05-22T23:58:31.096467+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:43:55.577072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108195+0000","last_change":"2024-05-22T23:57:32.316346+0000","last_active":"2024-05-22T23:58:31.108195+0000","last_peered":"2024-05-22T23:58:31.108195+0000","last_clean":"2024-05-22T23:58:31.108195+0000","last_became_active":"2024-05-22T23:57:32.316038+0000","last_became_peered":"2024-05-22T23:57:32.316038+0000","last_unstale":"2024-05-22T23:58:31.108195+0000","last_undegraded":"2024-05-22T23:58:31.108195+0000","last_fullsized":"2024-05-22T23:58:31.108195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:57:41.987250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108290+0000","last_change":"2024-05-22T23:57:33.329341+0000","last_active":"2024-05-22T23:59:30.108290+0000","last_peered":"2024-05-22T23:59:30.108290+0000","last_clean":"2024-05-22T23:59:30.108290+0000","last_became_active":"2024-05-22T23:57:33.329009+0000","last_became_peered":"2024-05-22T23:57:33.329009+0000","last_unstale":"2024-05-22T23:59:30.108290+0000","last_undegraded":"2024-05-22T23:59:30.108290+0000","last_fullsized":"2024-05-22T23:59:30.108290+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:53:07.591497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098490+0000","last_change":"2024-05-22T23:57:37.779097+0000","last_active":"2024-05-22T23:58:31.098490+0000","last_peered":"2024-05-22T23:58:31.098490+0000","last_clean":"2024-05-22T23:58:31.098490+0000","last_became_active":"2024-05-22T23:57:37.778564+0000","last_became_peered":"2024-05-22T23:57:37.778564+0000","last_unstale":"2024-05-22T23:58:31.098490+0000","last_undegraded":"2024-05-22T23:58:31.098490+0000","last_fullsized":"2024-05-22T23:58:31.098490+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:09:16.351947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'7","reported_seq":139,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:33.202083+0000","last_change":"2024-05-22T23:58:31.109013+0000","last_active":"2024-05-22T23:59:33.202083+0000","last_peered":"2024-05-22T23:59:33.202083+0000","last_clean":"2024-05-22T23:59:33.202083+0000","last_became_active":"2024-05-22T23:58:31.103858+0000","last_became_peered":"2024-05-22T23:58:31.103858+0000","last_unstale":"2024-05-22T23:59:33.202083+0000","last_undegraded":"2024-05-22T23:59:33.202083+0000","last_fullsized":"2024-05-22T23:59:33.202083+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:20:08.783189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1153,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.105341+0000","last_change":"2024-05-22T23:57:33.324587+0000","last_active":"2024-05-22T23:59:30.105341+0000","last_peered":"2024-05-22T23:59:30.105341+0000","last_clean":"2024-05-22T23:59:30.105341+0000","last_became_active":"2024-05-22T23:57:33.323972+0000","last_became_peered":"2024-05-22T23:57:33.323972+0000","last_unstale":"2024-05-22T23:59:30.105341+0000","last_undegraded":"2024-05-22T23:59:30.105341+0000","last_fullsized":"2024-05-22T23:59:30.105341+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:23:52.551699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098921+0000","last_change":"2024-05-22T23:57:37.791726+0000","last_active":"2024-05-22T23:58:31.098921+0000","last_peered":"2024-05-22T23:58:31.098921+0000","last_clean":"2024-05-22T23:58:31.098921+0000","last_became_active":"2024-05-22T23:57:37.791185+0000","last_became_peered":"2024-05-22T23:57:37.791185+0000","last_unstale":"2024-05-22T23:58:31.098921+0000","last_undegraded":"2024-05-22T23:58:31.098921+0000","last_fullsized":"2024-05-22T23:58:31.098921+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:55:30.994272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.116594+0000","last_change":"2024-05-22T23:58:31.116594+0000","last_active":"2024-05-22T23:58:31.116594+0000","last_peered":"2024-05-22T23:58:31.116594+0000","last_clean":"2024-05-22T23:58:31.116594+0000","last_became_active":"2024-05-22T23:58:31.113117+0000","last_became_peered":"2024-05-22T23:58:31.113117+0000","last_unstale":"2024-05-22T23:58:31.116594+0000","last_undegraded":"2024-05-22T23:58:31.116594+0000","last_fullsized":"2024-05-22T23:58:31.116594+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:05:30.758638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.117632+0000","last_change":"2024-05-22T23:58:31.116203+0000","last_active":"2024-05-22T23:59:30.117632+0000","last_peered":"2024-05-22T23:59:30.117632+0000","last_clean":"2024-05-22T23:59:30.117632+0000","last_became_active":"2024-05-22T23:58:31.115881+0000","last_became_peered":"2024-05-22T23:58:31.115881+0000","last_unstale":"2024-05-22T23:59:30.117632+0000","last_undegraded":"2024-05-22T23:59:30.117632+0000","last_fullsized":"2024-05-22T23:59:30.117632+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:39:31.247426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101119+0000","last_change":"2024-05-22T23:57:37.787910+0000","last_active":"2024-05-22T23:58:31.101119+0000","last_peered":"2024-05-22T23:58:31.101119+0000","last_clean":"2024-05-22T23:58:31.101119+0000","last_became_active":"2024-05-22T23:57:37.787465+0000","last_became_peered":"2024-05-22T23:57:37.787465+0000","last_unstale":"2024-05-22T23:58:31.101119+0000","last_undegraded":"2024-05-22T23:58:31.101119+0000","last_fullsized":"2024-05-22T23:58:31.101119+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:54:59.202855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107375+0000","last_change":"2024-05-22T23:57:32.329365+0000","last_active":"2024-05-22T23:58:31.107375+0000","last_peered":"2024-05-22T23:58:31.107375+0000","last_clean":"2024-05-22T23:58:31.107375+0000","last_became_active":"2024-05-22T23:57:32.327864+0000","last_became_peered":"2024-05-22T23:57:32.327864+0000","last_unstale":"2024-05-22T23:58:31.107375+0000","last_undegraded":"2024-05-22T23:58:31.107375+0000","last_fullsized":"2024-05-22T23:58:31.107375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:26.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.113454+0000","last_change":"2024-05-22T23:57:33.313930+0000","last_active":"2024-05-22T23:59:30.113454+0000","last_peered":"2024-05-22T23:59:30.113454+0000","last_clean":"2024-05-22T23:59:30.113454+0000","last_became_active":"2024-05-22T23:57:33.313612+0000","last_became_peered":"2024-05-22T23:57:33.313612+0000","last_unstale":"2024-05-22T23:59:30.113454+0000","last_undegraded":"2024-05-22T23:59:30.113454+0000","last_fullsized":"2024-05-22T23:59:30.113454+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:45:25.800338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100765+0000","last_change":"2024-05-22T23:57:37.791537+0000","last_active":"2024-05-22T23:58:31.100765+0000","last_peered":"2024-05-22T23:58:31.100765+0000","last_clean":"2024-05-22T23:58:31.100765+0000","last_became_active":"2024-05-22T23:57:37.791052+0000","last_became_peered":"2024-05-22T23:57:37.791052+0000","last_unstale":"2024-05-22T23:58:31.100765+0000","last_undegraded":"2024-05-22T23:58:31.100765+0000","last_fullsized":"2024-05-22T23:58:31.100765+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:18:11.759840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100298+0000","last_change":"2024-05-22T23:57:32.315907+0000","last_active":"2024-05-22T23:58:31.100298+0000","last_peered":"2024-05-22T23:58:31.100298+0000","last_clean":"2024-05-22T23:58:31.100298+0000","last_became_active":"2024-05-22T23:57:32.315576+0000","last_became_peered":"2024-05-22T23:57:32.315576+0000","last_unstale":"2024-05-22T23:58:31.100298+0000","last_undegraded":"2024-05-22T23:58:31.100298+0000","last_fullsized":"2024-05-22T23:58:31.100298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:45:48.998377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.110282+0000","last_change":"2024-05-22T23:57:33.338681+0000","last_active":"2024-05-22T23:59:30.110282+0000","last_peered":"2024-05-22T23:59:30.110282+0000","last_clean":"2024-05-22T23:59:30.110282+0000","last_became_active":"2024-05-22T23:57:33.334705+0000","last_became_peered":"2024-05-22T23:57:33.334705+0000","last_unstale":"2024-05-22T23:59:30.110282+0000","last_undegraded":"2024-05-22T23:59:30.110282+0000","last_fullsized":"2024-05-22T23:59:30.110282+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:22.135137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100227+0000","last_change":"2024-05-22T23:57:37.780470+0000","last_active":"2024-05-22T23:58:31.100227+0000","last_peered":"2024-05-22T23:58:31.100227+0000","last_clean":"2024-05-22T23:58:31.100227+0000","last_became_active":"2024-05-22T23:57:37.779110+0000","last_became_peered":"2024-05-22T23:57:37.779110+0000","last_unstale":"2024-05-22T23:58:31.100227+0000","last_undegraded":"2024-05-22T23:58:31.100227+0000","last_fullsized":"2024-05-22T23:58:31.100227+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:55:45.486862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'4","reported_seq":118,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:31.197448+0000","last_change":"2024-05-22T23:57:32.322489+0000","last_active":"2024-05-22T23:59:31.197448+0000","last_peered":"2024-05-22T23:59:31.197448+0000","last_clean":"2024-05-22T23:59:31.197448+0000","last_became_active":"2024-05-22T23:57:32.322125+0000","last_became_peered":"2024-05-22T23:57:32.322125+0000","last_unstale":"2024-05-22T23:59:31.197448+0000","last_undegraded":"2024-05-22T23:59:31.197448+0000","last_fullsized":"2024-05-22T23:59:31.197448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:11:47.707506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108893+0000","last_change":"2024-05-22T23:57:33.322417+0000","last_active":"2024-05-22T23:59:30.108893+0000","last_peered":"2024-05-22T23:59:30.108893+0000","last_clean":"2024-05-22T23:59:30.108893+0000","last_became_active":"2024-05-22T23:57:33.322102+0000","last_became_peered":"2024-05-22T23:57:33.322102+0000","last_unstale":"2024-05-22T23:59:30.108893+0000","last_undegraded":"2024-05-22T23:59:30.108893+0000","last_fullsized":"2024-05-22T23:59:30.108893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:26:43.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108560+0000","last_change":"2024-05-22T23:57:37.784424+0000","last_active":"2024-05-22T23:58:31.108560+0000","last_peered":"2024-05-22T23:58:31.108560+0000","last_clean":"2024-05-22T23:58:31.108560+0000","last_became_active":"2024-05-22T23:57:37.783949+0000","last_became_peered":"2024-05-22T23:57:37.783949+0000","last_unstale":"2024-05-22T23:58:31.108560+0000","last_undegraded":"2024-05-22T23:58:31.108560+0000","last_fullsized":"2024-05-22T23:58:31.108560+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:52:23.763273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110764+0000","last_change":"2024-05-22T23:58:31.110764+0000","last_active":"2024-05-22T23:58:31.110764+0000","last_peered":"2024-05-22T23:58:31.110764+0000","last_clean":"2024-05-22T23:58:31.110764+0000","last_became_active":"2024-05-22T23:58:31.103962+0000","last_became_peered":"2024-05-22T23:58:31.103962+0000","last_unstale":"2024-05-22T23:58:31.110764+0000","last_undegraded":"2024-05-22T23:58:31.110764+0000","last_fullsized":"2024-05-22T23:58:31.110764+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:14:24.248179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.122614+0000","last_change":"2024-05-22T23:57:33.324237+0000","last_active":"2024-05-22T23:59:30.122614+0000","last_peered":"2024-05-22T23:59:30.122614+0000","last_clean":"2024-05-22T23:59:30.122614+0000","last_became_active":"2024-05-22T23:57:33.323870+0000","last_became_peered":"2024-05-22T23:57:33.323870+0000","last_unstale":"2024-05-22T23:59:30.122614+0000","last_undegraded":"2024-05-22T23:59:30.122614+0000","last_fullsized":"2024-05-22T23:59:30.122614+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:41:29.328104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096679+0000","last_change":"2024-05-22T23:57:37.787795+0000","last_active":"2024-05-22T23:58:31.096679+0000","last_peered":"2024-05-22T23:58:31.096679+0000","last_clean":"2024-05-22T23:58:31.096679+0000","last_became_active":"2024-05-22T23:57:37.787210+0000","last_became_peered":"2024-05-22T23:57:37.787210+0000","last_unstale":"2024-05-22T23:58:31.096679+0000","last_undegraded":"2024-05-22T23:58:31.096679+0000","last_fullsized":"2024-05-22T23:58:31.096679+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:19:04.616758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102339+0000","last_change":"2024-05-22T23:57:32.321415+0000","last_active":"2024-05-22T23:58:31.102339+0000","last_peered":"2024-05-22T23:58:31.102339+0000","last_clean":"2024-05-22T23:58:31.102339+0000","last_became_active":"2024-05-22T23:57:32.321055+0000","last_became_peered":"2024-05-22T23:57:32.321055+0000","last_unstale":"2024-05-22T23:58:31.102339+0000","last_undegraded":"2024-05-22T23:58:31.102339+0000","last_fullsized":"2024-05-22T23:58:31.102339+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:28:58.577548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.109609+0000","last_change":"2024-05-22T23:58:31.116991+0000","last_active":"2024-05-22T23:59:30.109609+0000","last_peered":"2024-05-22T23:59:30.109609+0000","last_clean":"2024-05-22T23:59:30.109609+0000","last_became_active":"2024-05-22T23:58:31.116526+0000","last_became_peered":"2024-05-22T23:58:31.116526+0000","last_unstale":"2024-05-22T23:59:30.109609+0000","last_undegraded":"2024-05-22T23:59:30.109609+0000","last_fullsized":"2024-05-22T23:59:30.109609+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:51:23.350871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.109785+0000","last_change":"2024-05-22T23:57:37.781367+0000","last_active":"2024-05-22T23:58:31.109785+0000","last_peered":"2024-05-22T23:58:31.109785+0000","last_clean":"2024-05-22T23:58:31.109785+0000","last_became_active":"2024-05-22T23:57:37.780955+0000","last_became_peered":"2024-05-22T23:57:37.780955+0000","last_unstale":"2024-05-22T23:58:31.109785+0000","last_undegraded":"2024-05-22T23:58:31.109785+0000","last_fullsized":"2024-05-22T23:58:31.109785+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:36:18.962940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":247,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:32.199531+0000","last_change":"2024-05-22T23:57:32.313873+0000","last_active":"2024-05-22T23:59:32.199531+0000","last_peered":"2024-05-22T23:59:32.199531+0000","last_clean":"2024-05-22T23:59:32.199531+0000","last_became_active":"2024-05-22T23:57:32.313388+0000","last_became_peered":"2024-05-22T23:57:32.313388+0000","last_unstale":"2024-05-22T23:59:32.199531+0000","last_undegraded":"2024-05-22T23:59:32.199531+0000","last_fullsized":"2024-05-22T23:59:32.199531+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:16.383384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62580,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":183,"num_read_kb":152,"num_write":38,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.093545+0000","last_change":"2024-05-22T23:57:33.318751+0000","last_active":"2024-05-22T23:59:30.093545+0000","last_peered":"2024-05-22T23:59:30.093545+0000","last_clean":"2024-05-22T23:59:30.093545+0000","last_became_active":"2024-05-22T23:57:33.317710+0000","last_became_peered":"2024-05-22T23:57:33.317710+0000","last_unstale":"2024-05-22T23:59:30.093545+0000","last_undegraded":"2024-05-22T23:59:30.093545+0000","last_fullsized":"2024-05-22T23:59:30.093545+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:57:50.852615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102879+0000","last_change":"2024-05-22T23:57:37.790073+0000","last_active":"2024-05-22T23:58:31.102879+0000","last_peered":"2024-05-22T23:58:31.102879+0000","last_clean":"2024-05-22T23:58:31.102879+0000","last_became_active":"2024-05-22T23:57:37.789477+0000","last_became_peered":"2024-05-22T23:57:37.789477+0000","last_unstale":"2024-05-22T23:58:31.102879+0000","last_undegraded":"2024-05-22T23:58:31.102879+0000","last_fullsized":"2024-05-22T23:58:31.102879+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:11.003201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098794+0000","last_change":"2024-05-22T23:57:32.329846+0000","last_active":"2024-05-22T23:58:31.098794+0000","last_peered":"2024-05-22T23:58:31.098794+0000","last_clean":"2024-05-22T23:58:31.098794+0000","last_became_active":"2024-05-22T23:57:32.329414+0000","last_became_peered":"2024-05-22T23:57:32.329414+0000","last_unstale":"2024-05-22T23:58:31.098794+0000","last_undegraded":"2024-05-22T23:58:31.098794+0000","last_fullsized":"2024-05-22T23:58:31.098794+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:15:38.176790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.107191+0000","last_change":"2024-05-22T23:57:33.333298+0000","last_active":"2024-05-22T23:59:30.107191+0000","last_peered":"2024-05-22T23:59:30.107191+0000","last_clean":"2024-05-22T23:59:30.107191+0000","last_became_active":"2024-05-22T23:57:33.332934+0000","last_became_peered":"2024-05-22T23:57:33.332934+0000","last_unstale":"2024-05-22T23:59:30.107191+0000","last_undegraded":"2024-05-22T23:59:30.107191+0000","last_fullsized":"2024-05-22T23:59:30.107191+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:06:00.427627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100781+0000","last_change":"2024-05-22T23:57:37.795628+0000","last_active":"2024-05-22T23:58:31.100781+0000","last_peered":"2024-05-22T23:58:31.100781+0000","last_clean":"2024-05-22T23:58:31.100781+0000","last_became_active":"2024-05-22T23:57:37.795208+0000","last_became_peered":"2024-05-22T23:57:37.795208+0000","last_unstale":"2024-05-22T23:58:31.100781+0000","last_undegraded":"2024-05-22T23:58:31.100781+0000","last_fullsized":"2024-05-22T23:58:31.100781+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:03:12.684474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102064+0000","last_change":"2024-05-22T23:57:32.334039+0000","last_active":"2024-05-22T23:58:31.102064+0000","last_peered":"2024-05-22T23:58:31.102064+0000","last_clean":"2024-05-22T23:58:31.102064+0000","last_became_active":"2024-05-22T23:57:32.333709+0000","last_became_peered":"2024-05-22T23:57:32.333709+0000","last_unstale":"2024-05-22T23:58:31.102064+0000","last_undegraded":"2024-05-22T23:58:31.102064+0000","last_fullsized":"2024-05-22T23:58:31.102064+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:41:43.319094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.118518+0000","last_change":"2024-05-22T23:57:33.328056+0000","last_active":"2024-05-22T23:59:30.118518+0000","last_peered":"2024-05-22T23:59:30.118518+0000","last_clean":"2024-05-22T23:59:30.118518+0000","last_became_active":"2024-05-22T23:57:33.326559+0000","last_became_peered":"2024-05-22T23:57:33.326559+0000","last_unstale":"2024-05-22T23:59:30.118518+0000","last_undegraded":"2024-05-22T23:59:30.118518+0000","last_fullsized":"2024-05-22T23:59:30.118518+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:06.463837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110580+0000","last_change":"2024-05-22T23:57:37.795892+0000","last_active":"2024-05-22T23:58:31.110580+0000","last_peered":"2024-05-22T23:58:31.110580+0000","last_clean":"2024-05-22T23:58:31.110580+0000","last_became_active":"2024-05-22T23:57:37.792246+0000","last_became_peered":"2024-05-22T23:57:37.792246+0000","last_unstale":"2024-05-22T23:58:31.110580+0000","last_undegraded":"2024-05-22T23:58:31.110580+0000","last_fullsized":"2024-05-22T23:58:31.110580+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:18.462267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"93'5","reported_seq":125,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:32.429087+0000","last_change":"2024-05-22T23:57:32.319379+0000","last_active":"2024-05-22T23:59:32.429087+0000","last_peered":"2024-05-22T23:59:32.429087+0000","last_clean":"2024-05-22T23:59:32.429087+0000","last_became_active":"2024-05-22T23:57:32.318803+0000","last_became_peered":"2024-05-22T23:57:32.318803+0000","last_unstale":"2024-05-22T23:59:32.429087+0000","last_undegraded":"2024-05-22T23:59:32.429087+0000","last_fullsized":"2024-05-22T23:59:32.429087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:59:15.689119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1119,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108216+0000","last_change":"2024-05-22T23:57:33.325294+0000","last_active":"2024-05-22T23:59:30.108216+0000","last_peered":"2024-05-22T23:59:30.108216+0000","last_clean":"2024-05-22T23:59:30.108216+0000","last_became_active":"2024-05-22T23:57:33.324339+0000","last_became_peered":"2024-05-22T23:57:33.324339+0000","last_unstale":"2024-05-22T23:59:30.108216+0000","last_undegraded":"2024-05-22T23:59:30.108216+0000","last_fullsized":"2024-05-22T23:59:30.108216+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:29:47.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107019+0000","last_change":"2024-05-22T23:57:37.797961+0000","last_active":"2024-05-22T23:58:31.107019+0000","last_peered":"2024-05-22T23:58:31.107019+0000","last_clean":"2024-05-22T23:58:31.107019+0000","last_became_active":"2024-05-22T23:57:37.797532+0000","last_became_peered":"2024-05-22T23:57:37.797532+0000","last_unstale":"2024-05-22T23:58:31.107019+0000","last_undegraded":"2024-05-22T23:58:31.107019+0000","last_fullsized":"2024-05-22T23:58:31.107019+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:51:12.997674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107627+0000","last_change":"2024-05-22T23:57:32.317434+0000","last_active":"2024-05-22T23:58:31.107627+0000","last_peered":"2024-05-22T23:58:31.107627+0000","last_clean":"2024-05-22T23:58:31.107627+0000","last_became_active":"2024-05-22T23:57:32.316987+0000","last_became_peered":"2024-05-22T23:57:32.316987+0000","last_unstale":"2024-05-22T23:58:31.107627+0000","last_undegraded":"2024-05-22T23:58:31.107627+0000","last_fullsized":"2024-05-22T23:58:31.107627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:46:50.306756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.105085+0000","last_change":"2024-05-22T23:57:33.328670+0000","last_active":"2024-05-22T23:59:30.105085+0000","last_peered":"2024-05-22T23:59:30.105085+0000","last_clean":"2024-05-22T23:59:30.105085+0000","last_became_active":"2024-05-22T23:57:33.328294+0000","last_became_peered":"2024-05-22T23:57:33.328294+0000","last_unstale":"2024-05-22T23:59:30.105085+0000","last_undegraded":"2024-05-22T23:59:30.105085+0000","last_fullsized":"2024-05-22T23:59:30.105085+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:09:10.807844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101376+0000","last_change":"2024-05-22T23:57:37.788885+0000","last_active":"2024-05-22T23:58:31.101376+0000","last_peered":"2024-05-22T23:58:31.101376+0000","last_clean":"2024-05-22T23:58:31.101376+0000","last_became_active":"2024-05-22T23:57:37.788145+0000","last_became_peered":"2024-05-22T23:57:37.788145+0000","last_unstale":"2024-05-22T23:58:31.101376+0000","last_undegraded":"2024-05-22T23:58:31.101376+0000","last_fullsized":"2024-05-22T23:58:31.101376+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:14:50.027785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108211+0000","last_change":"2024-05-22T23:57:32.317855+0000","last_active":"2024-05-22T23:58:31.108211+0000","last_peered":"2024-05-22T23:58:31.108211+0000","last_clean":"2024-05-22T23:58:31.108211+0000","last_became_active":"2024-05-22T23:57:32.317087+0000","last_became_peered":"2024-05-22T23:57:32.317087+0000","last_unstale":"2024-05-22T23:58:31.108211+0000","last_undegraded":"2024-05-22T23:58:31.108211+0000","last_fullsized":"2024-05-22T23:58:31.108211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:49:40.486222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.093428+0000","last_change":"2024-05-22T23:57:33.323897+0000","last_active":"2024-05-22T23:59:30.093428+0000","last_peered":"2024-05-22T23:59:30.093428+0000","last_clean":"2024-05-22T23:59:30.093428+0000","last_became_active":"2024-05-22T23:57:33.323567+0000","last_became_peered":"2024-05-22T23:57:33.323567+0000","last_unstale":"2024-05-22T23:59:30.093428+0000","last_undegraded":"2024-05-22T23:59:30.093428+0000","last_fullsized":"2024-05-22T23:59:30.093428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:17:48.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107731+0000","last_change":"2024-05-22T23:57:38.218313+0000","last_active":"2024-05-22T23:58:31.107731+0000","last_peered":"2024-05-22T23:58:31.107731+0000","last_clean":"2024-05-22T23:58:31.107731+0000","last_became_active":"2024-05-22T23:57:38.217756+0000","last_became_peered":"2024-05-22T23:57:38.217756+0000","last_unstale":"2024-05-22T23:58:31.107731+0000","last_undegraded":"2024-05-22T23:58:31.107731+0000","last_fullsized":"2024-05-22T23:58:31.107731+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:07:55.122260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110929+0000","last_change":"2024-05-22T23:57:32.325729+0000","last_active":"2024-05-22T23:58:31.110929+0000","last_peered":"2024-05-22T23:58:31.110929+0000","last_clean":"2024-05-22T23:58:31.110929+0000","last_became_active":"2024-05-22T23:57:32.325383+0000","last_became_peered":"2024-05-22T23:57:32.325383+0000","last_unstale":"2024-05-22T23:58:31.110929+0000","last_undegraded":"2024-05-22T23:58:31.110929+0000","last_fullsized":"2024-05-22T23:58:31.110929+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:57.978806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.092090+0000","last_change":"2024-05-22T23:57:33.337499+0000","last_active":"2024-05-22T23:59:30.092090+0000","last_peered":"2024-05-22T23:59:30.092090+0000","last_clean":"2024-05-22T23:59:30.092090+0000","last_became_active":"2024-05-22T23:57:33.337271+0000","last_became_peered":"2024-05-22T23:57:33.337271+0000","last_unstale":"2024-05-22T23:59:30.092090+0000","last_undegraded":"2024-05-22T23:59:30.092090+0000","last_fullsized":"2024-05-22T23:59:30.092090+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:32:39.133984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.111255+0000","last_change":"2024-05-22T23:57:37.792707+0000","last_active":"2024-05-22T23:58:31.111255+0000","last_peered":"2024-05-22T23:58:31.111255+0000","last_clean":"2024-05-22T23:58:31.111255+0000","last_became_active":"2024-05-22T23:57:37.792209+0000","last_became_peered":"2024-05-22T23:57:37.792209+0000","last_unstale":"2024-05-22T23:58:31.111255+0000","last_undegraded":"2024-05-22T23:58:31.111255+0000","last_fullsized":"2024-05-22T23:58:31.111255+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:36:34.089424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.112612+0000","last_change":"2024-05-22T23:58:31.112612+0000","last_active":"2024-05-22T23:58:31.112612+0000","last_peered":"2024-05-22T23:58:31.112612+0000","last_clean":"2024-05-22T23:58:31.112612+0000","last_became_active":"2024-05-22T23:58:31.112181+0000","last_became_peered":"2024-05-22T23:58:31.112181+0000","last_unstale":"2024-05-22T23:58:31.112612+0000","last_undegraded":"2024-05-22T23:58:31.112612+0000","last_fullsized":"2024-05-22T23:58:31.112612+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:25:18.717294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.120922+0000","last_change":"2024-05-22T23:57:33.328129+0000","last_active":"2024-05-22T23:59:30.120922+0000","last_peered":"2024-05-22T23:59:30.120922+0000","last_clean":"2024-05-22T23:59:30.120922+0000","last_became_active":"2024-05-22T23:57:33.327732+0000","last_became_peered":"2024-05-22T23:57:33.327732+0000","last_unstale":"2024-05-22T23:59:30.120922+0000","last_undegraded":"2024-05-22T23:59:30.120922+0000","last_fullsized":"2024-05-22T23:59:30.120922+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:25:14.105235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107157+0000","last_change":"2024-05-22T23:57:38.220884+0000","last_active":"2024-05-22T23:58:31.107157+0000","last_peered":"2024-05-22T23:58:31.107157+0000","last_clean":"2024-05-22T23:58:31.107157+0000","last_became_active":"2024-05-22T23:57:38.219365+0000","last_became_peered":"2024-05-22T23:57:38.219365+0000","last_unstale":"2024-05-22T23:58:31.107157+0000","last_undegraded":"2024-05-22T23:58:31.107157+0000","last_fullsized":"2024-05-22T23:58:31.107157+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:54:56.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.107243+0000","last_change":"2024-05-22T23:57:33.328272+0000","last_active":"2024-05-22T23:59:30.107243+0000","last_peered":"2024-05-22T23:59:30.107243+0000","last_clean":"2024-05-22T23:59:30.107243+0000","last_became_active":"2024-05-22T23:57:33.327915+0000","last_became_peered":"2024-05-22T23:57:33.327915+0000","last_unstale":"2024-05-22T23:59:30.107243+0000","last_undegraded":"2024-05-22T23:59:30.107243+0000","last_fullsized":"2024-05-22T23:59:30.107243+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:59:09.085649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"93'5","reported_seq":121,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:32.428479+0000","last_change":"2024-05-22T23:57:32.331979+0000","last_active":"2024-05-22T23:59:32.428479+0000","last_peered":"2024-05-22T23:59:32.428479+0000","last_clean":"2024-05-22T23:59:32.428479+0000","last_became_active":"2024-05-22T23:57:32.330983+0000","last_became_peered":"2024-05-22T23:57:32.330983+0000","last_unstale":"2024-05-22T23:59:32.428479+0000","last_undegraded":"2024-05-22T23:59:32.428479+0000","last_fullsized":"2024-05-22T23:59:32.428479+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:07:19.137652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101829+0000","last_change":"2024-05-22T23:57:37.787935+0000","last_active":"2024-05-22T23:58:31.101829+0000","last_peered":"2024-05-22T23:58:31.101829+0000","last_clean":"2024-05-22T23:58:31.101829+0000","last_became_active":"2024-05-22T23:57:37.787495+0000","last_became_peered":"2024-05-22T23:57:37.787495+0000","last_unstale":"2024-05-22T23:58:31.101829+0000","last_undegraded":"2024-05-22T23:58:31.101829+0000","last_fullsized":"2024-05-22T23:58:31.101829+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:37.682711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.111144+0000","last_change":"2024-05-22T23:57:33.329451+0000","last_active":"2024-05-22T23:59:30.111144+0000","last_peered":"2024-05-22T23:59:30.111144+0000","last_clean":"2024-05-22T23:59:30.111144+0000","last_became_active":"2024-05-22T23:57:33.329107+0000","last_became_peered":"2024-05-22T23:57:33.329107+0000","last_unstale":"2024-05-22T23:59:30.111144+0000","last_undegraded":"2024-05-22T23:59:30.111144+0000","last_fullsized":"2024-05-22T23:59:30.111144+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:44:50.198040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096649+0000","last_change":"2024-05-22T23:57:32.321873+0000","last_active":"2024-05-22T23:58:31.096649+0000","last_peered":"2024-05-22T23:58:31.096649+0000","last_clean":"2024-05-22T23:58:31.096649+0000","last_became_active":"2024-05-22T23:57:32.321550+0000","last_became_peered":"2024-05-22T23:57:32.321550+0000","last_unstale":"2024-05-22T23:58:31.096649+0000","last_undegraded":"2024-05-22T23:58:31.096649+0000","last_fullsized":"2024-05-22T23:58:31.096649+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:30:17.479238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":361,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101701+0000","last_change":"2024-05-22T23:56:11.327788+0000","last_active":"2024-05-22T23:58:31.101701+0000","last_peered":"2024-05-22T23:58:31.101701+0000","last_clean":"2024-05-22T23:58:31.101701+0000","last_became_active":"2024-05-22T23:56:11.327232+0000","last_became_peered":"2024-05-22T23:56:11.327232+0000","last_unstale":"2024-05-22T23:58:31.101701+0000","last_undegraded":"2024-05-22T23:58:31.101701+0000","last_fullsized":"2024-05-22T23:58:31.101701+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_clean_scrub_stamp":"2024-05-22T23:52:16.719643+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:26:13.077470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102591+0000","last_change":"2024-05-22T23:57:38.219326+0000","last_active":"2024-05-22T23:58:31.102591+0000","last_peered":"2024-05-22T23:58:31.102591+0000","last_clean":"2024-05-22T23:58:31.102591+0000","last_became_active":"2024-05-22T23:57:38.218835+0000","last_became_peered":"2024-05-22T23:57:38.218835+0000","last_unstale":"2024-05-22T23:58:31.102591+0000","last_undegraded":"2024-05-22T23:58:31.102591+0000","last_fullsized":"2024-05-22T23:58:31.102591+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:29.722862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.103328+0000","last_change":"2024-05-22T23:57:33.316760+0000","last_active":"2024-05-22T23:59:30.103328+0000","last_peered":"2024-05-22T23:59:30.103328+0000","last_clean":"2024-05-22T23:59:30.103328+0000","last_became_active":"2024-05-22T23:57:33.316328+0000","last_became_peered":"2024-05-22T23:57:33.316328+0000","last_unstale":"2024-05-22T23:59:30.103328+0000","last_undegraded":"2024-05-22T23:59:30.103328+0000","last_fullsized":"2024-05-22T23:59:30.103328+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:53:11.592134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102301+0000","last_change":"2024-05-22T23:57:32.319154+0000","last_active":"2024-05-22T23:58:31.102301+0000","last_peered":"2024-05-22T23:58:31.102301+0000","last_clean":"2024-05-22T23:58:31.102301+0000","last_became_active":"2024-05-22T23:57:32.318808+0000","last_became_peered":"2024-05-22T23:57:32.318808+0000","last_unstale":"2024-05-22T23:58:31.102301+0000","last_undegraded":"2024-05-22T23:58:31.102301+0000","last_fullsized":"2024-05-22T23:58:31.102301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:37:47.271767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.101063+0000","last_change":"2024-05-22T23:57:37.791464+0000","last_active":"2024-05-22T23:58:31.101063+0000","last_peered":"2024-05-22T23:58:31.101063+0000","last_clean":"2024-05-22T23:58:31.101063+0000","last_became_active":"2024-05-22T23:57:37.791044+0000","last_became_peered":"2024-05-22T23:57:37.791044+0000","last_unstale":"2024-05-22T23:58:31.101063+0000","last_undegraded":"2024-05-22T23:58:31.101063+0000","last_fullsized":"2024-05-22T23:58:31.101063+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:47:00.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.108206+0000","last_change":"2024-05-22T23:57:32.322740+0000","last_active":"2024-05-22T23:58:31.108206+0000","last_peered":"2024-05-22T23:58:31.108206+0000","last_clean":"2024-05-22T23:58:31.108206+0000","last_became_active":"2024-05-22T23:57:32.322425+0000","last_became_peered":"2024-05-22T23:57:32.322425+0000","last_unstale":"2024-05-22T23:58:31.108206+0000","last_undegraded":"2024-05-22T23:58:31.108206+0000","last_fullsized":"2024-05-22T23:58:31.108206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:05:10.522462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.104443+0000","last_change":"2024-05-22T23:57:33.324667+0000","last_active":"2024-05-22T23:59:30.104443+0000","last_peered":"2024-05-22T23:59:30.104443+0000","last_clean":"2024-05-22T23:59:30.104443+0000","last_became_active":"2024-05-22T23:57:33.324251+0000","last_became_peered":"2024-05-22T23:57:33.324251+0000","last_unstale":"2024-05-22T23:59:30.104443+0000","last_undegraded":"2024-05-22T23:59:30.104443+0000","last_fullsized":"2024-05-22T23:59:30.104443+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:49:57.203996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097700+0000","last_change":"2024-05-22T23:57:38.222346+0000","last_active":"2024-05-22T23:58:31.097700+0000","last_peered":"2024-05-22T23:58:31.097700+0000","last_clean":"2024-05-22T23:58:31.097700+0000","last_became_active":"2024-05-22T23:57:38.220248+0000","last_became_peered":"2024-05-22T23:57:38.220248+0000","last_unstale":"2024-05-22T23:58:31.097700+0000","last_undegraded":"2024-05-22T23:58:31.097700+0000","last_fullsized":"2024-05-22T23:58:31.097700+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:26:44.026641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'9","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.126975+0000","last_change":"2024-05-22T23:58:31.126056+0000","last_active":"2024-05-22T23:58:31.126975+0000","last_peered":"2024-05-22T23:58:31.126975+0000","last_clean":"2024-05-22T23:58:31.126975+0000","last_became_active":"2024-05-22T23:58:31.117186+0000","last_became_peered":"2024-05-22T23:58:31.117186+0000","last_unstale":"2024-05-22T23:58:31.126975+0000","last_undegraded":"2024-05-22T23:58:31.126975+0000","last_fullsized":"2024-05-22T23:58:31.126975+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:21:34.156655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.097685+0000","last_change":"2024-05-22T23:58:31.116649+0000","last_active":"2024-05-22T23:59:30.097685+0000","last_peered":"2024-05-22T23:59:30.097685+0000","last_clean":"2024-05-22T23:59:30.097685+0000","last_became_active":"2024-05-22T23:58:31.116009+0000","last_became_peered":"2024-05-22T23:58:31.116009+0000","last_unstale":"2024-05-22T23:59:30.097685+0000","last_undegraded":"2024-05-22T23:59:30.097685+0000","last_fullsized":"2024-05-22T23:59:30.097685+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:00:29.964565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098180+0000","last_change":"2024-05-22T23:57:37.778524+0000","last_active":"2024-05-22T23:58:31.098180+0000","last_peered":"2024-05-22T23:58:31.098180+0000","last_clean":"2024-05-22T23:58:31.098180+0000","last_became_active":"2024-05-22T23:57:37.777827+0000","last_became_peered":"2024-05-22T23:57:37.777827+0000","last_unstale":"2024-05-22T23:58:31.098180+0000","last_undegraded":"2024-05-22T23:58:31.098180+0000","last_fullsized":"2024-05-22T23:58:31.098180+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:30:59.154921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097105+0000","last_change":"2024-05-22T23:57:32.317649+0000","last_active":"2024-05-22T23:58:31.097105+0000","last_peered":"2024-05-22T23:58:31.097105+0000","last_clean":"2024-05-22T23:58:31.097105+0000","last_became_active":"2024-05-22T23:57:32.317280+0000","last_became_peered":"2024-05-22T23:57:32.317280+0000","last_unstale":"2024-05-22T23:58:31.097105+0000","last_undegraded":"2024-05-22T23:58:31.097105+0000","last_fullsized":"2024-05-22T23:58:31.097105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:01:16.917079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.126177+0000","last_change":"2024-05-22T23:57:33.339296+0000","last_active":"2024-05-22T23:59:30.126177+0000","last_peered":"2024-05-22T23:59:30.126177+0000","last_clean":"2024-05-22T23:59:30.126177+0000","last_became_active":"2024-05-22T23:57:33.338457+0000","last_became_peered":"2024-05-22T23:57:33.338457+0000","last_unstale":"2024-05-22T23:59:30.126177+0000","last_undegraded":"2024-05-22T23:59:30.126177+0000","last_fullsized":"2024-05-22T23:59:30.126177+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:13:05.927893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097166+0000","last_change":"2024-05-22T23:57:38.218649+0000","last_active":"2024-05-22T23:58:31.097166+0000","last_peered":"2024-05-22T23:58:31.097166+0000","last_clean":"2024-05-22T23:58:31.097166+0000","last_became_active":"2024-05-22T23:57:38.218189+0000","last_became_peered":"2024-05-22T23:57:38.218189+0000","last_unstale":"2024-05-22T23:58:31.097166+0000","last_undegraded":"2024-05-22T23:58:31.097166+0000","last_fullsized":"2024-05-22T23:58:31.097166+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:30.747293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098608+0000","last_change":"2024-05-22T23:57:32.309379+0000","last_active":"2024-05-22T23:58:31.098608+0000","last_peered":"2024-05-22T23:58:31.098608+0000","last_clean":"2024-05-22T23:58:31.098608+0000","last_became_active":"2024-05-22T23:57:32.309068+0000","last_became_peered":"2024-05-22T23:57:32.309068+0000","last_unstale":"2024-05-22T23:58:31.098608+0000","last_undegraded":"2024-05-22T23:58:31.098608+0000","last_fullsized":"2024-05-22T23:58:31.098608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:43:54.787996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.079731+0000","last_change":"2024-05-22T23:57:33.320253+0000","last_active":"2024-05-22T23:59:30.079731+0000","last_peered":"2024-05-22T23:59:30.079731+0000","last_clean":"2024-05-22T23:59:30.079731+0000","last_became_active":"2024-05-22T23:57:33.319604+0000","last_became_peered":"2024-05-22T23:57:33.319604+0000","last_unstale":"2024-05-22T23:59:30.079731+0000","last_undegraded":"2024-05-22T23:59:30.079731+0000","last_fullsized":"2024-05-22T23:59:30.079731+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:23:51.745040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.110063+0000","last_change":"2024-05-22T23:57:37.795368+0000","last_active":"2024-05-22T23:58:31.110063+0000","last_peered":"2024-05-22T23:58:31.110063+0000","last_clean":"2024-05-22T23:58:31.110063+0000","last_became_active":"2024-05-22T23:57:37.792848+0000","last_became_peered":"2024-05-22T23:57:37.792848+0000","last_unstale":"2024-05-22T23:58:31.110063+0000","last_undegraded":"2024-05-22T23:58:31.110063+0000","last_fullsized":"2024-05-22T23:58:31.110063+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:45.859867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098291+0000","last_change":"2024-05-22T23:57:32.329147+0000","last_active":"2024-05-22T23:58:31.098291+0000","last_peered":"2024-05-22T23:58:31.098291+0000","last_clean":"2024-05-22T23:58:31.098291+0000","last_became_active":"2024-05-22T23:57:32.327254+0000","last_became_peered":"2024-05-22T23:57:32.327254+0000","last_unstale":"2024-05-22T23:58:31.098291+0000","last_undegraded":"2024-05-22T23:58:31.098291+0000","last_fullsized":"2024-05-22T23:58:31.098291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:55:24.915817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.119795+0000","last_change":"2024-05-22T23:57:33.332843+0000","last_active":"2024-05-22T23:59:30.119795+0000","last_peered":"2024-05-22T23:59:30.119795+0000","last_clean":"2024-05-22T23:59:30.119795+0000","last_became_active":"2024-05-22T23:57:33.332497+0000","last_became_peered":"2024-05-22T23:57:33.332497+0000","last_unstale":"2024-05-22T23:59:30.119795+0000","last_undegraded":"2024-05-22T23:59:30.119795+0000","last_fullsized":"2024-05-22T23:59:30.119795+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:55:16.109676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.096973+0000","last_change":"2024-05-22T23:57:37.787609+0000","last_active":"2024-05-22T23:58:31.096973+0000","last_peered":"2024-05-22T23:58:31.096973+0000","last_clean":"2024-05-22T23:58:31.096973+0000","last_became_active":"2024-05-22T23:57:37.787153+0000","last_became_peered":"2024-05-22T23:57:37.787153+0000","last_unstale":"2024-05-22T23:58:31.096973+0000","last_undegraded":"2024-05-22T23:58:31.096973+0000","last_fullsized":"2024-05-22T23:58:31.096973+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:57:51.640037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"91'1","reported_seq":28,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098316+0000","last_change":"2024-05-22T23:57:32.329784+0000","last_active":"2024-05-22T23:58:31.098316+0000","last_peered":"2024-05-22T23:58:31.098316+0000","last_clean":"2024-05-22T23:58:31.098316+0000","last_became_active":"2024-05-22T23:57:32.329393+0000","last_became_peered":"2024-05-22T23:57:32.329393+0000","last_unstale":"2024-05-22T23:58:31.098316+0000","last_undegraded":"2024-05-22T23:58:31.098316+0000","last_fullsized":"2024-05-22T23:58:31.098316+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:17:54.811962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.122622+0000","last_change":"2024-05-22T23:57:33.318803+0000","last_active":"2024-05-22T23:59:30.122622+0000","last_peered":"2024-05-22T23:59:30.122622+0000","last_clean":"2024-05-22T23:59:30.122622+0000","last_became_active":"2024-05-22T23:57:33.318170+0000","last_became_peered":"2024-05-22T23:57:33.318170+0000","last_unstale":"2024-05-22T23:59:30.122622+0000","last_undegraded":"2024-05-22T23:59:30.122622+0000","last_fullsized":"2024-05-22T23:59:30.122622+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:56:56.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.097015+0000","last_change":"2024-05-22T23:57:38.218583+0000","last_active":"2024-05-22T23:58:31.097015+0000","last_peered":"2024-05-22T23:58:31.097015+0000","last_clean":"2024-05-22T23:58:31.097015+0000","last_became_active":"2024-05-22T23:57:38.218048+0000","last_became_peered":"2024-05-22T23:57:38.218048+0000","last_unstale":"2024-05-22T23:58:31.097015+0000","last_undegraded":"2024-05-22T23:58:31.097015+0000","last_fullsized":"2024-05-22T23:58:31.097015+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:10:57.579656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.107852+0000","last_change":"2024-05-22T23:57:32.317908+0000","last_active":"2024-05-22T23:58:31.107852+0000","last_peered":"2024-05-22T23:58:31.107852+0000","last_clean":"2024-05-22T23:58:31.107852+0000","last_became_active":"2024-05-22T23:57:32.317580+0000","last_became_peered":"2024-05-22T23:57:32.317580+0000","last_unstale":"2024-05-22T23:58:31.107852+0000","last_undegraded":"2024-05-22T23:58:31.107852+0000","last_fullsized":"2024-05-22T23:58:31.107852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:50:02.044817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.109257+0000","last_change":"2024-05-22T23:57:33.321184+0000","last_active":"2024-05-22T23:59:30.109257+0000","last_peered":"2024-05-22T23:59:30.109257+0000","last_clean":"2024-05-22T23:59:30.109257+0000","last_became_active":"2024-05-22T23:57:33.320418+0000","last_became_peered":"2024-05-22T23:57:33.320418+0000","last_unstale":"2024-05-22T23:59:30.109257+0000","last_undegraded":"2024-05-22T23:59:30.109257+0000","last_fullsized":"2024-05-22T23:59:30.109257+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:22:43.019809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.098941+0000","last_change":"2024-05-22T23:57:37.795718+0000","last_active":"2024-05-22T23:58:31.098941+0000","last_peered":"2024-05-22T23:58:31.098941+0000","last_clean":"2024-05-22T23:58:31.098941+0000","last_became_active":"2024-05-22T23:57:37.795297+0000","last_became_peered":"2024-05-22T23:57:37.795297+0000","last_unstale":"2024-05-22T23:58:31.098941+0000","last_undegraded":"2024-05-22T23:58:31.098941+0000","last_fullsized":"2024-05-22T23:58:31.098941+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:56:47.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.099085+0000","last_change":"2024-05-22T23:57:32.311921+0000","last_active":"2024-05-22T23:58:31.099085+0000","last_peered":"2024-05-22T23:58:31.099085+0000","last_clean":"2024-05-22T23:58:31.099085+0000","last_became_active":"2024-05-22T23:57:32.311600+0000","last_became_peered":"2024-05-22T23:57:32.311600+0000","last_unstale":"2024-05-22T23:58:31.099085+0000","last_undegraded":"2024-05-22T23:58:31.099085+0000","last_fullsized":"2024-05-22T23:58:31.099085+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:35:21.882773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.104314+0000","last_change":"2024-05-22T23:57:33.321289+0000","last_active":"2024-05-22T23:59:30.104314+0000","last_peered":"2024-05-22T23:59:30.104314+0000","last_clean":"2024-05-22T23:59:30.104314+0000","last_became_active":"2024-05-22T23:57:33.320458+0000","last_became_peered":"2024-05-22T23:57:33.320458+0000","last_unstale":"2024-05-22T23:59:30.104314+0000","last_undegraded":"2024-05-22T23:59:30.104314+0000","last_fullsized":"2024-05-22T23:59:30.104314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:02:05.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.099889+0000","last_change":"2024-05-22T23:57:37.786062+0000","last_active":"2024-05-22T23:58:31.099889+0000","last_peered":"2024-05-22T23:58:31.099889+0000","last_clean":"2024-05-22T23:58:31.099889+0000","last_became_active":"2024-05-22T23:57:37.785624+0000","last_became_peered":"2024-05-22T23:57:37.785624+0000","last_unstale":"2024-05-22T23:58:31.099889+0000","last_undegraded":"2024-05-22T23:58:31.099889+0000","last_fullsized":"2024-05-22T23:58:31.099889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:38:56.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'1","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.102580+0000","last_change":"2024-05-22T23:57:32.333909+0000","last_active":"2024-05-22T23:58:31.102580+0000","last_peered":"2024-05-22T23:58:31.102580+0000","last_clean":"2024-05-22T23:58:31.102580+0000","last_became_active":"2024-05-22T23:57:32.333531+0000","last_became_peered":"2024-05-22T23:57:32.333531+0000","last_unstale":"2024-05-22T23:58:31.102580+0000","last_undegraded":"2024-05-22T23:58:31.102580+0000","last_fullsized":"2024-05-22T23:58:31.102580+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:53:21.843150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108530+0000","last_change":"2024-05-22T23:57:33.334366+0000","last_active":"2024-05-22T23:59:30.108530+0000","last_peered":"2024-05-22T23:59:30.108530+0000","last_clean":"2024-05-22T23:59:30.108530+0000","last_became_active":"2024-05-22T23:57:33.334037+0000","last_became_peered":"2024-05-22T23:57:33.334037+0000","last_unstale":"2024-05-22T23:59:30.108530+0000","last_undegraded":"2024-05-22T23:59:30.108530+0000","last_fullsized":"2024-05-22T23:59:30.108530+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:15:10.631087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.100135+0000","last_change":"2024-05-22T23:57:37.780241+0000","last_active":"2024-05-22T23:58:31.100135+0000","last_peered":"2024-05-22T23:58:31.100135+0000","last_clean":"2024-05-22T23:58:31.100135+0000","last_became_active":"2024-05-22T23:57:37.779816+0000","last_became_peered":"2024-05-22T23:57:37.779816+0000","last_unstale":"2024-05-22T23:58:31.100135+0000","last_undegraded":"2024-05-22T23:58:31.100135+0000","last_fullsized":"2024-05-22T23:58:31.100135+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:11.397681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:58:31.106923+0000","last_change":"2024-05-22T23:57:32.334829+0000","last_active":"2024-05-22T23:58:31.106923+0000","last_peered":"2024-05-22T23:58:31.106923+0000","last_clean":"2024-05-22T23:58:31.106923+0000","last_became_active":"2024-05-22T23:57:32.334368+0000","last_became_peered":"2024-05-22T23:57:32.334368+0000","last_unstale":"2024-05-22T23:58:31.106923+0000","last_undegraded":"2024-05-22T23:58:31.106923+0000","last_fullsized":"2024-05-22T23:58:31.106923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:17:42.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.105274+0000","last_change":"2024-05-22T23:57:33.318740+0000","last_active":"2024-05-22T23:59:30.105274+0000","last_peered":"2024-05-22T23:59:30.105274+0000","last_clean":"2024-05-22T23:59:30.105274+0000","last_became_active":"2024-05-22T23:57:33.318363+0000","last_became_peered":"2024-05-22T23:57:33.318363+0000","last_unstale":"2024-05-22T23:59:30.105274+0000","last_undegraded":"2024-05-22T23:59:30.105274+0000","last_fullsized":"2024-05-22T23:59:30.105274+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:51:42.319667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.096705+0000","last_change":"2024-05-22T23:57:33.325582+0000","last_active":"2024-05-22T23:59:30.096705+0000","last_peered":"2024-05-22T23:59:30.096705+0000","last_clean":"2024-05-22T23:59:30.096705+0000","last_became_active":"2024-05-22T23:57:33.325268+0000","last_became_peered":"2024-05-22T23:57:33.325268+0000","last_unstale":"2024-05-22T23:59:30.096705+0000","last_undegraded":"2024-05-22T23:59:30.096705+0000","last_fullsized":"2024-05-22T23:59:30.096705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:02:18.968634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.102477+0000","last_change":"2024-05-22T23:58:31.109626+0000","last_active":"2024-05-22T23:59:30.102477+0000","last_peered":"2024-05-22T23:59:30.102477+0000","last_clean":"2024-05-22T23:59:30.102477+0000","last_became_active":"2024-05-22T23:58:31.109179+0000","last_became_peered":"2024-05-22T23:58:31.109179+0000","last_unstale":"2024-05-22T23:59:30.102477+0000","last_undegraded":"2024-05-22T23:59:30.102477+0000","last_fullsized":"2024-05-22T23:59:30.102477+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:36.653833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.123926+0000","last_change":"2024-05-22T23:57:33.332935+0000","last_active":"2024-05-22T23:59:30.123926+0000","last_peered":"2024-05-22T23:59:30.123926+0000","last_clean":"2024-05-22T23:59:30.123926+0000","last_became_active":"2024-05-22T23:57:33.332607+0000","last_became_peered":"2024-05-22T23:57:33.332607+0000","last_unstale":"2024-05-22T23:59:30.123926+0000","last_undegraded":"2024-05-22T23:59:30.123926+0000","last_fullsized":"2024-05-22T23:59:30.123926+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:09.589896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.126677+0000","last_change":"2024-05-22T23:57:33.333269+0000","last_active":"2024-05-22T23:59:30.126677+0000","last_peered":"2024-05-22T23:59:30.126677+0000","last_clean":"2024-05-22T23:59:30.126677+0000","last_became_active":"2024-05-22T23:57:33.329017+0000","last_became_peered":"2024-05-22T23:57:33.329017+0000","last_unstale":"2024-05-22T23:59:30.126677+0000","last_undegraded":"2024-05-22T23:59:30.126677+0000","last_fullsized":"2024-05-22T23:59:30.126677+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:35:42.156434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.109808+0000","last_change":"2024-05-22T23:57:33.328721+0000","last_active":"2024-05-22T23:59:30.109808+0000","last_peered":"2024-05-22T23:59:30.109808+0000","last_clean":"2024-05-22T23:59:30.109808+0000","last_became_active":"2024-05-22T23:57:33.328318+0000","last_became_peered":"2024-05-22T23:57:33.328318+0000","last_unstale":"2024-05-22T23:59:30.109808+0000","last_undegraded":"2024-05-22T23:59:30.109808+0000","last_fullsized":"2024-05-22T23:59:30.109808+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:14.772688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.092872+0000","last_change":"2024-05-22T23:57:33.324777+0000","last_active":"2024-05-22T23:59:30.092872+0000","last_peered":"2024-05-22T23:59:30.092872+0000","last_clean":"2024-05-22T23:59:30.092872+0000","last_became_active":"2024-05-22T23:57:33.324446+0000","last_became_peered":"2024-05-22T23:57:33.324446+0000","last_unstale":"2024-05-22T23:59:30.092872+0000","last_undegraded":"2024-05-22T23:59:30.092872+0000","last_fullsized":"2024-05-22T23:59:30.092872+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:20:42.327094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.099266+0000","last_change":"2024-05-22T23:57:33.318783+0000","last_active":"2024-05-22T23:59:30.099266+0000","last_peered":"2024-05-22T23:59:30.099266+0000","last_clean":"2024-05-22T23:59:30.099266+0000","last_became_active":"2024-05-22T23:57:33.318481+0000","last_became_peered":"2024-05-22T23:57:33.318481+0000","last_unstale":"2024-05-22T23:59:30.099266+0000","last_undegraded":"2024-05-22T23:59:30.099266+0000","last_fullsized":"2024-05-22T23:59:30.099266+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:14:35.745683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":27,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.106040+0000","last_change":"2024-05-22T23:57:33.319939+0000","last_active":"2024-05-22T23:59:30.106040+0000","last_peered":"2024-05-22T23:59:30.106040+0000","last_clean":"2024-05-22T23:59:30.106040+0000","last_became_active":"2024-05-22T23:57:33.316029+0000","last_became_peered":"2024-05-22T23:57:33.316029+0000","last_unstale":"2024-05-22T23:59:30.106040+0000","last_undegraded":"2024-05-22T23:59:30.106040+0000","last_fullsized":"2024-05-22T23:59:30.106040+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:46:34.383953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.116019+0000","last_change":"2024-05-22T23:57:33.334625+0000","last_active":"2024-05-22T23:59:30.116019+0000","last_peered":"2024-05-22T23:59:30.116019+0000","last_clean":"2024-05-22T23:59:30.116019+0000","last_became_active":"2024-05-22T23:57:33.334341+0000","last_became_peered":"2024-05-22T23:57:33.334341+0000","last_unstale":"2024-05-22T23:59:30.116019+0000","last_undegraded":"2024-05-22T23:59:30.116019+0000","last_fullsized":"2024-05-22T23:59:30.116019+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:38.319721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.107679+0000","last_change":"2024-05-22T23:57:33.330958+0000","last_active":"2024-05-22T23:59:30.107679+0000","last_peered":"2024-05-22T23:59:30.107679+0000","last_clean":"2024-05-22T23:59:30.107679+0000","last_became_active":"2024-05-22T23:57:33.317355+0000","last_became_peered":"2024-05-22T23:57:33.317355+0000","last_unstale":"2024-05-22T23:59:30.107679+0000","last_undegraded":"2024-05-22T23:59:30.107679+0000","last_fullsized":"2024-05-22T23:59:30.107679+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:49:43.784780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.108012+0000","last_change":"2024-05-22T23:57:33.319530+0000","last_active":"2024-05-22T23:59:30.108012+0000","last_peered":"2024-05-22T23:59:30.108012+0000","last_clean":"2024-05-22T23:59:30.108012+0000","last_became_active":"2024-05-22T23:57:33.319122+0000","last_became_peered":"2024-05-22T23:57:33.319122+0000","last_unstale":"2024-05-22T23:59:30.108012+0000","last_undegraded":"2024-05-22T23:59:30.108012+0000","last_fullsized":"2024-05-22T23:59:30.108012+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:31:26.937804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.124388+0000","last_change":"2024-05-22T23:57:33.323722+0000","last_active":"2024-05-22T23:59:30.124388+0000","last_peered":"2024-05-22T23:59:30.124388+0000","last_clean":"2024-05-22T23:59:30.124388+0000","last_became_active":"2024-05-22T23:57:33.323343+0000","last_became_peered":"2024-05-22T23:57:33.323343+0000","last_unstale":"2024-05-22T23:59:30.124388+0000","last_undegraded":"2024-05-22T23:59:30.124388+0000","last_fullsized":"2024-05-22T23:59:30.124388+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:24:59.575536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-22T23:59:30.090145+0000","last_change":"2024-05-22T23:57:33.320925+0000","last_active":"2024-05-22T23:59:30.090145+0000","last_peered":"2024-05-22T23:59:30.090145+0000","last_clean":"2024-05-22T23:59:30.090145+0000","last_became_active":"2024-05-22T23:57:33.320570+0000","last_became_peered":"2024-05-22T23:57:33.320570+0000","last_unstale":"2024-05-22T23:59:30.090145+0000","last_undegraded":"2024-05-22T23:59:30.090145+0000","last_fullsized":"2024-05-22T23:59:30.090145+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:38:27.659025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":244946,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67252,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":558,"num_read_kb":350,"num_write":173,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1012,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":201756,"data_compressed":5079,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318311,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":508368,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579948,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32308,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712844,"statfs":{"total":95995035648,"available":95961952256,"internally_reserved":0,"allocated":1159168,"data_stored":1158747,"data_compressed":57992,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743473,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":512700,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005111,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":508524,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266750,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1105104,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430274,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":503780,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":566270,"data_compressed":22296,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738468,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":505168,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411418,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28084,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717068,"statfs":{"total":95995035648,"available":95966277632,"internally_reserved":0,"allocated":1093632,"data_stored":566998,"data_compressed":22296,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149780,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":503974,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888143,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":510114,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691915,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1100161,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34054,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":62580,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63308,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64689,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-22T23:59:37.430 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-22T23:59:37.430 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-22T23:59:37.430 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-22T23:59:37.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph health --format=json 2024-05-22T23:59:37.539 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:37 smithi103 ceph-mon[31370]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:59:37.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:37 smithi105 ceph-mon[36426]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:59:37.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:37 smithi174 ceph-mon[36179]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-22T23:59:37.903 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:38.307 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:38 smithi103 ceph-mon[31370]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:38.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:38 smithi105 ceph-mon[36426]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:38.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:38 smithi174 ceph-mon[36179]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.093+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2743093658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28074590 msgr2=0x7f9a28074970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.093+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2743093658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28074590 0x7f9a28074970 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9a180098f0 tx=0x7f9a18009c00 comp rx=0 tx=0).stop 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.093+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2743093658 shutdown_connections 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.093+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2743093658 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a28113530 0x7f9a28115920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.093+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2743093658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28074590 0x7f9a28074970 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.093+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2743093658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28074eb0 0x7f9a28112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.093+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2743093658 >> 172.21.15.103:0/2743093658 conn(0x7f9a280794c0 msgr2=0x7f9a280798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.093+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2743093658 shutdown_connections 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.094+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2743093658 wait complete. 2024-05-22T23:59:40.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.094+0000 7f9a2ec00700 1 Processor -- start 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a2ec00700 1 -- start start 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a2ec00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28074590 0x7f9a2819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a2ec00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a28074eb0 0x7f9a2819f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a2ec00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28113530 0x7f9a281a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a2ec00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a28118ce0 con 0x7f9a28074590 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a2ec00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a28118b60 con 0x7f9a28074eb0 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a2ec00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a28118e60 con 0x7f9a28113530 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a27e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28074590 0x7f9a2819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:40.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a27e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28074590 0x7f9a2819f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48634/0 (socket says 172.21.15.103:48634) 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a27e00700 1 -- 172.21.15.103:0/2141956714 learned_addr learned my addr 172.21.15.103:0/2141956714 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a2cc00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28113530 0x7f9a281a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.095+0000 7f9a27400700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a28074eb0 0x7f9a2819f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.096+0000 7f9a2cc00700 1 -- 172.21.15.103:0/2141956714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a28074eb0 msgr2=0x7f9a2819f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.096+0000 7f9a2cc00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a28074eb0 0x7f9a2819f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.096+0000 7f9a2cc00700 1 -- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28074590 msgr2=0x7f9a2819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.096+0000 7f9a2cc00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28074590 0x7f9a2819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.096+0000 7f9a2cc00700 1 -- 172.21.15.103:0/2141956714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a281a41a0 con 0x7f9a28113530 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.096+0000 7f9a2cc00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28113530 0x7f9a281a3a60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000ef90 tx=0x7f9a2000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.096+0000 7f9a24c00700 1 -- 172.21.15.103:0/2141956714 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a200092e0 con 0x7f9a28113530 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.096+0000 7f9a24c00700 1 -- 172.21.15.103:0/2141956714 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a20004bb0 con 0x7f9a28113530 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.097+0000 7f9a24c00700 1 -- 172.21.15.103:0/2141956714 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a20004280 con 0x7f9a28113530 2024-05-22T23:59:40.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.097+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a281a4430 con 0x7f9a28113530 2024-05-22T23:59:40.102 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.097+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a281abd10 con 0x7f9a28113530 2024-05-22T23:59:40.102 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.098+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a28069050 con 0x7f9a28113530 2024-05-22T23:59:40.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.100+0000 7f9a24c00700 1 -- 172.21.15.103:0/2141956714 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f9a20044d10 con 0x7f9a28113530 2024-05-22T23:59:40.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.101+0000 7f9a24c00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9a0c0a2ca0 0x7f9a0c0a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:40.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.101+0000 7f9a24c00700 1 -- 172.21.15.103:0/2141956714 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9a200cf320 con 0x7f9a28113530 2024-05-22T23:59:40.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.102+0000 7f9a27e00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9a0c0a2ca0 0x7f9a0c0a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:40.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.102+0000 7f9a27e00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9a0c0a2ca0 0x7f9a0c0a5160 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800c010 tx=0x7f9a18019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:40.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.104+0000 7f9a24c00700 1 -- 172.21.15.103:0/2141956714 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9a20022020 con 0x7f9a28113530 2024-05-22T23:59:40.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.383+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f9a281abee0 con 0x7f9a28113530 2024-05-22T23:59:40.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.383+0000 7f9a24c00700 1 -- 172.21.15.103:0/2141956714 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f9a20010040 con 0x7f9a28113530 2024-05-22T23:59:40.387 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:40.388 INFO:teuthology.orchestra.run.smithi103.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-22T23:59:40.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.386+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9a0c0a2ca0 msgr2=0x7f9a0c0a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:40.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.386+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9a0c0a2ca0 0x7f9a0c0a5160 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a1800c010 tx=0x7f9a18019040 comp rx=0 tx=0).stop 2024-05-22T23:59:40.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28113530 msgr2=0x7f9a281a3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28113530 0x7f9a281a3a60 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9a2000ef90 tx=0x7f9a2000c3a0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 shutdown_connections 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9a0c0a2ca0 0x7f9a0c0a5160 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a28113530 0x7f9a281a3a60 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a28074590 0x7f9a2819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 --2- 172.21.15.103:0/2141956714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a28074eb0 0x7f9a2819f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 >> 172.21.15.103:0/2141956714 conn(0x7f9a280794c0 msgr2=0x7f9a28112360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 shutdown_connections 2024-05-22T23:59:40.391 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:40.387+0000 7f9a2ec00700 1 -- 172.21.15.103:0/2141956714 wait complete. 2024-05-22T23:59:40.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:40 smithi105 ceph-mon[36426]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:40.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:40 smithi103 ceph-mon[31370]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:40.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:40 smithi174 ceph-mon[36179]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:41.209 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-22T23:59:41.210 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:59:41.427 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 -- 172.21.15.103:0/1878016785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c104ae0 msgr2=0x7f5d7c104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 --2- 172.21.15.103:0/1878016785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c104ae0 0x7f5d7c104f60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009860 tx=0x7f5d6802a340 comp rx=0 tx=0).stop 2024-05-22T23:59:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 -- 172.21.15.103:0/1878016785 shutdown_connections 2024-05-22T23:59:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 --2- 172.21.15.103:0/1878016785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d7c1054a0 0x7f5d7c10bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 --2- 172.21.15.103:0/1878016785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c104ae0 0x7f5d7c104f60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 --2- 172.21.15.103:0/1878016785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d7c1038f0 0x7f5d7c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 -- 172.21.15.103:0/1878016785 >> 172.21.15.103:0/1878016785 conn(0x7f5d7c0fee80 msgr2=0x7f5d7c1012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 -- 172.21.15.103:0/1878016785 shutdown_connections 2024-05-22T23:59:41.428 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.424+0000 7f5d80c00640 1 -- 172.21.15.103:0/1878016785 wait complete. 2024-05-22T23:59:41.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.425+0000 7f5d80c00640 1 Processor -- start 2024-05-22T23:59:41.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.425+0000 7f5d80c00640 1 -- start start 2024-05-22T23:59:41.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.425+0000 7f5d80c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d7c1038f0 0x7f5d7c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.425+0000 7f5d80c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d7c104ae0 0x7f5d7c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.425+0000 7f5d80c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c1054a0 0x7f5d7c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.425+0000 7f5d80c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c10f570 con 0x7f5d7c1038f0 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.425+0000 7f5d80c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c10f3f0 con 0x7f5d7c1054a0 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.425+0000 7f5d80c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7c10f6f0 con 0x7f5d7c104ae0 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d7c1038f0 0x7f5d7c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d7c1038f0 0x7f5d7c199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48658/0 (socket says 172.21.15.103:48658) 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7b400640 1 -- 172.21.15.103:0/3545043414 learned_addr learned my addr 172.21.15.103:0/3545043414 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7be00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c1054a0 0x7f5d7c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7aa00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d7c104ae0 0x7f5d7c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:41.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7be00640 1 -- 172.21.15.103:0/3545043414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d7c104ae0 msgr2=0x7f5d7c19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7be00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d7c104ae0 0x7f5d7c19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7be00640 1 -- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d7c1038f0 msgr2=0x7f5d7c199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7be00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d7c1038f0 0x7f5d7c199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7be00640 1 -- 172.21.15.103:0/3545043414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d7c1a1800 con 0x7f5d7c1054a0 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7b400640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d7c1038f0 0x7f5d7c199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.426+0000 7f5d7be00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c1054a0 0x7f5d7c1a1190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009fd0 tx=0x7f5d7400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.427+0000 7f5d73e00640 1 -- 172.21.15.103:0/3545043414 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d7400ce20 con 0x7f5d7c1054a0 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.427+0000 7f5d73e00640 1 -- 172.21.15.103:0/3545043414 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d74005020 con 0x7f5d7c1054a0 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.427+0000 7f5d73e00640 1 -- 172.21.15.103:0/3545043414 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d74010720 con 0x7f5d7c1054a0 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.427+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d7c1a1af0 con 0x7f5d7c1054a0 2024-05-22T23:59:41.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.427+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d7c1a2010 con 0x7f5d7c1054a0 2024-05-22T23:59:41.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.428+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7c10ac10 con 0x7f5d7c1054a0 2024-05-22T23:59:41.433 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.430+0000 7f5d73e00640 1 -- 172.21.15.103:0/3545043414 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f5d74005540 con 0x7f5d7c1054a0 2024-05-22T23:59:41.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.431+0000 7f5d73e00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5d600a2b90 0x7f5d600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:41.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.431+0000 7f5d73e00640 1 -- 172.21.15.103:0/3545043414 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f5d740cf9c0 con 0x7f5d7c1054a0 2024-05-22T23:59:41.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.431+0000 7f5d7b400640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5d600a2b90 0x7f5d600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:41.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.432+0000 7f5d7b400640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5d600a2b90 0x7f5d600a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0096c0 tx=0x7f5d6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:41.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.433+0000 7f5d73e00640 1 -- 172.21.15.103:0/3545043414 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f5d7409c560 con 0x7f5d7c1054a0 2024-05-22T23:59:41.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:41 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2141956714' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-22T23:59:41.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:41 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2141956714' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-22T23:59:41.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:41 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2141956714' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-22T23:59:41.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.636+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d7c02d3d0 con 0x7f5d7c1054a0 2024-05-22T23:59:41.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.639+0000 7f5d73e00640 1 -- 172.21.15.103:0/3545043414 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24658 (secure 0 0 0) 0x7f5d7409be60 con 0x7f5d7c1054a0 2024-05-22T23:59:41.643 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:41.643 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:58:31.088381+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:59:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5d600a2b90 msgr2=0x7f5d600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5d600a2b90 0x7f5d600a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c0096c0 tx=0x7f5d6c009260 comp rx=0 tx=0).stop 2024-05-22T23:59:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c1054a0 msgr2=0x7f5d7c1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c1054a0 0x7f5d7c1a1190 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5d74009fd0 tx=0x7f5d7400ef10 comp rx=0 tx=0).stop 2024-05-22T23:59:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 shutdown_connections 2024-05-22T23:59:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f5d600a2b90 0x7f5d600a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d7c1054a0 0x7f5d7c1a1190 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5d7c104ae0 0x7f5d7c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 --2- 172.21.15.103:0/3545043414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5d7c1038f0 0x7f5d7c199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.641+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 >> 172.21.15.103:0/3545043414 conn(0x7f5d7c0fee80 msgr2=0x7f5d7c107de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.642+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 shutdown_connections 2024-05-22T23:59:41.646 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.642+0000 7f5d80c00640 1 -- 172.21.15.103:0/3545043414 wait complete. 2024-05-22T23:59:41.666 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-22T23:59:41.667 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-22T23:59:41.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.862+0000 7f0692800640 1 -- 172.21.15.103:0/1619726855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f068c107db0 msgr2=0x7f068c1081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.862+0000 7f0692800640 1 --2- 172.21.15.103:0/1619726855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f068c107db0 0x7f068c1081d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0680009ca0 tx=0x7f068002a8b0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.863+0000 7f0692800640 1 -- 172.21.15.103:0/1619726855 shutdown_connections 2024-05-22T23:59:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.863+0000 7f0692800640 1 --2- 172.21.15.103:0/1619726855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f068c109960 0x7f068c1101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.863+0000 7f0692800640 1 --2- 172.21.15.103:0/1619726855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f068c108fa0 0x7f068c109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.863+0000 7f0692800640 1 --2- 172.21.15.103:0/1619726855 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f068c107db0 0x7f068c1081d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.863+0000 7f0692800640 1 -- 172.21.15.103:0/1619726855 >> 172.21.15.103:0/1619726855 conn(0x7f068c076d40 msgr2=0x7f068c079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.863+0000 7f0692800640 1 -- 172.21.15.103:0/1619726855 shutdown_connections 2024-05-22T23:59:41.867 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.863+0000 7f0692800640 1 -- 172.21.15.103:0/1619726855 wait complete. 2024-05-22T23:59:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0692800640 1 Processor -- start 2024-05-22T23:59:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0692800640 1 -- start start 2024-05-22T23:59:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0692800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f068c107db0 0x7f068c19e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0692800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f068c108fa0 0x7f068c19e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:41.870 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0692800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f068c109960 0x7f068c1a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0692800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f068c113a30 con 0x7f068c109960 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0692800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1138b0 con 0x7f068c108fa0 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0692800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f068c113bb0 con 0x7f068c107db0 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0690a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f068c108fa0 0x7f068c19e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0691e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f068c109960 0x7f068c1a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0690a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f068c108fa0 0x7f068c19e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:47820/0 (socket says 172.21.15.103:47820) 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.864+0000 7f0690a00640 1 -- 172.21.15.103:0/4023110692 learned_addr learned my addr 172.21.15.103:0/4023110692 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f0691400640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f068c107db0 0x7f068c19e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f0690a00640 1 -- 172.21.15.103:0/4023110692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f068c107db0 msgr2=0x7f068c19e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f0690a00640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f068c107db0 0x7f068c19e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f0690a00640 1 -- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f068c109960 msgr2=0x7f068c1a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f0690a00640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f068c109960 0x7f068c1a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f0690a00640 1 -- 172.21.15.103:0/4023110692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c1a5ca0 con 0x7f068c108fa0 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f0691e00640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f068c109960 0x7f068c1a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f0690a00640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f068c108fa0 0x7f068c19e610 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f067800ea60 tx=0x7f067800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.865+0000 7f068a000640 1 -- 172.21.15.103:0/4023110692 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067800cde0 con 0x7f068c108fa0 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.866+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c1a5f90 con 0x7f068c108fa0 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.866+0000 7f068a000640 1 -- 172.21.15.103:0/4023110692 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06780035f0 con 0x7f068c108fa0 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.866+0000 7f068a000640 1 -- 172.21.15.103:0/4023110692 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0678010670 con 0x7f068c108fa0 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.866+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c1a64a0 con 0x7f068c108fa0 2024-05-22T23:59:41.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.867+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068c10f070 con 0x7f068c108fa0 2024-05-22T23:59:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.869+0000 7f068a000640 1 -- 172.21.15.103:0/4023110692 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f06780037a0 con 0x7f068c108fa0 2024-05-22T23:59:41.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.870+0000 7f068a000640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06740a2b90 0x7f06740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.870+0000 7f068a000640 1 -- 172.21.15.103:0/4023110692 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f06780cf750 con 0x7f068c108fa0 2024-05-22T23:59:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.871+0000 7f0691400640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06740a2b90 0x7f06740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:41.874 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.871+0000 7f0691400640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06740a2b90 0x7f06740a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0680000f80 tx=0x7f0680030fe0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:41.875 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:41.872+0000 7f068a000640 1 -- 172.21.15.103:0/4023110692 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f067809c380 con 0x7f068c108fa0 2024-05-22T23:59:42.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.076+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f068c02d400 con 0x7f068c108fa0 2024-05-22T23:59:42.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.077+0000 7f068a000640 1 -- 172.21.15.103:0/4023110692 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f067809bc80 con 0x7f068c108fa0 2024-05-22T23:59:42.080 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-22T23:59:42.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.079+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06740a2b90 msgr2=0x7f06740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06740a2b90 0x7f06740a5050 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0680000f80 tx=0x7f0680030fe0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f068c108fa0 msgr2=0x7f068c19e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f068c108fa0 0x7f068c19e610 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f067800ea60 tx=0x7f067800ef20 comp rx=0 tx=0).stop 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 shutdown_connections 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06740a2b90 0x7f06740a5050 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f068c109960 0x7f068c1a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f068c108fa0 0x7f068c19e610 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 --2- 172.21.15.103:0/4023110692 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f068c107db0 0x7f068c19e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 >> 172.21.15.103:0/4023110692 conn(0x7f068c076d40 msgr2=0x7f068c10c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 shutdown_connections 2024-05-22T23:59:42.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.080+0000 7f0692800640 1 -- 172.21.15.103:0/4023110692 wait complete. 2024-05-22T23:59:42.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-22T23:59:42.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 -- 172.21.15.103:0/2861932603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a0104ce0 msgr2=0x7f89a01070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 --2- 172.21.15.103:0/2861932603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a0104ce0 0x7f89a01070d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f898c009900 tx=0x7f898c02a3f0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 -- 172.21.15.103:0/2861932603 shutdown_connections 2024-05-22T23:59:42.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 --2- 172.21.15.103:0/2861932603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0107610 0x7f89a0109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 --2- 172.21.15.103:0/2861932603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a0104ce0 0x7f89a01070d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 --2- 172.21.15.103:0/2861932603 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a01023b0 0x7f89a01047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 -- 172.21.15.103:0/2861932603 >> 172.21.15.103:0/2861932603 conn(0x7f89a00fbd00 msgr2=0x7f89a00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:42.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 -- 172.21.15.103:0/2861932603 shutdown_connections 2024-05-22T23:59:42.316 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.311+0000 7f89a8000640 1 -- 172.21.15.103:0/2861932603 wait complete. 2024-05-22T23:59:42.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.312+0000 7f89a8000640 1 Processor -- start 2024-05-22T23:59:42.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.312+0000 7f89a8000640 1 -- start start 2024-05-22T23:59:42.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a8000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01023b0 0x7f89a0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:42.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a8000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0104ce0 0x7f89a01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:42.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a8000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107610 0x7f89a019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:42.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a8000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010d530 con 0x7f89a01023b0 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a8000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010d3b0 con 0x7f89a0107610 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a8000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89a010d6b0 con 0x7f89a0104ce0 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a4c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0104ce0 0x7f89a01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a4c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0104ce0 0x7f89a01980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:46196/0 (socket says 172.21.15.103:46196) 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a4c00640 1 -- 172.21.15.103:0/2842128374 learned_addr learned my addr 172.21.15.103:0/2842128374 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a5600640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01023b0 0x7f89a0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.313+0000 7f89a6000640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107610 0x7f89a019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f89a4c00640 1 -- 172.21.15.103:0/2842128374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107610 msgr2=0x7f89a019f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f89a4c00640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107610 0x7f89a019f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f89a4c00640 1 -- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01023b0 msgr2=0x7f89a0197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f89a4c00640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01023b0 0x7f89a0197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f89a4c00640 1 -- 172.21.15.103:0/2842128374 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89a019f7d0 con 0x7f89a0104ce0 2024-05-22T23:59:42.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f89a4c00640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0104ce0 0x7f89a01980f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f898c031440 tx=0x7f898c032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:42.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f899e000640 1 -- 172.21.15.103:0/2842128374 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c03f030 con 0x7f89a0104ce0 2024-05-22T23:59:42.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f899e000640 1 -- 172.21.15.103:0/2842128374 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f898c003ce0 con 0x7f89a0104ce0 2024-05-22T23:59:42.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f899e000640 1 -- 172.21.15.103:0/2842128374 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898c035070 con 0x7f89a0104ce0 2024-05-22T23:59:42.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89a019fa60 con 0x7f89a0104ce0 2024-05-22T23:59:42.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.314+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89a019ff50 con 0x7f89a0104ce0 2024-05-22T23:59:42.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.316+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89a00fd7f0 con 0x7f89a0104ce0 2024-05-22T23:59:42.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.317+0000 7f899e000640 1 -- 172.21.15.103:0/2842128374 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f898c042030 con 0x7f89a0104ce0 2024-05-22T23:59:42.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.319+0000 7f899e000640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89880a2b90 0x7f89880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:42.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.319+0000 7f899e000640 1 -- 172.21.15.103:0/2842128374 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f898c0ef5a0 con 0x7f89a0104ce0 2024-05-22T23:59:42.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.319+0000 7f89a5600640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89880a2b90 0x7f89880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:42.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.319+0000 7f89a5600640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89880a2b90 0x7f89880a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8994009bd0 tx=0x7f8994005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:42.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.323+0000 7f899e000640 1 -- 172.21.15.103:0/2842128374 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f898c0bbaf0 con 0x7f89a0104ce0 2024-05-22T23:59:42.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.523+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f89a0062470 con 0x7f89a0104ce0 2024-05-22T23:59:42.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.524+0000 7f899e000640 1 -- 172.21.15.103:0/2842128374 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f898c0bfdf0 con 0x7f89a0104ce0 2024-05-22T23:59:42.528 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-22T23:59:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89880a2b90 msgr2=0x7f89880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89880a2b90 0x7f89880a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8994009bd0 tx=0x7f8994005f90 comp rx=0 tx=0).stop 2024-05-22T23:59:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0104ce0 msgr2=0x7f89a01980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0104ce0 0x7f89a01980f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f898c031440 tx=0x7f898c032ce0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 shutdown_connections 2024-05-22T23:59:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f89880a2b90 0x7f89880a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f89a0107610 0x7f89a019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89a0104ce0 0x7f89a01980f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 --2- 172.21.15.103:0/2842128374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89a01023b0 0x7f89a0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 >> 172.21.15.103:0/2842128374 conn(0x7f89a00fbd00 msgr2=0x7f89a0102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 shutdown_connections 2024-05-22T23:59:42.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.527+0000 7f89a8000640 1 -- 172.21.15.103:0/2842128374 wait complete. 2024-05-22T23:59:42.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:42 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3545043414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:42.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:42 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/4023110692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:59:42.547 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:42 smithi103 ceph-mon[31370]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-22T23:59:42.550 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-22T23:59:42.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:42 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3545043414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:42.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:42 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/4023110692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:59:42.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:42 smithi105 ceph-mon[36426]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-22T23:59:42.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:42 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3545043414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:42.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:42 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/4023110692' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:59:42.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:42 smithi174 ceph-mon[36179]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-22T23:59:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 -- 172.21.15.103:0/1694360821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa728109660 msgr2=0x7fa72810baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 --2- 172.21.15.103:0/1694360821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa728109660 0x7fa72810baf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa720009be0 tx=0x7fa72002a6c0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 -- 172.21.15.103:0/1694360821 shutdown_connections 2024-05-22T23:59:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 --2- 172.21.15.103:0/1694360821 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa728109660 0x7fa72810baf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 --2- 172.21.15.103:0/1694360821 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7280742d0 0x7fa728074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 --2- 172.21.15.103:0/1694360821 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7280758b0 0x7fa728073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 -- 172.21.15.103:0/1694360821 >> 172.21.15.103:0/1694360821 conn(0x7fa7280fcd10 msgr2=0x7fa7280ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:42.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 -- 172.21.15.103:0/1694360821 shutdown_connections 2024-05-22T23:59:42.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.753+0000 7fa727200640 1 -- 172.21.15.103:0/1694360821 wait complete. 2024-05-22T23:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.754+0000 7fa727200640 1 Processor -- start 2024-05-22T23:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.754+0000 7fa727200640 1 -- start start 2024-05-22T23:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.754+0000 7fa727200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7280742d0 0x7fa728199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.754+0000 7fa727200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7280758b0 0x7fa72819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.754+0000 7fa727200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa728109660 0x7fa7281a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa727200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa72810f600 con 0x7fa7280742d0 2024-05-22T23:59:42.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa727200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa72810f480 con 0x7fa7280758b0 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa727200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa72810f780 con 0x7fa728109660 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7280742d0 0x7fa728199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7280742d0 0x7fa728199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48712/0 (socket says 172.21.15.103:48712) 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725e00640 1 -- 172.21.15.103:0/3339627793 learned_addr learned my addr 172.21.15.103:0/3339627793 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa726800640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa728109660 0x7fa7281a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725400640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7280758b0 0x7fa72819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725e00640 1 -- 172.21.15.103:0/3339627793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa728109660 msgr2=0x7fa7281a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725e00640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa728109660 0x7fa7281a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725e00640 1 -- 172.21.15.103:0/3339627793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7280758b0 msgr2=0x7fa72819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725e00640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7280758b0 0x7fa72819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.755+0000 7fa725e00640 1 -- 172.21.15.103:0/3339627793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7281a1840 con 0x7fa7280742d0 2024-05-22T23:59:42.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.756+0000 7fa725e00640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7280742d0 0x7fa728199c30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa71800e9c0 tx=0x7fa71800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:42.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.756+0000 7fa71ea00640 1 -- 172.21.15.103:0/3339627793 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa71800ccc0 con 0x7fa7280742d0 2024-05-22T23:59:42.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.756+0000 7fa71ea00640 1 -- 172.21.15.103:0/3339627793 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa718005500 con 0x7fa7280742d0 2024-05-22T23:59:42.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.756+0000 7fa71ea00640 1 -- 172.21.15.103:0/3339627793 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa718010430 con 0x7fa7280742d0 2024-05-22T23:59:42.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.756+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7281a1b30 con 0x7fa7280742d0 2024-05-22T23:59:42.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.756+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7281a2030 con 0x7fa7280742d0 2024-05-22T23:59:42.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.757+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa728108390 con 0x7fa7280742d0 2024-05-22T23:59:42.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.759+0000 7fa71ea00640 1 -- 172.21.15.103:0/3339627793 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa7180046c0 con 0x7fa7280742d0 2024-05-22T23:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.760+0000 7fa71ea00640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa70c0a2c10 0x7fa70c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.761+0000 7fa71ea00640 1 -- 172.21.15.103:0/3339627793 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa7180cf6e0 con 0x7fa7280742d0 2024-05-22T23:59:42.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.761+0000 7fa725400640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa70c0a2c10 0x7fa70c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:42.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.761+0000 7fa725400640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa70c0a2c10 0x7fa70c0a50d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa714009bd0 tx=0x7fa714005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:42.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.765+0000 7fa71ea00640 1 -- 172.21.15.103:0/3339627793 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa71809c280 con 0x7fa7280742d0 2024-05-22T23:59:42.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.965+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa728062510 con 0x7fa7280742d0 2024-05-22T23:59:42.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.966+0000 7fa71ea00640 1 -- 172.21.15.103:0/3339627793 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fa71809bb80 con 0x7fa7280742d0 2024-05-22T23:59:42.969 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-22T23:59:42.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.968+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa70c0a2c10 msgr2=0x7fa70c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.968+0000 7fa727200640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa70c0a2c10 0x7fa70c0a50d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa714009bd0 tx=0x7fa714005f90 comp rx=0 tx=0).stop 2024-05-22T23:59:42.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7280742d0 msgr2=0x7fa728199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7280742d0 0x7fa728199c30 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa71800e9c0 tx=0x7fa71800ee80 comp rx=0 tx=0).stop 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 shutdown_connections 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa70c0a2c10 0x7fa70c0a50d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa728109660 0x7fa7281a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7280758b0 0x7fa72819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 --2- 172.21.15.103:0/3339627793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7280742d0 0x7fa728199c30 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 >> 172.21.15.103:0/3339627793 conn(0x7fa7280fcd10 msgr2=0x7fa728105560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 shutdown_connections 2024-05-22T23:59:42.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:42.969+0000 7fa727200640 1 -- 172.21.15.103:0/3339627793 wait complete. 2024-05-22T23:59:42.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 -- 172.21.15.103:0/10276011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0742d0 msgr2=0x7f919c074750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 --2- 172.21.15.103:0/10276011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0742d0 0x7f919c074750 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9184009900 tx=0x7f918402a3f0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 -- 172.21.15.103:0/10276011 shutdown_connections 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 --2- 172.21.15.103:0/10276011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f919c109760 0x7f919c10bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 --2- 172.21.15.103:0/10276011 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0742d0 0x7f919c074750 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 --2- 172.21.15.103:0/10276011 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f919c0758b0 0x7f919c073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 -- 172.21.15.103:0/10276011 >> 172.21.15.103:0/10276011 conn(0x7f919c0fce30 msgr2=0x7f919c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 -- 172.21.15.103:0/10276011 shutdown_connections 2024-05-22T23:59:43.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.189+0000 7f91a3a00640 1 -- 172.21.15.103:0/10276011 wait complete. 2024-05-22T23:59:43.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.190+0000 7f91a3a00640 1 Processor -- start 2024-05-22T23:59:43.195 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.190+0000 7f91a3a00640 1 -- start start 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.190+0000 7f91a3a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f919c0742d0 0x7f919c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.190+0000 7f91a3a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0758b0 0x7f919c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.190+0000 7f91a3a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f919c109760 0x7f919c1a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a3a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f919c10f730 con 0x7f919c0758b0 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a3a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f919c10f5b0 con 0x7f919c109760 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a3a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f919c10f8b0 con 0x7f919c0742d0 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f9193e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0758b0 0x7f919c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f9193e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0758b0 0x7f919c19a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48728/0 (socket says 172.21.15.103:48728) 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f9193e00640 1 -- 172.21.15.103:0/3574303335 learned_addr learned my addr 172.21.15.103:0/3574303335 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a1000640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f919c0742d0 0x7f919c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a1a00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f919c109760 0x7f919c1a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a1000640 1 -- 172.21.15.103:0/3574303335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f919c109760 msgr2=0x7f919c1a1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.196 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a1000640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f919c109760 0x7f919c1a1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a1000640 1 -- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0758b0 msgr2=0x7f919c19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a1000640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0758b0 0x7f919c19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.191+0000 7f91a1000640 1 -- 172.21.15.103:0/3574303335 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f919c1a19e0 con 0x7f919c0742d0 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.192+0000 7f91a1000640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f919c0742d0 0x7f919c199dc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f918c00e9f0 tx=0x7f918c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.192+0000 7f919a000640 1 -- 172.21.15.103:0/3574303335 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c00ccc0 con 0x7f919c0742d0 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.192+0000 7f919a000640 1 -- 172.21.15.103:0/3574303335 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f918c0035f0 con 0x7f919c0742d0 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.192+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919c1a1cd0 con 0x7f919c0742d0 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.192+0000 7f919a000640 1 -- 172.21.15.103:0/3574303335 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c00ce60 con 0x7f919c0742d0 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.192+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919c1a21d0 con 0x7f919c0742d0 2024-05-22T23:59:43.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.193+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f919c108490 con 0x7f919c0742d0 2024-05-22T23:59:43.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.195+0000 7f919a000640 1 -- 172.21.15.103:0/3574303335 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f918c0037a0 con 0x7f919c0742d0 2024-05-22T23:59:43.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.196+0000 7f919a000640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91800a2b90 0x7f91800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:43.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.197+0000 7f919a000640 1 -- 172.21.15.103:0/3574303335 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f918c0cf650 con 0x7f919c0742d0 2024-05-22T23:59:43.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.197+0000 7f9193e00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91800a2b90 0x7f91800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:43.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.198+0000 7f9193e00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91800a2b90 0x7f91800a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f919c19b2d0 tx=0x7f91840315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:43.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.201+0000 7f919a000640 1 -- 172.21.15.103:0/3574303335 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f918c09c1f0 con 0x7f919c0742d0 2024-05-22T23:59:43.239 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:43 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2842128374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:59:43.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.403+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f919c062510 con 0x7f919c0742d0 2024-05-22T23:59:43.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.404+0000 7f919a000640 1 -- 172.21.15.103:0/3574303335 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f918c09baf0 con 0x7f919c0742d0 2024-05-22T23:59:43.408 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-22T23:59:43.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91800a2b90 msgr2=0x7f91800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91800a2b90 0x7f91800a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f919c19b2d0 tx=0x7f91840315b0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f919c0742d0 msgr2=0x7f919c199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f919c0742d0 0x7f919c199dc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f918c00e9f0 tx=0x7f918c00edf0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 shutdown_connections 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f91800a2b90 0x7f91800a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f919c109760 0x7f919c1a1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f919c0758b0 0x7f919c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 --2- 172.21.15.103:0/3574303335 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f919c0742d0 0x7f919c199dc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 >> 172.21.15.103:0/3574303335 conn(0x7f919c0fce30 msgr2=0x7f919c105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 shutdown_connections 2024-05-22T23:59:43.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.407+0000 7f91a3a00640 1 -- 172.21.15.103:0/3574303335 wait complete. 2024-05-22T23:59:43.430 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:59:43.500 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:43 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3339627793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:59:43.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:43 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2842128374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:59:43.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:43 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3339627793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:59:43.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:43 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2842128374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:59:43.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:43 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3339627793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:59:43.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.631+0000 7fa9c5400640 1 -- 172.21.15.103:0/1842935758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 msgr2=0x7fa9c0108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.631+0000 7fa9c5400640 1 --2- 172.21.15.103:0/1842935758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 0x7fa9c0108110 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009ca0 tx=0x7fa9b802a8c0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.632+0000 7fa9c5400640 1 -- 172.21.15.103:0/1842935758 shutdown_connections 2024-05-22T23:59:43.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.632+0000 7fa9c5400640 1 --2- 172.21.15.103:0/1842935758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9c01098a0 0x7fa9c0110130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.632+0000 7fa9c5400640 1 --2- 172.21.15.103:0/1842935758 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9c0108ee0 0x7fa9c0109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.632+0000 7fa9c5400640 1 --2- 172.21.15.103:0/1842935758 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 0x7fa9c0108110 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.632+0000 7fa9c5400640 1 -- 172.21.15.103:0/1842935758 >> 172.21.15.103:0/1842935758 conn(0x7fa9c0076cb0 msgr2=0x7fa9c00790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:43.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.632+0000 7fa9c5400640 1 -- 172.21.15.103:0/1842935758 shutdown_connections 2024-05-22T23:59:43.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.632+0000 7fa9c5400640 1 -- 172.21.15.103:0/1842935758 wait complete. 2024-05-22T23:59:43.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.633+0000 7fa9c5400640 1 Processor -- start 2024-05-22T23:59:43.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.633+0000 7fa9c5400640 1 -- start start 2024-05-22T23:59:43.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.633+0000 7fa9c5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 0x7fa9c019e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:43.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.633+0000 7fa9c5400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9c0108ee0 0x7fa9c019e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:43.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9c5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9c01098a0 0x7fa9c01a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:43.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9c5400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0113970 con 0x7fa9c01098a0 2024-05-22T23:59:43.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9c5400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c01137f0 con 0x7fa9c0107cf0 2024-05-22T23:59:43.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9c5400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c0113af0 con 0x7fa9c0108ee0 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9c4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9c01098a0 0x7fa9c01a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9bfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 0x7fa9c019e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9bf400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9c0108ee0 0x7fa9c019e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9c4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9c01098a0 0x7fa9c01a55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48746/0 (socket says 172.21.15.103:48746) 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9c4a00640 1 -- 172.21.15.103:0/2975271164 learned_addr learned my addr 172.21.15.103:0/2975271164 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9bf400640 1 -- 172.21.15.103:0/2975271164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 msgr2=0x7fa9c019e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9bf400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 0x7fa9c019e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9bf400640 1 -- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9c01098a0 msgr2=0x7fa9c01a55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9bf400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9c01098a0 0x7fa9c01a55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9bf400640 1 -- 172.21.15.103:0/2975271164 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c01a5c40 con 0x7fa9c0108ee0 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.634+0000 7fa9bfe00640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 0x7fa9c019e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.635+0000 7fa9bf400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9c0108ee0 0x7fa9c019e560 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800e9f0 tx=0x7fa9a800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.635+0000 7fa9bcc00640 1 -- 172.21.15.103:0/2975271164 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a800ccc0 con 0x7fa9c0108ee0 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.635+0000 7fa9bcc00640 1 -- 172.21.15.103:0/2975271164 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9a8010910 con 0x7fa9c0108ee0 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.635+0000 7fa9bcc00640 1 -- 172.21.15.103:0/2975271164 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8018960 con 0x7fa9c0108ee0 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.635+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c01a5f30 con 0x7fa9c0108ee0 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.635+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c01a63a0 con 0x7fa9c0108ee0 2024-05-22T23:59:43.640 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.636+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c010f090 con 0x7fa9c0108ee0 2024-05-22T23:59:43.642 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.639+0000 7fa9bcc00640 1 -- 172.21.15.103:0/2975271164 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa9a8010ac0 con 0x7fa9c0108ee0 2024-05-22T23:59:43.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.640+0000 7fa9bcc00640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9a00a2b90 0x7fa9a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:43.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.640+0000 7fa9bcc00640 1 -- 172.21.15.103:0/2975271164 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa9a80cfc50 con 0x7fa9c0108ee0 2024-05-22T23:59:43.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.640+0000 7fa9bfe00640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9a00a2b90 0x7fa9a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:43.644 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.641+0000 7fa9bfe00640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9a00a2b90 0x7fa9a00a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8006010 tx=0x7fa9b8000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:43.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.642+0000 7fa9bcc00640 1 -- 172.21.15.103:0/2975271164 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa9a809c1a0 con 0x7fa9c0108ee0 2024-05-22T23:59:43.885 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.882+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9c002d200 con 0x7fa9c0108ee0 2024-05-22T23:59:43.886 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.884+0000 7fa9bcc00640 1 -- 172.21.15.103:0/2975271164 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11896 (secure 0 0 0) 0x7fa9a80a04a0 con 0x7fa9c0108ee0 2024-05-22T23:59:43.887 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:43.887 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-22T23:58:05.260333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24559,24563,24575]},"id":1}]} 2024-05-22T23:59:43.887 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 24 2024-05-22T23:59:43.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9a00a2b90 msgr2=0x7fa9a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9a00a2b90 0x7fa9a00a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8006010 tx=0x7fa9b8000f40 comp rx=0 tx=0).stop 2024-05-22T23:59:43.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9c0108ee0 msgr2=0x7fa9c019e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:43.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9c0108ee0 0x7fa9c019e560 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800e9f0 tx=0x7fa9a800eeb0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 shutdown_connections 2024-05-22T23:59:43.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9a00a2b90 0x7fa9a00a5050 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9c01098a0 0x7fa9c01a55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9c0108ee0 0x7fa9c019e560 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 --2- 172.21.15.103:0/2975271164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9c0107cf0 0x7fa9c019e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:43.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 >> 172.21.15.103:0/2975271164 conn(0x7fa9c0076cb0 msgr2=0x7fa9c010c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:43.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 shutdown_connections 2024-05-22T23:59:43.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:43.886+0000 7fa9c5400640 1 -- 172.21.15.103:0/2975271164 wait complete. 2024-05-22T23:59:43.909 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:59:44.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 -- 172.21.15.103:0/3660179312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f01098a0 msgr2=0x7fd6f0110130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 --2- 172.21.15.103:0/3660179312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f01098a0 0x7fd6f0110130 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fd6e8009ca0 tx=0x7fd6e802a8c0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 -- 172.21.15.103:0/3660179312 shutdown_connections 2024-05-22T23:59:44.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 --2- 172.21.15.103:0/3660179312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f01098a0 0x7fd6f0110130 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 --2- 172.21.15.103:0/3660179312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f0108ee0 0x7fd6f0109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 --2- 172.21.15.103:0/3660179312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6f0107cf0 0x7fd6f0108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 -- 172.21.15.103:0/3660179312 >> 172.21.15.103:0/3660179312 conn(0x7fd6f0076cb0 msgr2=0x7fd6f00790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:44.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 -- 172.21.15.103:0/3660179312 shutdown_connections 2024-05-22T23:59:44.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.109+0000 7fd6efe00640 1 -- 172.21.15.103:0/3660179312 wait complete. 2024-05-22T23:59:44.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.110+0000 7fd6efe00640 1 Processor -- start 2024-05-22T23:59:44.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.110+0000 7fd6efe00640 1 -- start start 2024-05-22T23:59:44.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.110+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0107cf0 0x7fd6f019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:44.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6f0108ee0 0x7fd6f019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:44.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6efe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f01098a0 0x7fd6f01a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6efe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f0113950 con 0x7fd6f0107cf0 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6efe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f01137d0 con 0x7fd6f0108ee0 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6efe00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f0113ad0 con 0x7fd6f01098a0 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6eea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0107cf0 0x7fd6f019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6eea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0107cf0 0x7fd6f019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34746/0 (socket says 172.21.15.103:34746) 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6eea00640 1 -- 172.21.15.103:0/623870784 learned_addr learned my addr 172.21.15.103:0/623870784 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6ef400640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f01098a0 0x7fd6f01a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6ee000640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6f0108ee0 0x7fd6f019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6eea00640 1 -- 172.21.15.103:0/623870784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f01098a0 msgr2=0x7fd6f01a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6eea00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f01098a0 0x7fd6f01a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6eea00640 1 -- 172.21.15.103:0/623870784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6f0108ee0 msgr2=0x7fd6f019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6eea00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6f0108ee0 0x7fd6f019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.111+0000 7fd6eea00640 1 -- 172.21.15.103:0/623870784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f01a5c20 con 0x7fd6f0107cf0 2024-05-22T23:59:44.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.112+0000 7fd6eea00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0107cf0 0x7fd6f019e000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9c0 tx=0x7fd6e000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:44.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.112+0000 7fd6e7400640 1 -- 172.21.15.103:0/623870784 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e000cde0 con 0x7fd6f0107cf0 2024-05-22T23:59:44.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.112+0000 7fd6e7400640 1 -- 172.21.15.103:0/623870784 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6e0010910 con 0x7fd6f0107cf0 2024-05-22T23:59:44.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.112+0000 7fd6e7400640 1 -- 172.21.15.103:0/623870784 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0018a00 con 0x7fd6f0107cf0 2024-05-22T23:59:44.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.112+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f01a5f10 con 0x7fd6f0107cf0 2024-05-22T23:59:44.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.112+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f01a6370 con 0x7fd6f0107cf0 2024-05-22T23:59:44.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.113+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f010efb0 con 0x7fd6f0107cf0 2024-05-22T23:59:44.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.115+0000 7fd6e7400640 1 -- 172.21.15.103:0/623870784 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd6e001f020 con 0x7fd6f0107cf0 2024-05-22T23:59:44.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.117+0000 7fd6e7400640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd6d40a2b90 0x7fd6d40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:44.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.118+0000 7fd6e7400640 1 -- 172.21.15.103:0/623870784 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fd6e00cf5f0 con 0x7fd6f0107cf0 2024-05-22T23:59:44.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.118+0000 7fd6ee000640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd6d40a2b90 0x7fd6d40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:44.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.118+0000 7fd6ee000640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd6d40a2b90 0x7fd6d40a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd6f019f510 tx=0x7fd6dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:44.126 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.122+0000 7fd6e7400640 1 -- 172.21.15.103:0/623870784 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd6e009c190 con 0x7fd6f0107cf0 2024-05-22T23:59:44.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.333+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd6f0062570 con 0x7fd6f0107cf0 2024-05-22T23:59:44.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.338+0000 7fd6e7400640 1 -- 172.21.15.103:0/623870784 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24658 (secure 0 0 0) 0x7fd6e009ba90 con 0x7fd6f0107cf0 2024-05-22T23:59:44.342 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:44.342 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:58:31.088381+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:59:44.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.340+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd6d40a2b90 msgr2=0x7fd6d40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.340+0000 7fd6efe00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd6d40a2b90 0x7fd6d40a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd6f019f510 tx=0x7fd6dc005f90 comp rx=0 tx=0).stop 2024-05-22T23:59:44.344 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0107cf0 msgr2=0x7fd6f019e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0107cf0 0x7fd6f019e000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd6e000e9c0 tx=0x7fd6e000ee80 comp rx=0 tx=0).stop 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 shutdown_connections 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd6d40a2b90 0x7fd6d40a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6f01098a0 0x7fd6f01a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6f0108ee0 0x7fd6f019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 --2- 172.21.15.103:0/623870784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd6f0107cf0 0x7fd6f019e000 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 >> 172.21.15.103:0/623870784 conn(0x7fd6f0076cb0 msgr2=0x7fd6f010c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 shutdown_connections 2024-05-22T23:59:44.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.341+0000 7fd6efe00640 1 -- 172.21.15.103:0/623870784 wait complete. 2024-05-22T23:59:44.364 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-22T23:59:44.364 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-22T23:59:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:44 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3574303335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:59:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:44 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2975271164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:44 smithi105 ceph-mon[36426]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:44.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.574+0000 7fe9afc00640 1 -- 172.21.15.103:0/4133428166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9a8077e10 msgr2=0x7fe9a80786b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.574+0000 7fe9afc00640 1 --2- 172.21.15.103:0/4133428166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9a8077e10 0x7fe9a80786b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0009ca0 tx=0x7fe9a002a8c0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.575+0000 7fe9afc00640 1 -- 172.21.15.103:0/4133428166 shutdown_connections 2024-05-22T23:59:44.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.575+0000 7fe9afc00640 1 --2- 172.21.15.103:0/4133428166 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9a8077e10 0x7fe9a80786b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.575+0000 7fe9afc00640 1 --2- 172.21.15.103:0/4133428166 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9a8077470 0x7fe9a80778d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.575+0000 7fe9afc00640 1 --2- 172.21.15.103:0/4133428166 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9a8076230 0x7fe9a8076650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.575+0000 7fe9afc00640 1 -- 172.21.15.103:0/4133428166 >> 172.21.15.103:0/4133428166 conn(0x7fe9a80ff060 msgr2=0x7fe9a81014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:44.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.575+0000 7fe9afc00640 1 -- 172.21.15.103:0/4133428166 shutdown_connections 2024-05-22T23:59:44.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.575+0000 7fe9afc00640 1 -- 172.21.15.103:0/4133428166 wait complete. 2024-05-22T23:59:44.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:44 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3574303335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:59:44.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:44 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2975271164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:44.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:44 smithi103 ceph-mon[31370]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.576+0000 7fe9afc00640 1 Processor -- start 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.576+0000 7fe9afc00640 1 -- start start 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9afc00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9a8076230 0x7fe9a819e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9afc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9a8077470 0x7fe9a819e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9afc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9a8077e10 0x7fe9a81a57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9afc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a8113b80 con 0x7fe9a8077470 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9afc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a8113a00 con 0x7fe9a8077e10 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9afc00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a8113d00 con 0x7fe9a8076230 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9a8077470 0x7fe9a819e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:44.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9ad200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9a8076230 0x7fe9a819e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9a7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9a8077470 0x7fe9a819e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34764/0 (socket says 172.21.15.103:34764) 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9a7e00640 1 -- 172.21.15.103:0/1876126154 learned_addr learned my addr 172.21.15.103:0/1876126154 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9adc00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9a8077e10 0x7fe9a81a57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9a7e00640 1 -- 172.21.15.103:0/1876126154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9a8076230 msgr2=0x7fe9a819e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9a7e00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9a8076230 0x7fe9a819e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9a7e00640 1 -- 172.21.15.103:0/1876126154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9a8077e10 msgr2=0x7fe9a81a57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9a7e00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9a8077e10 0x7fe9a81a57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9a7e00640 1 -- 172.21.15.103:0/1876126154 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a81a5e50 con 0x7fe9a8077470 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.577+0000 7fe9adc00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9a8077e10 0x7fe9a81a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.578+0000 7fe9a7e00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9a8077470 0x7fe9a819e770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe99400e9f0 tx=0x7fe99400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.578+0000 7fe9a5600640 1 -- 172.21.15.103:0/1876126154 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99400cd80 con 0x7fe9a8077470 2024-05-22T23:59:44.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.578+0000 7fe9a5600640 1 -- 172.21.15.103:0/1876126154 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe994005500 con 0x7fe9a8077470 2024-05-22T23:59:44.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.578+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a81a6140 con 0x7fe9a8077470 2024-05-22T23:59:44.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.578+0000 7fe9a5600640 1 -- 172.21.15.103:0/1876126154 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe994010690 con 0x7fe9a8077470 2024-05-22T23:59:44.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.578+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a81a6640 con 0x7fe9a8077470 2024-05-22T23:59:44.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.580+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a8110460 con 0x7fe9a8077470 2024-05-22T23:59:44.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.582+0000 7fe9a5600640 1 -- 172.21.15.103:0/1876126154 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe9940046c0 con 0x7fe9a8077470 2024-05-22T23:59:44.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.584+0000 7fe9a5600640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9900a2b90 0x7fe9900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:44.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.584+0000 7fe9a5600640 1 -- 172.21.15.103:0/1876126154 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe9940cf740 con 0x7fe9a8077470 2024-05-22T23:59:44.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.584+0000 7fe9ad200640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9900a2b90 0x7fe9900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:44.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.585+0000 7fe9ad200640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9900a2b90 0x7fe9900a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00bbe0 tx=0x7fe99c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.588+0000 7fe9a5600640 1 -- 172.21.15.103:0/1876126154 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe99409c340 con 0x7fe9a8077470 2024-05-22T23:59:44.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:44 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3574303335' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:59:44.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:44 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2975271164' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:44.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:44 smithi174 ceph-mon[36179]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-22T23:59:44.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.787+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe9a8062470 con 0x7fe9a8077470 2024-05-22T23:59:44.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.788+0000 7fe9a5600640 1 -- 172.21.15.103:0/1876126154 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fe99409bc40 con 0x7fe9a8077470 2024-05-22T23:59:44.792 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-22T23:59:44.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9900a2b90 msgr2=0x7fe9900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9900a2b90 0x7fe9900a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe99c00bbe0 tx=0x7fe99c00b390 comp rx=0 tx=0).stop 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9a8077470 msgr2=0x7fe9a819e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9a8077470 0x7fe9a819e770 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe99400e9f0 tx=0x7fe99400edf0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 shutdown_connections 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9900a2b90 0x7fe9900a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9a8077e10 0x7fe9a81a57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9a8077470 0x7fe9a819e770 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 --2- 172.21.15.103:0/1876126154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9a8076230 0x7fe9a819e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 >> 172.21.15.103:0/1876126154 conn(0x7fe9a80ff060 msgr2=0x7fe9a8100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:44.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 shutdown_connections 2024-05-22T23:59:44.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:44.791+0000 7fe9afc00640 1 -- 172.21.15.103:0/1876126154 wait complete. 2024-05-22T23:59:44.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-22T23:59:45.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 -- 172.21.15.103:0/3927106441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490109880 msgr2=0x7f2490110110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 --2- 172.21.15.103:0/3927106441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490109880 0x7f2490110110 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2488009ca0 tx=0x7f248802a8c0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 -- 172.21.15.103:0/3927106441 shutdown_connections 2024-05-22T23:59:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 --2- 172.21.15.103:0/3927106441 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490109880 0x7f2490110110 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 --2- 172.21.15.103:0/3927106441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2490108ec0 0x7f2490109340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 --2- 172.21.15.103:0/3927106441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2490107cd0 0x7f24901080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 -- 172.21.15.103:0/3927106441 >> 172.21.15.103:0/3927106441 conn(0x7f2490076c70 msgr2=0x7f24900790b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 -- 172.21.15.103:0/3927106441 shutdown_connections 2024-05-22T23:59:45.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.021+0000 7f2495600640 1 -- 172.21.15.103:0/3927106441 wait complete. 2024-05-22T23:59:45.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f2495600640 1 Processor -- start 2024-05-22T23:59:45.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f2495600640 1 -- start start 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f2495600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490107cd0 0x7f249019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f2495600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2490108ec0 0x7f249019e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f2495600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2490109880 0x7f24901a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f2495600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2490113950 con 0x7f2490108ec0 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f2495600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f24901137d0 con 0x7f2490107cd0 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f2495600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2490113ad0 con 0x7f2490109880 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f248fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490107cd0 0x7f249019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f248f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2490108ec0 0x7f249019e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.022+0000 7f248fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490107cd0 0x7f249019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48350/0 (socket says 172.21.15.103:48350) 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f248fe00640 1 -- 172.21.15.103:0/2961764957 learned_addr learned my addr 172.21.15.103:0/2961764957 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:45.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f2494c00640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2490109880 0x7f24901a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f248fe00640 1 -- 172.21.15.103:0/2961764957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2490109880 msgr2=0x7f24901a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f248fe00640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2490109880 0x7f24901a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f248fe00640 1 -- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2490108ec0 msgr2=0x7f249019e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f248fe00640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2490108ec0 0x7f249019e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f248fe00640 1 -- 172.21.15.103:0/2961764957 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24901a5b50 con 0x7f2490107cd0 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f2494c00640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2490109880 0x7f24901a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f248f400640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2490108ec0 0x7f249019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.023+0000 7f248fe00640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490107cd0 0x7f249019df30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f248000e9c0 tx=0x7f248000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.024+0000 7f248cc00640 1 -- 172.21.15.103:0/2961764957 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f248000cd80 con 0x7f2490107cd0 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.024+0000 7f248cc00640 1 -- 172.21.15.103:0/2961764957 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24800035f0 con 0x7f2490107cd0 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.024+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24901a5e40 con 0x7f2490107cd0 2024-05-22T23:59:45.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.024+0000 7f248cc00640 1 -- 172.21.15.103:0/2961764957 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2480010690 con 0x7f2490107cd0 2024-05-22T23:59:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.024+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24901a62a0 con 0x7f2490107cd0 2024-05-22T23:59:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.025+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f249010eff0 con 0x7f2490107cd0 2024-05-22T23:59:45.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.027+0000 7f248cc00640 1 -- 172.21.15.103:0/2961764957 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f24800037a0 con 0x7f2490107cd0 2024-05-22T23:59:45.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.028+0000 7f248cc00640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f24740a2c10 0x7f24740a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.028+0000 7f248cc00640 1 -- 172.21.15.103:0/2961764957 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f24800cf790 con 0x7f2490107cd0 2024-05-22T23:59:45.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.028+0000 7f248f400640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f24740a2c10 0x7f24740a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.029+0000 7f248f400640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f24740a2c10 0x7f24740a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f249019f440 tx=0x7f247c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:45.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.030+0000 7f248cc00640 1 -- 172.21.15.103:0/2961764957 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f248009c330 con 0x7f2490107cd0 2024-05-22T23:59:45.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.233+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f249002d4b0 con 0x7f2490107cd0 2024-05-22T23:59:45.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.234+0000 7f248cc00640 1 -- 172.21.15.103:0/2961764957 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f248009bc30 con 0x7f2490107cd0 2024-05-22T23:59:45.237 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-22T23:59:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f24740a2c10 msgr2=0x7f24740a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f24740a2c10 0x7f24740a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f249019f440 tx=0x7f247c005f90 comp rx=0 tx=0).stop 2024-05-22T23:59:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490107cd0 msgr2=0x7f249019df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490107cd0 0x7f249019df30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f248000e9c0 tx=0x7f248000ee80 comp rx=0 tx=0).stop 2024-05-22T23:59:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 shutdown_connections 2024-05-22T23:59:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f24740a2c10 0x7f24740a50d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2490109880 0x7f24901a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2490108ec0 0x7f249019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 --2- 172.21.15.103:0/2961764957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2490107cd0 0x7f249019df30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.236+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 >> 172.21.15.103:0/2961764957 conn(0x7f2490076c70 msgr2=0x7f249010c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.237+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 shutdown_connections 2024-05-22T23:59:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.237+0000 7f2495600640 1 -- 172.21.15.103:0/2961764957 wait complete. 2024-05-22T23:59:45.259 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-22T23:59:45.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.441+0000 7f259b200640 1 -- 172.21.15.103:0/1804157098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c107db0 msgr2=0x7f259c1081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.441+0000 7f259b200640 1 --2- 172.21.15.103:0/1804157098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c107db0 0x7f259c1081d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f258c009860 tx=0x7f258c02a340 comp rx=0 tx=0).stop 2024-05-22T23:59:45.445 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.441+0000 7f259b200640 1 -- 172.21.15.103:0/1804157098 shutdown_connections 2024-05-22T23:59:45.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.441+0000 7f259b200640 1 --2- 172.21.15.103:0/1804157098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f259c109960 0x7f259c1101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.441+0000 7f259b200640 1 --2- 172.21.15.103:0/1804157098 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c108fa0 0x7f259c109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.441+0000 7f259b200640 1 --2- 172.21.15.103:0/1804157098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c107db0 0x7f259c1081d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.441+0000 7f259b200640 1 -- 172.21.15.103:0/1804157098 >> 172.21.15.103:0/1804157098 conn(0x7f259c076d40 msgr2=0x7f259c079180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:45.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.442+0000 7f259b200640 1 -- 172.21.15.103:0/1804157098 shutdown_connections 2024-05-22T23:59:45.446 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.442+0000 7f259b200640 1 -- 172.21.15.103:0/1804157098 wait complete. 2024-05-22T23:59:45.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.443+0000 7f259b200640 1 Processor -- start 2024-05-22T23:59:45.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.443+0000 7f259b200640 1 -- start start 2024-05-22T23:59:45.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.443+0000 7f259b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c107db0 0x7f259c19e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.443+0000 7f259b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c108fa0 0x7f259c19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.443+0000 7f259b200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f259c109960 0x7f259c1a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.443+0000 7f259b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f259c113a00 con 0x7f259c107db0 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.443+0000 7f259b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f259c113880 con 0x7f259c108fa0 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.443+0000 7f259b200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f259c113b80 con 0x7f259c109960 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c107db0 0x7f259c19e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c107db0 0x7f259c19e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34802/0 (socket says 172.21.15.103:34802) 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 -- 172.21.15.103:0/3503284302 learned_addr learned my addr 172.21.15.103:0/3503284302 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2591400640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c108fa0 0x7f259c19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f259a800640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f259c109960 0x7f259c1a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 -- 172.21.15.103:0/3503284302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f259c109960 msgr2=0x7f259c1a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f259c109960 0x7f259c1a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 -- 172.21.15.103:0/3503284302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c108fa0 msgr2=0x7f259c19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c108fa0 0x7f259c19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 -- 172.21.15.103:0/3503284302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f259c1a5ce0 con 0x7f259c107db0 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2591400640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c108fa0 0x7f259c19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.444+0000 7f2599e00640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c107db0 0x7f259c19e0c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f258c0305b0 tx=0x7f258c033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.445+0000 7f2593400640 1 -- 172.21.15.103:0/3503284302 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c03f020 con 0x7f259c107db0 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.445+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259c1a5f70 con 0x7f259c107db0 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.445+0000 7f2593400640 1 -- 172.21.15.103:0/3503284302 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f258c030850 con 0x7f259c107db0 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.445+0000 7f2593400640 1 -- 172.21.15.103:0/3503284302 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c037040 con 0x7f259c107db0 2024-05-22T23:59:45.450 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.445+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f259c1a63d0 con 0x7f259c107db0 2024-05-22T23:59:45.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.446+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f259c10f0d0 con 0x7f259c107db0 2024-05-22T23:59:45.451 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.448+0000 7f2593400640 1 -- 172.21.15.103:0/3503284302 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f258c042020 con 0x7f259c107db0 2024-05-22T23:59:45.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.450+0000 7f2593400640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f25800a2b90 0x7f25800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.453 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.450+0000 7f2593400640 1 -- 172.21.15.103:0/3503284302 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f258c0ef880 con 0x7f259c107db0 2024-05-22T23:59:45.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.450+0000 7f2591400640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f25800a2b90 0x7f25800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.454 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.451+0000 7f2591400640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f25800a2b90 0x7f25800a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f259c19f5d0 tx=0x7f2584009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:45.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.455+0000 7f2593400640 1 -- 172.21.15.103:0/3503284302 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f258c049020 con 0x7f259c107db0 2024-05-22T23:59:45.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:45 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/623870784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:45.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:45 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1876126154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:59:45.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:45 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2961764957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:59:45.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:45 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/623870784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:45.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:45 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1876126154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:59:45.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:45 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2961764957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:59:45.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:45 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/623870784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:45.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:45 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1876126154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-22T23:59:45.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:45 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2961764957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-22T23:59:45.662 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.659+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f259c062510 con 0x7f259c107db0 2024-05-22T23:59:45.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.660+0000 7f2593400640 1 -- 172.21.15.103:0/3503284302 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f258c0bc420 con 0x7f259c107db0 2024-05-22T23:59:45.663 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.662+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f25800a2b90 msgr2=0x7f25800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.662+0000 7f259b200640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f25800a2b90 0x7f25800a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f259c19f5d0 tx=0x7f2584009260 comp rx=0 tx=0).stop 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.662+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c107db0 msgr2=0x7f259c19e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.662+0000 7f259b200640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c107db0 0x7f259c19e0c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f258c0305b0 tx=0x7f258c033d90 comp rx=0 tx=0).stop 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.663+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 shutdown_connections 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.663+0000 7f259b200640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f25800a2b90 0x7f25800a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.663+0000 7f259b200640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f259c109960 0x7f259c1a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.663+0000 7f259b200640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f259c108fa0 0x7f259c19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.663+0000 7f259b200640 1 --2- 172.21.15.103:0/3503284302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f259c107db0 0x7f259c19e0c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.663+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 >> 172.21.15.103:0/3503284302 conn(0x7f259c076d40 msgr2=0x7f259c10c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.663+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 shutdown_connections 2024-05-22T23:59:45.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.663+0000 7f259b200640 1 -- 172.21.15.103:0/3503284302 wait complete. 2024-05-22T23:59:45.686 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 -- 172.21.15.103:0/917174612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 msgr2=0x7fbc1c104f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 --2- 172.21.15.103:0/917174612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 0x7fbc1c104f40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009900 tx=0x7fbc0802a3f0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 -- 172.21.15.103:0/917174612 shutdown_connections 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 --2- 172.21.15.103:0/917174612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c105480 0x7fbc1c10bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 --2- 172.21.15.103:0/917174612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 0x7fbc1c104f40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 --2- 172.21.15.103:0/917174612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc1c1038d0 0x7fbc1c103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 -- 172.21.15.103:0/917174612 >> 172.21.15.103:0/917174612 conn(0x7fbc1c0fee40 msgr2=0x7fbc1c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 -- 172.21.15.103:0/917174612 shutdown_connections 2024-05-22T23:59:45.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.891+0000 7fbc22000640 1 -- 172.21.15.103:0/917174612 wait complete. 2024-05-22T23:59:45.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.892+0000 7fbc22000640 1 Processor -- start 2024-05-22T23:59:45.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.892+0000 7fbc22000640 1 -- start start 2024-05-22T23:59:45.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc22000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c1038d0 0x7fbc1c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc22000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 0x7fbc1c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc22000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc1c105480 0x7fbc1c1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc22000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c10f550 con 0x7fbc1c1038d0 2024-05-22T23:59:45.897 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc22000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c10f3d0 con 0x7fbc1c105480 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc22000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbc1c10f6d0 con 0x7fbc1c104ac0 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc20c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c1038d0 0x7fbc1c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc20c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c1038d0 0x7fbc1c199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34818/0 (socket says 172.21.15.103:34818) 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc20c00640 1 -- 172.21.15.103:0/3550459130 learned_addr learned my addr 172.21.15.103:0/3550459130 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc21600640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc1c105480 0x7fbc1c1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc1be00640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 0x7fbc1c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc20c00640 1 -- 172.21.15.103:0/3550459130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 msgr2=0x7fbc1c19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.893+0000 7fbc20c00640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 0x7fbc1c19a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc20c00640 1 -- 172.21.15.103:0/3550459130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc1c105480 msgr2=0x7fbc1c1a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc20c00640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc1c105480 0x7fbc1c1a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc20c00640 1 -- 172.21.15.103:0/3550459130 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c1a17f0 con 0x7fbc1c1038d0 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc1be00640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 0x7fbc1c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:59:45.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc20c00640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c1038d0 0x7fbc1c199bd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000e9c0 tx=0x7fbc1000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:45.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc19600640 1 -- 172.21.15.103:0/3550459130 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000ccc0 con 0x7fbc1c1038d0 2024-05-22T23:59:45.899 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc19600640 1 -- 172.21.15.103:0/3550459130 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc100035f0 con 0x7fbc1c1038d0 2024-05-22T23:59:45.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc1c1a1ae0 con 0x7fbc1c1038d0 2024-05-22T23:59:45.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc19600640 1 -- 172.21.15.103:0/3550459130 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1000ce60 con 0x7fbc1c1038d0 2024-05-22T23:59:45.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.894+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc1c1a2000 con 0x7fbc1c1038d0 2024-05-22T23:59:45.900 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.896+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc1c10acc0 con 0x7fbc1c1038d0 2024-05-22T23:59:45.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.898+0000 7fbc19600640 1 -- 172.21.15.103:0/3550459130 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbc100037a0 con 0x7fbc1c1038d0 2024-05-22T23:59:45.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.899+0000 7fbc19600640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc040a2c10 0x7fbc040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:45.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.899+0000 7fbc19600640 1 -- 172.21.15.103:0/3550459130 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fbc100cf780 con 0x7fbc1c1038d0 2024-05-22T23:59:45.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.899+0000 7fbc1be00640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc040a2c10 0x7fbc040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:45.903 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.900+0000 7fbc1be00640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc040a2c10 0x7fbc040a50d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbc08000c00 tx=0x7fbc080095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:45.906 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:45.903+0000 7fbc19600640 1 -- 172.21.15.103:0/3550459130 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbc1009c320 con 0x7fbc1c1038d0 2024-05-22T23:59:46.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.101+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbc1c062510 con 0x7fbc1c1038d0 2024-05-22T23:59:46.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.102+0000 7fbc19600640 1 -- 172.21.15.103:0/3550459130 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fbc100860a0 con 0x7fbc1c1038d0 2024-05-22T23:59:46.106 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-22T23:59:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.104+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc040a2c10 msgr2=0x7fbc040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.104+0000 7fbc22000640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc040a2c10 0x7fbc040a50d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbc08000c00 tx=0x7fbc080095f0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.104+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c1038d0 msgr2=0x7fbc1c199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.104+0000 7fbc22000640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c1038d0 0x7fbc1c199bd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000e9c0 tx=0x7fbc1000edc0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.105+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 shutdown_connections 2024-05-22T23:59:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.105+0000 7fbc22000640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbc040a2c10 0x7fbc040a50d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.105+0000 7fbc22000640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc1c105480 0x7fbc1c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.105+0000 7fbc22000640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc1c104ac0 0x7fbc1c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.105+0000 7fbc22000640 1 --2- 172.21.15.103:0/3550459130 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbc1c1038d0 0x7fbc1c199bd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.105+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 >> 172.21.15.103:0/3550459130 conn(0x7fbc1c0fee40 msgr2=0x7fbc1c107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.105+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 shutdown_connections 2024-05-22T23:59:46.109 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.105+0000 7fbc22000640 1 -- 172.21.15.103:0/3550459130 wait complete. 2024-05-22T23:59:46.129 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:59:46.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 -- 172.21.15.103:0/1483246482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6478109060 msgr2=0x7f64781094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 --2- 172.21.15.103:0/1483246482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6478109060 0x7f64781094e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6460009900 tx=0x7f646002a3f0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 -- 172.21.15.103:0/1483246482 shutdown_connections 2024-05-22T23:59:46.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 --2- 172.21.15.103:0/1483246482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6478109a20 0x7f6478110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 --2- 172.21.15.103:0/1483246482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6478109060 0x7f64781094e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 --2- 172.21.15.103:0/1483246482 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6478107e80 0x7f64781082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.325 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 -- 172.21.15.103:0/1483246482 >> 172.21.15.103:0/1483246482 conn(0x7f6478076e90 msgr2=0x7f64780792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:46.325 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 -- 172.21.15.103:0/1483246482 shutdown_connections 2024-05-22T23:59:46.325 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.320+0000 7f647e800640 1 -- 172.21.15.103:0/1483246482 wait complete. 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.321+0000 7f647e800640 1 Processor -- start 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.321+0000 7f647e800640 1 -- start start 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f647e800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6478107e80 0x7f647819e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f647e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6478109060 0x7f647819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f647e800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6478109a20 0x7f64781a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f647e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6478113b80 con 0x7f6478109060 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f647e800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6478113a00 con 0x7f6478107e80 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f647e800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6478113d00 con 0x7f6478109a20 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f646ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6478109060 0x7f647819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f6477400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6478107e80 0x7f647819e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f6477400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6478107e80 0x7f647819e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48420/0 (socket says 172.21.15.103:48420) 2024-05-22T23:59:46.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f6477400640 1 -- 172.21.15.103:0/1367994968 learned_addr learned my addr 172.21.15.103:0/1367994968 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.322+0000 7f6477e00640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6478109a20 0x7f64781a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6477400640 1 -- 172.21.15.103:0/1367994968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6478109a20 msgr2=0x7f64781a5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6477400640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6478109a20 0x7f64781a5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6477400640 1 -- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6478109060 msgr2=0x7f647819e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6477400640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6478109060 0x7f647819e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6477400640 1 -- 172.21.15.103:0/1367994968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64781a5dc0 con 0x7f6478107e80 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6477400640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6478107e80 0x7f647819e1a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f646800e9c0 tx=0x7f646800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6474c00640 1 -- 172.21.15.103:0/1367994968 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646800cd80 con 0x7f6478107e80 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64781a60b0 con 0x7f6478107e80 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6474c00640 1 -- 172.21.15.103:0/1367994968 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64680035f0 con 0x7f6478107e80 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f6474c00640 1 -- 172.21.15.103:0/1367994968 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6468010690 con 0x7f6478107e80 2024-05-22T23:59:46.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.323+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64781a6620 con 0x7f6478107e80 2024-05-22T23:59:46.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.324+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f647810f220 con 0x7f6478107e80 2024-05-22T23:59:46.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.326+0000 7f6474c00640 1 -- 172.21.15.103:0/1367994968 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f64680037a0 con 0x7f6478107e80 2024-05-22T23:59:46.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.328+0000 7f6474c00640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f645c0a2b90 0x7f645c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:46.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.328+0000 7f6474c00640 1 -- 172.21.15.103:0/1367994968 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f64680cf7b0 con 0x7f6478107e80 2024-05-22T23:59:46.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.328+0000 7f646ea00640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f645c0a2b90 0x7f645c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:46.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.329+0000 7f646ea00640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f645c0a2b90 0x7f645c0a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f64600314e0 tx=0x7f6460000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:46.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.330+0000 7f6474c00640 1 -- 172.21.15.103:0/1367994968 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f646809c350 con 0x7f6478107e80 2024-05-22T23:59:46.519 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:46 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3503284302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:59:46.519 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:46 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3550459130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:59:46.519 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:46 smithi174 ceph-mon[36179]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-22T23:59:46.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:46 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3503284302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:59:46.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:46 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3550459130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:59:46.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:46 smithi105 ceph-mon[36426]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-22T23:59:46.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.570+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f647802d200 con 0x7f6478107e80 2024-05-22T23:59:46.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.571+0000 7f6474c00640 1 -- 172.21.15.103:0/1367994968 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11896 (secure 0 0 0) 0x7f646809bc50 con 0x7f6478107e80 2024-05-22T23:59:46.575 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:46.575 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-22T23:58:05.260333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24559,24563,24575]},"id":1}]} 2024-05-22T23:59:46.575 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 24 2024-05-22T23:59:46.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f645c0a2b90 msgr2=0x7f645c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f645c0a2b90 0x7f645c0a5050 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f64600314e0 tx=0x7f6460000bc0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6478107e80 msgr2=0x7f647819e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6478107e80 0x7f647819e1a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f646800e9c0 tx=0x7f646800ee80 comp rx=0 tx=0).stop 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 shutdown_connections 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f645c0a2b90 0x7f645c0a5050 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6478109a20 0x7f64781a5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6478109060 0x7f647819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 --2- 172.21.15.103:0/1367994968 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6478107e80 0x7f647819e1a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 >> 172.21.15.103:0/1367994968 conn(0x7f6478076e90 msgr2=0x7f647810c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 shutdown_connections 2024-05-22T23:59:46.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.574+0000 7f647e800640 1 -- 172.21.15.103:0/1367994968 wait complete. 2024-05-22T23:59:46.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:46 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3503284302' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-22T23:59:46.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:46 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3550459130' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-22T23:59:46.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:46 smithi103 ceph-mon[31370]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2024-05-22T23:59:46.597 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-22T23:59:46.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.798+0000 7f9b80200640 1 -- 172.21.15.103:0/2497343531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 msgr2=0x7f9b780778d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.798+0000 7f9b80200640 1 --2- 172.21.15.103:0/2497343531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 0x7f9b780778d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9b64009900 tx=0x7f9b6402a400 comp rx=0 tx=0).stop 2024-05-22T23:59:46.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.799+0000 7f9b80200640 1 -- 172.21.15.103:0/2497343531 shutdown_connections 2024-05-22T23:59:46.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.799+0000 7f9b80200640 1 --2- 172.21.15.103:0/2497343531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b78077e10 0x7f9b780786b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.799+0000 7f9b80200640 1 --2- 172.21.15.103:0/2497343531 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 0x7f9b780778d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.799+0000 7f9b80200640 1 --2- 172.21.15.103:0/2497343531 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b78076230 0x7f9b78076650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.799+0000 7f9b80200640 1 -- 172.21.15.103:0/2497343531 >> 172.21.15.103:0/2497343531 conn(0x7f9b780ff060 msgr2=0x7f9b781014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:46.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.799+0000 7f9b80200640 1 -- 172.21.15.103:0/2497343531 shutdown_connections 2024-05-22T23:59:46.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.799+0000 7f9b80200640 1 -- 172.21.15.103:0/2497343531 wait complete. 2024-05-22T23:59:46.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b80200640 1 Processor -- start 2024-05-22T23:59:46.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b80200640 1 -- start start 2024-05-22T23:59:46.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b80200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b78076230 0x7f9b7819e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:46.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b80200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 0x7f9b7819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:46.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b80200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b78077e10 0x7f9b781a5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:46.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b80200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78113bc0 con 0x7f9b78077e10 2024-05-22T23:59:46.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b80200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78113a40 con 0x7f9b78077470 2024-05-22T23:59:46.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b80200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9b78113d40 con 0x7f9b78076230 2024-05-22T23:59:46.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b7e200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b78077e10 0x7f9b781a5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:46.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b7ce00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 0x7f9b7819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:46.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b7ce00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 0x7f9b7819e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48448/0 (socket says 172.21.15.103:48448) 2024-05-22T23:59:46.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b7d800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b78076230 0x7f9b7819e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:46.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b7ce00640 1 -- 172.21.15.103:0/3949877454 learned_addr learned my addr 172.21.15.103:0/3949877454 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:46.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.800+0000 7f9b7e200640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b78077e10 0x7f9b781a5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34850/0 (socket says 172.21.15.103:34850) 2024-05-22T23:59:46.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b7d800640 1 -- 172.21.15.103:0/3949877454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 msgr2=0x7f9b7819e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b7d800640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 0x7f9b7819e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b7d800640 1 -- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b78077e10 msgr2=0x7f9b781a5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:46.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b7d800640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b78077e10 0x7f9b781a5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:46.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b7d800640 1 -- 172.21.15.103:0/3949877454 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b781a5e70 con 0x7f9b78076230 2024-05-22T23:59:46.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b7d800640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b78076230 0x7f9b7819e250 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c00e9f0 tx=0x7f9b6c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:46.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b76000640 1 -- 172.21.15.103:0/3949877454 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c00ccc0 con 0x7f9b78076230 2024-05-22T23:59:46.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b76000640 1 -- 172.21.15.103:0/3949877454 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b6c005020 con 0x7f9b78076230 2024-05-22T23:59:46.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.801+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b781a6160 con 0x7f9b78076230 2024-05-22T23:59:46.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.802+0000 7f9b76000640 1 -- 172.21.15.103:0/3949877454 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0049e0 con 0x7f9b78076230 2024-05-22T23:59:46.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.802+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b781a66b0 con 0x7f9b78076230 2024-05-22T23:59:46.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.803+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b78110640 con 0x7f9b78076230 2024-05-22T23:59:46.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.805+0000 7f9b76000640 1 -- 172.21.15.103:0/3949877454 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f9b6c005560 con 0x7f9b78076230 2024-05-22T23:59:46.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.806+0000 7f9b76000640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9b600a2c10 0x7f9b600a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:46.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.806+0000 7f9b76000640 1 -- 172.21.15.103:0/3949877454 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f9b6c0cf360 con 0x7f9b78076230 2024-05-22T23:59:46.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.806+0000 7f9b7ce00640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9b600a2c10 0x7f9b600a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:46.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.807+0000 7f9b7ce00640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9b600a2c10 0x7f9b600a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9b7819f760 tx=0x7f9b640315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:46.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:46.808+0000 7f9b76000640 1 -- 172.21.15.103:0/3949877454 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9b6c0051d0 con 0x7f9b78076230 2024-05-22T23:59:47.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.009+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b78062470 con 0x7f9b78076230 2024-05-22T23:59:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.012+0000 7f9b76000640 1 -- 172.21.15.103:0/3949877454 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24658 (secure 0 0 0) 0x7f9b6c09bf90 con 0x7f9b78076230 2024-05-22T23:59:47.016 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:47.016 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:58:31.088381+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9b600a2c10 msgr2=0x7f9b600a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9b600a2c10 0x7f9b600a50d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9b7819f760 tx=0x7f9b640315b0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b78076230 msgr2=0x7f9b7819e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b78076230 0x7f9b7819e250 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c00e9f0 tx=0x7f9b6c00eeb0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 shutdown_connections 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f9b600a2c10 0x7f9b600a50d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b78077e10 0x7f9b781a5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9b78077470 0x7f9b7819e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 --2- 172.21.15.103:0/3949877454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b78076230 0x7f9b7819e250 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 >> 172.21.15.103:0/3949877454 conn(0x7f9b780ff060 msgr2=0x7f9b78100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 shutdown_connections 2024-05-22T23:59:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.014+0000 7f9b80200640 1 -- 172.21.15.103:0/3949877454 wait complete. 2024-05-22T23:59:47.038 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-22T23:59:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.240+0000 7f8394a00640 1 -- 172.21.15.103:0/712295059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83901098a0 msgr2=0x7f8390110130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.240+0000 7f8394a00640 1 --2- 172.21.15.103:0/712295059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83901098a0 0x7f8390110130 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8388009ca0 tx=0x7f838802a8c0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.241+0000 7f8394a00640 1 -- 172.21.15.103:0/712295059 shutdown_connections 2024-05-22T23:59:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.241+0000 7f8394a00640 1 --2- 172.21.15.103:0/712295059 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83901098a0 0x7f8390110130 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.241+0000 7f8394a00640 1 --2- 172.21.15.103:0/712295059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8390108ee0 0x7f8390109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.241+0000 7f8394a00640 1 --2- 172.21.15.103:0/712295059 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8390107cf0 0x7f8390108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.241+0000 7f8394a00640 1 -- 172.21.15.103:0/712295059 >> 172.21.15.103:0/712295059 conn(0x7f8390076cb0 msgr2=0x7f83900790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.241+0000 7f8394a00640 1 -- 172.21.15.103:0/712295059 shutdown_connections 2024-05-22T23:59:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.241+0000 7f8394a00640 1 -- 172.21.15.103:0/712295059 wait complete. 2024-05-22T23:59:47.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f8394a00640 1 Processor -- start 2024-05-22T23:59:47.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f8394a00640 1 -- start start 2024-05-22T23:59:47.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f8394a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8390107cf0 0x7f839019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f8394a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8390108ee0 0x7f839019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f8394a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83901098a0 0x7f83901a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f8394a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f83901139c0 con 0x7f83901098a0 2024-05-22T23:59:47.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f8394a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8390113840 con 0x7f8390107cf0 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f8394a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8390113b40 con 0x7f8390108ee0 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f838e800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8390107cf0 0x7f839019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f838e800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8390107cf0 0x7f839019e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48472/0 (socket says 172.21.15.103:48472) 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f838f200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83901098a0 0x7f83901a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.242+0000 7f838e800640 1 -- 172.21.15.103:0/4241603822 learned_addr learned my addr 172.21.15.103:0/4241603822 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f8385e00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8390108ee0 0x7f839019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f838e800640 1 -- 172.21.15.103:0/4241603822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8390108ee0 msgr2=0x7f839019e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f838e800640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8390108ee0 0x7f839019e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f838e800640 1 -- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83901098a0 msgr2=0x7f83901a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f838e800640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83901098a0 0x7f83901a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f838e800640 1 -- 172.21.15.103:0/4241603822 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83901a5bd0 con 0x7f8390107cf0 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f838e800640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8390107cf0 0x7f839019e000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f838000e9c0 tx=0x7f838000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f8387e00640 1 -- 172.21.15.103:0/4241603822 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838000cde0 con 0x7f8390107cf0 2024-05-22T23:59:47.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f8387e00640 1 -- 172.21.15.103:0/4241603822 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8380010910 con 0x7f8390107cf0 2024-05-22T23:59:47.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83901a5ec0 con 0x7f8390107cf0 2024-05-22T23:59:47.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.243+0000 7f8387e00640 1 -- 172.21.15.103:0/4241603822 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8380018a00 con 0x7f8390107cf0 2024-05-22T23:59:47.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.244+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83901a6320 con 0x7f8390107cf0 2024-05-22T23:59:47.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.245+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f839010efb0 con 0x7f8390107cf0 2024-05-22T23:59:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.247+0000 7f8387e00640 1 -- 172.21.15.103:0/4241603822 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f838001f020 con 0x7f8390107cf0 2024-05-22T23:59:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.248+0000 7f8387e00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f83740a2b90 0x7f83740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.248+0000 7f8387e00640 1 -- 172.21.15.103:0/4241603822 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f83800cf730 con 0x7f8390107cf0 2024-05-22T23:59:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.248+0000 7f8385e00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f83740a2b90 0x7f83740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.249+0000 7f8385e00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f83740a2b90 0x7f83740a5050 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f839019f4c0 tx=0x7f8378005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.250+0000 7f8387e00640 1 -- 172.21.15.103:0/4241603822 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f838009c2d0 con 0x7f8390107cf0 2024-05-22T23:59:47.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:47 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1367994968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:47.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:47 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3949877454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:47.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.499+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f839002d2a0 con 0x7f8390107cf0 2024-05-22T23:59:47.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.501+0000 7f8387e00640 1 -- 172.21.15.103:0/4241603822 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11896 (secure 0 0 0) 0x7f838009bbd0 con 0x7f8390107cf0 2024-05-22T23:59:47.504 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-22T23:59:47.505 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-22T23:58:05.260333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24559,24563,24575]},"id":1}]} 2024-05-22T23:59:47.505 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 24 2024-05-22T23:59:47.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f83740a2b90 msgr2=0x7f83740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f83740a2b90 0x7f83740a5050 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f839019f4c0 tx=0x7f8378005f90 comp rx=0 tx=0).stop 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8390107cf0 msgr2=0x7f839019e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8390107cf0 0x7f839019e000 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f838000e9c0 tx=0x7f838000ee80 comp rx=0 tx=0).stop 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 shutdown_connections 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f83740a2b90 0x7f83740a5050 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83901098a0 0x7f83901a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8390108ee0 0x7f839019e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 --2- 172.21.15.103:0/4241603822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8390107cf0 0x7f839019e000 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 >> 172.21.15.103:0/4241603822 conn(0x7f8390076cb0 msgr2=0x7f839010c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 shutdown_connections 2024-05-22T23:59:47.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.504+0000 7f8394a00640 1 -- 172.21.15.103:0/4241603822 wait complete. 2024-05-22T23:59:47.526 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-22T23:57:32.804111+0000', 'modified': '2024-05-22T23:58:05.260333+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24575, 'mds_1': 24563, 'mds_2': 24559}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6837/2433750399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 2433750399}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 2433750399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/3839469517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 3839469517}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 3839469517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6837/2641083628', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 2641083628}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 2641083628}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24559': {'gid': 24559, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.174:6839/3021454884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3021454884}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3021454884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24563': {'gid': 24563, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.103:6841/4258722678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6840', 'nonce': 4258722678}, {'type': 'v1', 'addr': '172.21.15.103:6841', 'nonce': 4258722678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.105:6839/160741513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 160741513}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 160741513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24575, 'qdb_cluster': [24559, 24563, 24575]} 2024-05-22T23:59:47.526 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-22T23:59:47.527 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-22T23:59:47.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:47 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1367994968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:47.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:47 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3949877454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:47.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:47 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1367994968' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:47.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:47 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3949877454' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-22T23:59:47.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.729+0000 7fa242e00640 1 -- 172.21.15.103:0/2118330161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c102970 msgr2=0x7fa23c102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.729+0000 7fa242e00640 1 --2- 172.21.15.103:0/2118330161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c102970 0x7fa23c102df0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa228009900 tx=0x7fa22802a410 comp rx=0 tx=0).stop 2024-05-22T23:59:47.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.729+0000 7fa242e00640 1 -- 172.21.15.103:0/2118330161 shutdown_connections 2024-05-22T23:59:47.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.729+0000 7fa242e00640 1 --2- 172.21.15.103:0/2118330161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa23c103330 0x7fa23c109bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.729+0000 7fa242e00640 1 --2- 172.21.15.103:0/2118330161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c102970 0x7fa23c102df0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.729+0000 7fa242e00640 1 --2- 172.21.15.103:0/2118330161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa23c101780 0x7fa23c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.729+0000 7fa242e00640 1 -- 172.21.15.103:0/2118330161 >> 172.21.15.103:0/2118330161 conn(0x7fa23c0fcd10 msgr2=0x7fa23c0ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.729+0000 7fa242e00640 1 -- 172.21.15.103:0/2118330161 shutdown_connections 2024-05-22T23:59:47.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.730+0000 7fa242e00640 1 -- 172.21.15.103:0/2118330161 wait complete. 2024-05-22T23:59:47.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.731+0000 7fa242e00640 1 Processor -- start 2024-05-22T23:59:47.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.731+0000 7fa242e00640 1 -- start start 2024-05-22T23:59:47.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.731+0000 7fa242e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c101780 0x7fa23c075620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.731+0000 7fa242e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa23c102970 0x7fa23c073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.731+0000 7fa242e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa23c103330 0x7fa23c0741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.731+0000 7fa242e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa23c10d400 con 0x7fa23c103330 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.731+0000 7fa242e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa23c10d280 con 0x7fa23c102970 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.731+0000 7fa242e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa23c10d580 con 0x7fa23c101780 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c101780 0x7fa23c075620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c101780 0x7fa23c075620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34416/0 (socket says 172.21.15.103:34416) 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 -- 172.21.15.103:0/2709712881 learned_addr learned my addr 172.21.15.103:0/2709712881 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa242400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa23c103330 0x7fa23c0741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241000640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa23c102970 0x7fa23c073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 -- 172.21.15.103:0/2709712881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa23c102970 msgr2=0x7fa23c073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa23c102970 0x7fa23c073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 -- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa23c103330 msgr2=0x7fa23c0741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa23c103330 0x7fa23c0741a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 -- 172.21.15.103:0/2709712881 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23c074a30 con 0x7fa23c101780 2024-05-22T23:59:47.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa242400640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa23c103330 0x7fa23c0741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:59:47.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.732+0000 7fa241a00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c101780 0x7fa23c075620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa23000e9c0 tx=0x7fa23000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.733+0000 7fa23a000640 1 -- 172.21.15.103:0/2709712881 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23000cd80 con 0x7fa23c101780 2024-05-22T23:59:47.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.733+0000 7fa23a000640 1 -- 172.21.15.103:0/2709712881 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2300035f0 con 0x7fa23c101780 2024-05-22T23:59:47.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.733+0000 7fa242e00640 1 -- 172.21.15.103:0/2709712881 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa23c1a1bc0 con 0x7fa23c101780 2024-05-22T23:59:47.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.733+0000 7fa23a000640 1 -- 172.21.15.103:0/2709712881 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230010690 con 0x7fa23c101780 2024-05-22T23:59:47.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.733+0000 7fa242e00640 1 -- 172.21.15.103:0/2709712881 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa23c1a20e0 con 0x7fa23c101780 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa2471c8740 1 Processor -- start 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa23a000640 1 -- 172.21.15.103:0/2709712881 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa2300037a0 con 0x7fa23c101780 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa2471c8740 1 -- start start 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691041450 0x55e691041870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691046940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691046fb0 0x55e691049440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa2471c8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e691008db0 con 0x55e691041e40 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa2471c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e690fa5490 con 0x55e691041450 2024-05-22T23:59:47.741 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.736+0000 7fa2471c8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e690f4d100 con 0x55e691046fb0 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691046940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa242400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691041450 0x55e691041870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691046fb0 0x55e691049440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691046940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34906/0 (socket says 172.21.15.103:34906) 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 -- 172.21.15.103:0/1014593790 learned_addr learned my addr 172.21.15.103:0/1014593790 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa242400640 1 --2- 172.21.15.103:0/1014593790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691041450 0x55e691041870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48490/0 (socket says 172.21.15.103:48490) 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 -- 172.21.15.103:0/1014593790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691046fb0 msgr2=0x55e691049440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 --2- 172.21.15.103:0/1014593790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691046fb0 0x55e691049440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 -- 172.21.15.103:0/1014593790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691041450 msgr2=0x55e691041870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 --2- 172.21.15.103:0/1014593790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691041450 0x55e691041870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 -- 172.21.15.103:0/1014593790 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e690cb92d0 con 0x55e691041e40 2024-05-22T23:59:47.742 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.737+0000 7fa241a00640 1 --2- 172.21.15.103:0/1014593790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691046940 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa2300d5a90 tx=0x7fa2300d5b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf51642313c84043 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.738+0000 7fa23a000640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2240a2c10 0x7fa2240a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.738+0000 7fa23a000640 1 -- 172.21.15.103:0/2709712881 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa2300cf7c0 con 0x7fa23c101780 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.738+0000 7fa22ea00640 1 -- 172.21.15.103:0/1014593790 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2300de070 con 0x55e691041e40 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.738+0000 7fa22ea00640 1 -- 172.21.15.103:0/1014593790 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa23009c360 con 0x55e691041e40 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.738+0000 7fa22ea00640 1 -- 172.21.15.103:0/1014593790 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23009b5c0 con 0x55e691041e40 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.738+0000 7fa242400640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2240a2c10 0x7fa2240a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.738+0000 7fa2471c8740 1 -- 172.21.15.103:0/1014593790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 msgr2=0x55e691046940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.738+0000 7fa2471c8740 1 --2- 172.21.15.103:0/1014593790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691046940 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa2300d5a90 tx=0x7fa2300d5b80 comp rx=0 tx=0).stop 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa242400640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2240a2c10 0x7fa2240a50d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55e6910426a0 tx=0x7fa2340098a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 -- 172.21.15.103:0/1014593790 shutdown_connections 2024-05-22T23:59:47.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 --2- 172.21.15.103:0/1014593790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691046fb0 0x55e691049440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 --2- 172.21.15.103:0/1014593790 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691046940 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 --2- 172.21.15.103:0/1014593790 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691041450 0x55e691041870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 -- 172.21.15.103:0/1014593790 >> 172.21.15.103:0/1014593790 conn(0x55e69103cac0 msgr2=0x55e69103cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 -- 172.21.15.103:0/1014593790 shutdown_connections 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 -- 172.21.15.103:0/1014593790 wait complete. 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 -- start start 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.739+0000 7fa2471c8740 1 Processor -- start 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e69104b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e69104bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691052c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa2471c8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e690f60520 con 0x55e691041e40 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa2471c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e690fa1450 con 0x55e691046fb0 2024-05-22T23:59:47.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa2471c8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e691049f60 con 0x55e691041450 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa2471c8740 10 client.0 _ll_register_callbacks cb 0x55e690cb8ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa241a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e69104b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa241000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e69104bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa242400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691052c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa241a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e69104b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34438/0 (socket says 172.21.15.103:34438) 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.740+0000 7fa241a00640 1 -- 172.21.15.103:0/2841580249 learned_addr learned my addr 172.21.15.103:0/2841580249 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa241a00640 1 -- 172.21.15.103:0/2841580249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 msgr2=0x55e691052c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa241a00640 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691052c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa241a00640 1 -- 172.21.15.103:0/2841580249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 msgr2=0x55e69104bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa241a00640 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e69104bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa241a00640 1 -- 172.21.15.103:0/2841580249 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e690d836b0 con 0x55e691041450 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa241a00640 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e69104b6f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa230004630 tx=0x7fa230003e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed35374cccb228b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa223400640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa223400640 1 -- 172.21.15.103:0/2841580249 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2300de070 con 0x55e691041450 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa223400640 1 -- 172.21.15.103:0/2841580249 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa230003b10 con 0x55e691041450 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa2471c8740 1 -- 172.21.15.103:0/2841580249 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa23009c360 con 0x55e691041450 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa223400640 1 -- 172.21.15.103:0/2841580249 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa2300b4e60 con 0x55e691041450 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa2471c8740 10 client.25162 fetch_fsmap learned FSMap version 24 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.741+0000 7fa2471c8740 1 -- 172.21.15.103:0/2841580249 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e691055fe0 con 0x55e691041450 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa223400640 1 -- 172.21.15.103:0/2841580249 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa23009c360 con 0x55e691041450 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa223400640 1 -- 172.21.15.103:0/2841580249 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa23009bec0 con 0x55e691041450 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa2471c8740 10 client.25162 fetch_fsmap finished waiting for FSMap version 24 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa2471c8740 10 client.25162 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa2471c8740 20 client.25162 populate_metadata read hostname 'smithi103' 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa2471c8740 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55e691057330 0x55e691059530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa2471c8740 4 client.25162 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-22T23:59:47.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa2471c8740 1 -- 172.21.15.103:0/2841580249 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e690fb2bc0 con 0x55e691057330 2024-05-22T23:59:47.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.742+0000 7fa241000640 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55e691057330 0x55e691059530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.743+0000 7fa241000640 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55e691057330 0x55e691059530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e7979bcf8691f82c server_cookie=45ab9e6ef5afe726 in_seq=0 out_seq=0 2024-05-22T23:59:47.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.743+0000 7fa223400640 10 client.25162 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-22T23:59:47.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa223400640 1 -- 172.21.15.103:0/2841580249 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e690fb2bc0 con 0x55e691057330 2024-05-22T23:59:47.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa223400640 10 client.25162 handle_command_reply: tid=0 2024-05-22T23:59:47.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 client.25162 shutdown 2024-05-22T23:59:47.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 -- 172.21.15.103:0/2841580249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 msgr2=0x55e69104b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e69104b6f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa230004630 tx=0x7fa230003e80 comp rx=0 tx=0).stop 2024-05-22T23:59:47.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 -- 172.21.15.103:0/2841580249 shutdown_connections 2024-05-22T23:59:47.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55e691057330 0x55e691059530 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691052c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e69104bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa223400640 0 client.25162 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-22T23:59:47.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 --2- 172.21.15.103:0/2841580249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e69104b6f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 -- 172.21.15.103:0/2841580249 >> 172.21.15.103:0/2841580249 conn(0x55e69103cac0 msgr2=0x55e69103bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.744+0000 7fa2471c8740 1 -- 172.21.15.103:0/2841580249 shutdown_connections 2024-05-22T23:59:47.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.745+0000 7fa2471c8740 1 -- 172.21.15.103:0/2841580249 wait complete. 2024-05-22T23:59:47.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.745+0000 7fa2471c8740 20 client.25162 trim_cache size 0 max 16384 2024-05-22T23:59:47.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.764+0000 7fa2471c8740 1 Processor -- start 2024-05-22T23:59:47.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.764+0000 7fa2471c8740 1 -- start start 2024-05-22T23:59:47.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.764+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e691037c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.764+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691052c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691046760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa2471c8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e691057870 con 0x55e691041e40 2024-05-22T23:59:47.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa2471c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e691057f40 con 0x55e691046fb0 2024-05-22T23:59:47.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa2471c8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e691058610 con 0x55e691041450 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa241a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e691037c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa241a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e691037c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34446/0 (socket says 172.21.15.103:34446) 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa241a00640 1 -- 172.21.15.103:0/118231455 learned_addr learned my addr 172.21.15.103:0/118231455 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa242400640 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691046760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa241000640 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691052c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa242400640 1 -- 172.21.15.103:0/118231455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 msgr2=0x55e691037c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa242400640 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e691037c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa242400640 1 -- 172.21.15.103:0/118231455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 msgr2=0x55e691052c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa242400640 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691052c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa242400640 1 -- 172.21.15.103:0/118231455 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e691038490 con 0x55e691046fb0 2024-05-22T23:59:47.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.765+0000 7fa241a00640 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e691037c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:47.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.766+0000 7fa242400640 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691046760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa234005fd0 tx=0x7fa234015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b12a40a5554e2342 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.766+0000 7fa22e000640 1 -- 172.21.15.103:0/118231455 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23402c070 con 0x55e691046fb0 2024-05-22T23:59:47.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.766+0000 7fa22e000640 1 -- 172.21.15.103:0/118231455 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa234010d60 con 0x55e691046fb0 2024-05-22T23:59:47.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.766+0000 7fa22e000640 1 -- 172.21.15.103:0/118231455 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23401e870 con 0x55e691046fb0 2024-05-22T23:59:47.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 -- 172.21.15.103:0/118231455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 msgr2=0x55e691046760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691046760 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa234005fd0 tx=0x7fa234015000 comp rx=0 tx=0).stop 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 -- 172.21.15.103:0/118231455 shutdown_connections 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e691046760 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041e40 0x55e691052c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 --2- 172.21.15.103:0/118231455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041450 0x55e691037c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 -- 172.21.15.103:0/118231455 >> 172.21.15.103:0/118231455 conn(0x55e69103cac0 msgr2=0x55e69103cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 -- 172.21.15.103:0/118231455 shutdown_connections 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 -- 172.21.15.103:0/118231455 wait complete. 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 -- start start 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.767+0000 7fa2471c8740 1 Processor -- start 2024-05-22T23:59:47.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041450 0x55e691050b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041e40 0x55e6910ba050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa2471c8740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e6910bc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa2471c8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e6910593b0 con 0x55e691041450 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa2471c8740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55e691058ce0 con 0x55e691046fb0 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa2471c8740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55e691059a80 con 0x55e691041e40 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa2471c8740 10 client.0 _ll_register_callbacks cb 0x7fa234009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa241a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041450 0x55e691050b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa241a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041450 0x55e691050b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34926/0 (socket says 172.21.15.103:34926) 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa241a00640 1 -- 172.21.15.103:0/4065738992 learned_addr learned my addr 172.21.15.103:0/4065738992 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.768+0000 7fa242400640 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e6910bc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa241a00640 1 -- 172.21.15.103:0/4065738992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041e40 msgr2=0x55e6910ba050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa241a00640 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041e40 0x55e6910ba050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa241a00640 1 -- 172.21.15.103:0/4065738992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 msgr2=0x55e6910bc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa241a00640 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e6910bc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa241a00640 1 -- 172.21.15.103:0/4065738992 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e6910bce30 con 0x55e691041450 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa241a00640 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041450 0x55e691050b80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa23009c230 tx=0x7fa230009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffdf8d335bf90b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa223400640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa223400640 1 -- 172.21.15.103:0/4065738992 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2300de070 con 0x55e691041450 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa223400640 1 -- 172.21.15.103:0/4065738992 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa23009b0b0 con 0x55e691041450 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa2471c8740 1 -- 172.21.15.103:0/4065738992 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa234010d60 con 0x55e691041450 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa223400640 1 -- 172.21.15.103:0/4065738992 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa23009b7a0 con 0x55e691041450 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa2471c8740 10 client.15054 fetch_fsmap learned FSMap version 24 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.769+0000 7fa2471c8740 1 -- 172.21.15.103:0/4065738992 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55e6910be130 con 0x55e691041450 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa223400640 1 -- 172.21.15.103:0/4065738992 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa234010d60 con 0x55e691041450 2024-05-22T23:59:47.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa223400640 1 -- 172.21.15.103:0/4065738992 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fa230004000 con 0x55e691041450 2024-05-22T23:59:47.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa2471c8740 10 client.15054 fetch_fsmap finished waiting for FSMap version 24 2024-05-22T23:59:47.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa2471c8740 10 client.15054 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-22T23:59:47.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa2471c8740 20 client.15054 populate_metadata read hostname 'smithi103' 2024-05-22T23:59:47.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa2471c8740 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55e6910c2b50 0x55e6910c1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa2471c8740 4 client.15054 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-22T23:59:47.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa2471c8740 1 -- 172.21.15.103:0/4065738992 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "status"}) -- 0x55e690facf10 con 0x55e6910c2b50 2024-05-22T23:59:47.777 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.770+0000 7fa241000640 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55e6910c2b50 0x55e6910c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.771+0000 7fa241000640 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55e6910c2b50 0x55e6910c1860 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aed8b16ced70d10c server_cookie=3458ea0e21798b0a in_seq=0 out_seq=0 2024-05-22T23:59:47.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.771+0000 7fa223400640 10 client.15054 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-22T23:59:47.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.771+0000 7fa223400640 1 -- 172.21.15.103:0/4065738992 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55e690facf10 con 0x55e6910c2b50 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.771+0000 7fa223400640 10 client.15054 handle_command_reply: tid=0 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.771+0000 7fa2471c8740 1 client.15054 shutdown 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 -- 172.21.15.103:0/4065738992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041450 msgr2=0x55e691050b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041450 0x55e691050b80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa23009c230 tx=0x7fa230009760 comp rx=0 tx=0).stop 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 -- 172.21.15.103:0/4065738992 shutdown_connections 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55e6910c2b50 0x55e6910c1860 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e691046fb0 0x55e6910bc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55e691041e40 0x55e6910ba050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 --2- 172.21.15.103:0/4065738992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e691041450 0x55e691050b80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa223400640 0 client.15054 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 -- 172.21.15.103:0/4065738992 >> 172.21.15.103:0/4065738992 conn(0x55e69103cac0 msgr2=0x55e691046760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 -- 172.21.15.103:0/4065738992 shutdown_connections 2024-05-22T23:59:47.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 1 -- 172.21.15.103:0/4065738992 wait complete. 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.772+0000 7fa2471c8740 20 client.15054 trim_cache size 0 max 16384 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stdout: "whoami": 2, 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 24559, 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stdout: "want_state": "up:active", 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "up:active", 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs", 2024-05-22T23:59:47.780 INFO:teuthology.orchestra.run.smithi103.stdout: "rank_uptime": 103.528418043, 2024-05-22T23:59:47.781 INFO:teuthology.orchestra.run.smithi103.stdout: "mdsmap_epoch": 24, 2024-05-22T23:59:47.781 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch": 93, 2024-05-22T23:59:47.781 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch_barrier": 91, 2024-05-22T23:59:47.781 INFO:teuthology.orchestra.run.smithi103.stdout: "uptime": 151.818463232 2024-05-22T23:59:47.781 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-22T23:59:47.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 -- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2240a2c10 msgr2=0x7fa2240a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2240a2c10 0x7fa2240a50d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x55e6910426a0 tx=0x7fa2340098a0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 -- 172.21.15.103:0/2709712881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c101780 msgr2=0x7fa23c075620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c101780 0x7fa23c075620 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa23000e9c0 tx=0x7fa23000ee80 comp rx=0 tx=0).stop 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 -- 172.21.15.103:0/2709712881 shutdown_connections 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa2240a2c10 0x7fa2240a50d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa23c103330 0x7fa23c0741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa23c102970 0x7fa23c073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 --2- 172.21.15.103:0/2709712881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa23c101780 0x7fa23c075620 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 -- 172.21.15.103:0/2709712881 >> 172.21.15.103:0/2709712881 conn(0x7fa23c0fcd10 msgr2=0x7fa23c105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 -- 172.21.15.103:0/2709712881 shutdown_connections 2024-05-22T23:59:47.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.773+0000 7fa242e00640 1 -- 172.21.15.103:0/2709712881 wait complete. 2024-05-22T23:59:47.794 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-22T23:59:47.989 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 -- 172.21.15.103:0/316770742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcee00742d0 msgr2=0x7fcee0074750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 --2- 172.21.15.103:0/316770742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcee00742d0 0x7fcee0074750 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcecc009900 tx=0x7fcecc02a410 comp rx=0 tx=0).stop 2024-05-22T23:59:47.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 -- 172.21.15.103:0/316770742 shutdown_connections 2024-05-22T23:59:47.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 --2- 172.21.15.103:0/316770742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee0109770 0x7fcee010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 --2- 172.21.15.103:0/316770742 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcee00742d0 0x7fcee0074750 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 --2- 172.21.15.103:0/316770742 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00758b0 0x7fcee0073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 -- 172.21.15.103:0/316770742 >> 172.21.15.103:0/316770742 conn(0x7fcee00fce20 msgr2=0x7fcee00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:47.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 -- 172.21.15.103:0/316770742 shutdown_connections 2024-05-22T23:59:47.990 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.986+0000 7fcee8600640 1 -- 172.21.15.103:0/316770742 wait complete. 2024-05-22T23:59:47.992 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee8600640 1 Processor -- start 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee8600640 1 -- start start 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee8600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcee00742d0 0x7fcee0199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee8600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00758b0 0x7fcee019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee8600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee0109770 0x7fcee01a13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee8600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcee010f760 con 0x7fcee0109770 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee8600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcee010f5e0 con 0x7fcee00758b0 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee8600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcee010f8e0 con 0x7fcee00742d0 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.987+0000 7fcee6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee0109770 0x7fcee01a13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00758b0 0x7fcee019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00758b0 0x7fcee019a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48530/0 (socket says 172.21.15.103:48530) 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee0109770 0x7fcee01a13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34940/0 (socket says 172.21.15.103:34940) 2024-05-22T23:59:47.993 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 -- 172.21.15.103:0/673102696 learned_addr learned my addr 172.21.15.103:0/673102696 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5c00640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcee00742d0 0x7fcee0199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 -- 172.21.15.103:0/673102696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcee00742d0 msgr2=0x7fcee0199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcee00742d0 0x7fcee0199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 -- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee0109770 msgr2=0x7fcee01a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee0109770 0x7fcee01a13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 -- 172.21.15.103:0/673102696 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcee01a1a40 con 0x7fcee00758b0 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee6600640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee0109770 0x7fcee01a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.988+0000 7fcee5200640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00758b0 0x7fcee019a360 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcecc031440 tx=0x7fcecc032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.989+0000 7fcede000640 1 -- 172.21.15.103:0/673102696 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc03f030 con 0x7fcee00758b0 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.989+0000 7fcee8600640 1 -- 172.21.15.103:0/673102696 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcee01a1cd0 con 0x7fcee00758b0 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.989+0000 7fcee8600640 1 -- 172.21.15.103:0/673102696 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcee01a21c0 con 0x7fcee00758b0 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.989+0000 7fcede000640 1 -- 172.21.15.103:0/673102696 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcecc003ce0 con 0x7fcee00758b0 2024-05-22T23:59:47.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.989+0000 7fcede000640 1 -- 172.21.15.103:0/673102696 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcecc049060 con 0x7fcee00758b0 2024-05-22T23:59:47.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.992+0000 7fceeb3d6740 1 Processor -- start 2024-05-22T23:59:47.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.992+0000 7fceeb3d6740 1 -- start start 2024-05-22T23:59:47.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.992+0000 7fcede000640 1 -- 172.21.15.103:0/673102696 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fcecc042030 con 0x7fcee00758b0 2024-05-22T23:59:47.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.992+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e0965280 0x5601e09656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.992+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e096a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e096ade0 0x5601e096d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5601e092cdb0 con 0x5601e0965280 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5601e08c9490 con 0x5601e096ade0 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5601e0871100 con 0x5601e0965c70 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e0965280 0x5601e09656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e0965280 0x5601e09656a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34946/0 (socket says 172.21.15.103:34946) 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee6600640 1 -- 172.21.15.103:0/245908705 learned_addr learned my addr 172.21.15.103:0/245908705 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee5200640 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e096ade0 0x5601e096d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee5c00640 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e096a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee6600640 1 -- 172.21.15.103:0/245908705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 msgr2=0x5601e096a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee6600640 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e096a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee6600640 1 -- 172.21.15.103:0/245908705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e096ade0 msgr2=0x5601e096d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:47.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee6600640 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e096ade0 0x5601e096d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:47.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.993+0000 7fcee6600640 1 -- 172.21.15.103:0/245908705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601e05dd2d0 con 0x5601e0965280 2024-05-22T23:59:47.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.994+0000 7fcee6600640 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e0965280 0x5601e09656a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fced80117e0 tx=0x7fced8011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17693c4b9e7aba5 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:47.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.994+0000 7fced2a00640 1 -- 172.21.15.103:0/245908705 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8011310 con 0x5601e0965280 2024-05-22T23:59:47.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.994+0000 7fced2a00640 1 -- 172.21.15.103:0/245908705 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced8003e50 con 0x5601e0965280 2024-05-22T23:59:47.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.994+0000 7fced2a00640 1 -- 172.21.15.103:0/245908705 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced8004590 con 0x5601e0965280 2024-05-22T23:59:47.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.994+0000 7fcede000640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcec80a2b90 0x7fcec80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.994+0000 7fcede000640 1 -- 172.21.15.103:0/673102696 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fcecc0ef4f0 con 0x7fcee00758b0 2024-05-22T23:59:48.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fcee5c00640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcec80a2b90 0x7fcec80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 -- 172.21.15.103:0/245908705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e0965280 msgr2=0x5601e09656a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e0965280 0x5601e09656a0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fced80117e0 tx=0x7fced8011ca0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 -- 172.21.15.103:0/245908705 shutdown_connections 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e096ade0 0x5601e096d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e096a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/245908705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e0965280 0x5601e09656a0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 -- 172.21.15.103:0/245908705 >> 172.21.15.103:0/245908705 conn(0x5601e09608f0 msgr2=0x5601e0960cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 -- 172.21.15.103:0/245908705 shutdown_connections 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fceeb3d6740 1 -- 172.21.15.103:0/245908705 wait complete. 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.995+0000 7fcee5c00640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcec80a2b90 0x7fcec80a5050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5601e09617c0 tx=0x7fced4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 1 -- start start 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 1 Processor -- start 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e096f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e096fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0976b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5601e0884520 con 0x5601e096ade0 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5601e08c5450 con 0x5601e0965280 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5601e096dd90 con 0x5601e0965c70 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fceeb3d6740 10 client.0 _ll_register_callbacks cb 0x5601e05dcac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fcee5c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e096f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fcee5c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e096f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48552/0 (socket says 172.21.15.103:48552) 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.996+0000 7fcee5c00640 1 -- 172.21.15.103:0/3685157443 learned_addr learned my addr 172.21.15.103:0/3685157443 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee5200640 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e096fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee6600640 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0976b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee5c00640 1 -- 172.21.15.103:0/3685157443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 msgr2=0x5601e096fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e096fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee5c00640 1 -- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 msgr2=0x5601e0976b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0976b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee5c00640 1 -- 172.21.15.103:0/3685157443 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601e06a76b0 con 0x5601e0965280 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee6600640 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0976b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e096f5e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fced401e970 tx=0x7fced4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0ed8cd993c0b5a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcec7400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.997+0000 7fcec7400640 1 -- 172.21.15.103:0/3685157443 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced402e060 con 0x5601e0965280 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3685157443 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced8003e50 con 0x5601e0965280 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fcec7400640 1 -- 172.21.15.103:0/3685157443 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fced4018030 con 0x5601e0965280 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fcec7400640 1 -- 172.21.15.103:0/3685157443 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fced4035050 con 0x5601e0965280 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fceeb3d6740 10 client.25169 fetch_fsmap learned FSMap version 24 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3685157443 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5601e0979f90 con 0x5601e0965280 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fcec7400640 1 -- 172.21.15.103:0/3685157443 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced8003e50 con 0x5601e0965280 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fcec7400640 1 -- 172.21.15.103:0/3685157443 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fced401c430 con 0x5601e0965280 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fceeb3d6740 10 client.25169 fetch_fsmap finished waiting for FSMap version 24 2024-05-22T23:59:48.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fceeb3d6740 10 client.25169 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fceeb3d6740 20 client.25169 populate_metadata read hostname 'smithi103' 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.998+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5601e097c670 0x5601e097dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.999+0000 7fceeb3d6740 4 client.25169 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.999+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3685157443 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601e08d6bc0 con 0x5601e097c670 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.999+0000 7fcee5200640 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5601e097c670 0x5601e097dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.999+0000 7fcee5200640 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5601e097c670 0x5601e097dae0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=749be635fbdfed77 server_cookie=31158861b5040654 in_seq=0 out_seq=0 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:47.999+0000 7fcec7400640 10 client.25169 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.000+0000 7fcec7400640 1 -- 172.21.15.103:0/3685157443 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5601e08d6bc0 con 0x5601e097c670 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.000+0000 7fcec7400640 10 client.25169 handle_command_reply: tid=0 2024-05-22T23:59:48.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.000+0000 7fceeb3d6740 1 client.25169 shutdown 2024-05-22T23:59:48.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.000+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3685157443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 msgr2=0x5601e096f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.000+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e096f5e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fced401e970 tx=0x7fced4004730 comp rx=0 tx=0).stop 2024-05-22T23:59:48.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3685157443 shutdown_connections 2024-05-22T23:59:48.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0976b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5601e097c670 0x5601e097dae0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e096fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3685157443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e096f5e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3685157443 >> 172.21.15.103:0/3685157443 conn(0x5601e09608f0 msgr2=0x5601e095fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fcec7400640 0 client.25169 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-22T23:59:48.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3685157443 shutdown_connections 2024-05-22T23:59:48.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3685157443 wait complete. 2024-05-22T23:59:48.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.001+0000 7fceeb3d6740 20 client.25169 trim_cache size 0 max 16384 2024-05-22T23:59:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.020+0000 7fceeb3d6740 1 Processor -- start 2024-05-22T23:59:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.020+0000 7fceeb3d6740 1 -- start start 2024-05-22T23:59:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e0963800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0976af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0963d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5601e097be20 con 0x5601e096ade0 2024-05-22T23:59:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5601e097c4f0 con 0x5601e0965280 2024-05-22T23:59:48.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5601e097cbc0 con 0x5601e0965c70 2024-05-22T23:59:48.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e0963800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e0963800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48556/0 (socket says 172.21.15.103:48556) 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5c00640 1 -- 172.21.15.103:0/3797186040 learned_addr learned my addr 172.21.15.103:0/3797186040 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee6600640 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0963d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5200640 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0976af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5c00640 1 -- 172.21.15.103:0/3797186040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 msgr2=0x5601e0976af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0976af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5c00640 1 -- 172.21.15.103:0/3797186040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 msgr2=0x5601e0963d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0963d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.021+0000 7fcee5c00640 1 -- 172.21.15.103:0/3797186040 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601e0960720 con 0x5601e0965280 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.022+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e0963800 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fced4009710 tx=0x7fced4009c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e089d7797124d26 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.022+0000 7fced2000640 1 -- 172.21.15.103:0/3797186040 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced401c3a0 con 0x5601e0965280 2024-05-22T23:59:48.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.022+0000 7fced2000640 1 -- 172.21.15.103:0/3797186040 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced401c540 con 0x5601e0965280 2024-05-22T23:59:48.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.022+0000 7fced2000640 1 -- 172.21.15.103:0/3797186040 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced40047a0 con 0x5601e0965280 2024-05-22T23:59:48.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.022+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3797186040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 msgr2=0x5601e0963800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.022+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e0963800 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fced4009710 tx=0x7fced4009c50 comp rx=0 tx=0).stop 2024-05-22T23:59:48.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3797186040 shutdown_connections 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0963d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0976af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3797186040 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e0963800 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3797186040 >> 172.21.15.103:0/3797186040 conn(0x5601e09608f0 msgr2=0x5601e09670d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3797186040 shutdown_connections 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3797186040 wait complete. 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 -- start start 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.023+0000 7fceeb3d6740 1 Processor -- start 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e09725a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0972ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fceeb3d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0975b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5601e097d960 con 0x5601e096ade0 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5601e097d290 con 0x5601e0965280 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fceeb3d6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5601e097e030 con 0x5601e0965c70 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fceeb3d6740 10 client.0 _ll_register_callbacks cb 0x7fced4009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee6600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0975b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0972ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e09725a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0972ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34480/0 (socket says 172.21.15.103:34480) 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e09725a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48572/0 (socket says 172.21.15.103:48572) 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5200640 1 -- 172.21.15.103:0/3602343012 learned_addr learned my addr 172.21.15.103:0/3602343012 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5c00640 1 -- 172.21.15.103:0/3602343012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 msgr2=0x5601e0972ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0972ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5c00640 1 -- 172.21.15.103:0/3602343012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 msgr2=0x5601e0975b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0975b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5c00640 1 -- 172.21.15.103:0/3602343012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601e09760f0 con 0x5601e0965280 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.024+0000 7fcee5200640 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0972ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fcee5c00640 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e09725a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fced4009710 tx=0x7fced401c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b23a214dc3db4e8e server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fcec7400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fcec7400640 1 -- 172.21.15.103:0/3602343012 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced4048060 con 0x5601e0965280 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3602343012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fced401c540 con 0x5601e0965280 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fcec7400640 1 -- 172.21.15.103:0/3602343012 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fced4024040 con 0x5601e0965280 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fcec7400640 1 -- 172.21.15.103:0/3602343012 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fced401c3a0 con 0x5601e0965280 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fceeb3d6740 10 client.25175 fetch_fsmap learned FSMap version 24 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3602343012 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5601e09eb1f0 con 0x5601e0965280 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.025+0000 7fcec7400640 1 -- 172.21.15.103:0/3602343012 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fced401c540 con 0x5601e0965280 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.026+0000 7fcec7400640 1 -- 172.21.15.103:0/3602343012 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fced4039430 con 0x5601e0965280 2024-05-22T23:59:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.026+0000 7fceeb3d6740 10 client.25175 fetch_fsmap finished waiting for FSMap version 24 2024-05-22T23:59:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.026+0000 7fceeb3d6740 10 client.25175 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-22T23:59:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.026+0000 7fceeb3d6740 20 client.25175 populate_metadata read hostname 'smithi103' 2024-05-22T23:59:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.026+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5601e09efa10 0x5601e09ee720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.026+0000 7fceeb3d6740 4 client.25175 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-22T23:59:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.026+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3602343012 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "status"}) -- 0x5601e08d0f10 con 0x5601e09efa10 2024-05-22T23:59:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.026+0000 7fcee5200640 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5601e09efa10 0x5601e09ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fcee5200640 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5601e09efa10 0x5601e09ee720 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=52a83f72fd1b8ba0 server_cookie=2662ded8b09ddec1 in_seq=0 out_seq=0 2024-05-22T23:59:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fcec7400640 10 client.25175 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-22T23:59:48.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fcec7400640 1 -- 172.21.15.103:0/3602343012 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5601e08d0f10 con 0x5601e09efa10 2024-05-22T23:59:48.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fcec7400640 10 client.25175 handle_command_reply: tid=0 2024-05-22T23:59:48.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 client.25175 shutdown 2024-05-22T23:59:48.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3602343012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 msgr2=0x5601e09725a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e09725a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fced4009710 tx=0x7fced401c760 comp rx=0 tx=0).stop 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3602343012 shutdown_connections 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5601e096ade0 0x5601e0975b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5601e09efa10 0x5601e09ee720 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5601e0965c70 0x5601e0972ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 --2- 172.21.15.103:0/3602343012 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5601e0965280 0x5601e09725a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fcec7400640 0 client.25175 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3602343012 >> 172.21.15.103:0/3602343012 conn(0x5601e09608f0 msgr2=0x5601e095f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3602343012 shutdown_connections 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 1 -- 172.21.15.103:0/3602343012 wait complete. 2024-05-22T23:59:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.027+0000 7fceeb3d6740 20 client.25175 trim_cache size 0 max 16384 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "whoami": 1, 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 24563, 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "want_state": "up:active", 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "up:active", 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs", 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "rank_uptime": 104.798353869, 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "mdsmap_epoch": 24, 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch": 93, 2024-05-22T23:59:48.036 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch_barrier": 91, 2024-05-22T23:59:48.037 INFO:teuthology.orchestra.run.smithi103.stdout: "uptime": 144.06794094399999 2024-05-22T23:59:48.037 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-22T23:59:48.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 -- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcec80a2b90 msgr2=0x7fcec80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcec80a2b90 0x7fcec80a5050 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5601e09617c0 tx=0x7fced4005f90 comp rx=0 tx=0).stop 2024-05-22T23:59:48.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 -- 172.21.15.103:0/673102696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00758b0 msgr2=0x7fcee019a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00758b0 0x7fcee019a360 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcecc031440 tx=0x7fcecc032ce0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 -- 172.21.15.103:0/673102696 shutdown_connections 2024-05-22T23:59:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fcec80a2b90 0x7fcec80a5050 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcee0109770 0x7fcee01a13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00758b0 0x7fcee019a360 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 --2- 172.21.15.103:0/673102696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcee00742d0 0x7fcee0199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 -- 172.21.15.103:0/673102696 >> 172.21.15.103:0/673102696 conn(0x7fcee00fce20 msgr2=0x7fcee0105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 -- 172.21.15.103:0/673102696 shutdown_connections 2024-05-22T23:59:48.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.028+0000 7fcee8600640 1 -- 172.21.15.103:0/673102696 wait complete. 2024-05-22T23:59:48.050 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-22T23:59:48.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.251+0000 7f267e400640 1 -- 172.21.15.103:0/2489488172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26780758b0 msgr2=0x7f2678073d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.251+0000 7f267e400640 1 --2- 172.21.15.103:0/2489488172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26780758b0 0x7f2678073d00 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f266c009900 tx=0x7f266c02a400 comp rx=0 tx=0).stop 2024-05-22T23:59:48.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.252+0000 7f267e400640 1 -- 172.21.15.103:0/2489488172 shutdown_connections 2024-05-22T23:59:48.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.252+0000 7f267e400640 1 --2- 172.21.15.103:0/2489488172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2678109640 0x7f267810bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.252+0000 7f267e400640 1 --2- 172.21.15.103:0/2489488172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26780742d0 0x7f2678074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.252+0000 7f267e400640 1 --2- 172.21.15.103:0/2489488172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f26780758b0 0x7f2678073d00 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.252+0000 7f267e400640 1 -- 172.21.15.103:0/2489488172 >> 172.21.15.103:0/2489488172 conn(0x7f26780fcd10 msgr2=0x7f26780ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.252+0000 7f267e400640 1 -- 172.21.15.103:0/2489488172 shutdown_connections 2024-05-22T23:59:48.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.252+0000 7f267e400640 1 -- 172.21.15.103:0/2489488172 wait complete. 2024-05-22T23:59:48.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.253+0000 7f267e400640 1 Processor -- start 2024-05-22T23:59:48.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.253+0000 7f267e400640 1 -- start start 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.253+0000 7f267e400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26780742d0 0x7f2678199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.253+0000 7f267e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26780758b0 0x7f267819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.253+0000 7f267e400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2678109640 0x7f26781a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.253+0000 7f267e400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f267810f5c0 con 0x7f26780758b0 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.253+0000 7f267e400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f267810f440 con 0x7f26780742d0 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.253+0000 7f267e400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f267810f740 con 0x7f2678109640 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26780742d0 0x7f2678199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f2677e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26780758b0 0x7f267819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26780742d0 0x7f2678199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48604/0 (socket says 172.21.15.103:48604) 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 -- 172.21.15.103:0/2132721315 learned_addr learned my addr 172.21.15.103:0/2132721315 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267da00640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2678109640 0x7f26781a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 -- 172.21.15.103:0/2132721315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2678109640 msgr2=0x7f26781a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2678109640 0x7f26781a11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 -- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26780758b0 msgr2=0x7f267819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26780758b0 0x7f267819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 -- 172.21.15.103:0/2132721315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26781a1890 con 0x7f26780742d0 2024-05-22T23:59:48.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.254+0000 7f267d000640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26780742d0 0x7f2678199c80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f266c009230 tx=0x7f266c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.255+0000 7f2675600640 1 -- 172.21.15.103:0/2132721315 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266c03f030 con 0x7f26780742d0 2024-05-22T23:59:48.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.255+0000 7f267e400640 1 -- 172.21.15.103:0/2132721315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26781a1b20 con 0x7f26780742d0 2024-05-22T23:59:48.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.255+0000 7f2675600640 1 -- 172.21.15.103:0/2132721315 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f266c003ce0 con 0x7f26780742d0 2024-05-22T23:59:48.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.255+0000 7f2675600640 1 -- 172.21.15.103:0/2132721315 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266c035070 con 0x7f26780742d0 2024-05-22T23:59:48.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.255+0000 7f267e400640 1 -- 172.21.15.103:0/2132721315 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26781a2010 con 0x7f26780742d0 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.258+0000 7f2675600640 1 -- 172.21.15.103:0/2132721315 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f266c042030 con 0x7f26780742d0 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.258+0000 7f268296a740 1 Processor -- start 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f268296a740 1 -- start start 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f268296a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c8c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f268296a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c8ce40 0x564ab1c91940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f268296a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c91fb0 0x564ab1c94440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f268296a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ab1c53db0 con 0x564ab1c91fb0 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f268296a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564ab1bf0490 con 0x564ab1c8c450 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f268296a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564ab1b98100 con 0x564ab1c8ce40 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f267d000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c8ce40 0x564ab1c91940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f267da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c8c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f267da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c8c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48612/0 (socket says 172.21.15.103:48612) 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f267da00640 1 -- 172.21.15.103:0/2123490866 learned_addr learned my addr 172.21.15.103:0/2123490866 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.259+0000 7f2677e00640 1 --2- 172.21.15.103:0/2123490866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c91fb0 0x564ab1c94440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f267da00640 1 -- 172.21.15.103:0/2123490866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c8ce40 msgr2=0x564ab1c91940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f267da00640 1 --2- 172.21.15.103:0/2123490866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c8ce40 0x564ab1c91940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f267da00640 1 -- 172.21.15.103:0/2123490866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c91fb0 msgr2=0x564ab1c94440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f267da00640 1 --2- 172.21.15.103:0/2123490866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c91fb0 0x564ab1c94440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f267da00640 1 -- 172.21.15.103:0/2123490866 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ab19042d0 con 0x564ab1c8c450 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f2675600640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26600a2b90 0x7f26600a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f267da00640 1 --2- 172.21.15.103:0/2123490866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c8c870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2670011440 tx=0x7f2670011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e867286ef7863d9c server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f2675600640 1 -- 172.21.15.103:0/2132721315 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f266c0ef8b0 con 0x7f26780742d0 2024-05-22T23:59:48.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.260+0000 7f267d000640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26600a2b90 0x7f26600a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.261+0000 7f267d000640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26600a2b90 0x7f26600a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f266c0ed3c0 tx=0x7f266c0f5550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.262+0000 7f266a000640 1 -- 172.21.15.103:0/2123490866 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670003c30 con 0x564ab1c8c450 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.262+0000 7f266a000640 1 -- 172.21.15.103:0/2123490866 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2670003dd0 con 0x564ab1c8c450 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.262+0000 7f266a000640 1 -- 172.21.15.103:0/2123490866 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267000fb50 con 0x564ab1c8c450 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 -- 172.21.15.103:0/2123490866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 msgr2=0x564ab1c8c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 --2- 172.21.15.103:0/2123490866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c8c870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2670011440 tx=0x7f2670011900 comp rx=0 tx=0).stop 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 -- 172.21.15.103:0/2123490866 shutdown_connections 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 --2- 172.21.15.103:0/2123490866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c91fb0 0x564ab1c94440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 --2- 172.21.15.103:0/2123490866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c8ce40 0x564ab1c91940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 --2- 172.21.15.103:0/2123490866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c8c870 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 -- 172.21.15.103:0/2123490866 >> 172.21.15.103:0/2123490866 conn(0x564ab1c87ac0 msgr2=0x564ab1c87ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 -- 172.21.15.103:0/2123490866 shutdown_connections 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 -- 172.21.15.103:0/2123490866 wait complete. 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.263+0000 7f268296a740 1 -- start start 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f268296a740 1 Processor -- start 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f268296a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c96750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f268296a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c96c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f268296a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c9dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f268296a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ab1bab520 con 0x564ab1c8c450 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f268296a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564ab1bec450 con 0x564ab1c8ce40 2024-05-22T23:59:48.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f268296a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564ab1c94f60 con 0x564ab1c91fb0 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f268296a740 10 client.0 _ll_register_callbacks cb 0x564ab1903ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f267da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c96750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.264+0000 7f2677e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c96c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f2677e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c96c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48624/0 (socket says 172.21.15.103:48624) 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f267da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c96750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34994/0 (socket says 172.21.15.103:34994) 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f2677e00640 1 -- 172.21.15.103:0/3062498319 learned_addr learned my addr 172.21.15.103:0/3062498319 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f267da00640 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c9dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f2677e00640 1 -- 172.21.15.103:0/3062498319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 msgr2=0x564ab1c9dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f2677e00640 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c9dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f2677e00640 1 -- 172.21.15.103:0/3062498319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 msgr2=0x564ab1c96750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f2677e00640 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c96750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f2677e00640 1 -- 172.21.15.103:0/3062498319 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ab19ce6b0 con 0x564ab1c8ce40 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f267da00640 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c96750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:48.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f2677e00640 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c96c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f26640126e0 tx=0x7f2664012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5aa367a1d16f4cc server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.265+0000 7f265ea00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-22T23:59:48.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.268+0000 7f265ea00640 1 -- 172.21.15.103:0/3062498319 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26640120d0 con 0x564ab1c8ce40 2024-05-22T23:59:48.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.268+0000 7f265ea00640 1 -- 172.21.15.103:0/3062498319 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f266400f920 con 0x564ab1c8ce40 2024-05-22T23:59:48.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.268+0000 7f268296a740 1 -- 172.21.15.103:0/3062498319 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2670003dd0 con 0x564ab1c8ce40 2024-05-22T23:59:48.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.268+0000 7f265ea00640 1 -- 172.21.15.103:0/3062498319 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2664012270 con 0x564ab1c8ce40 2024-05-22T23:59:48.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f268296a740 10 client.25187 fetch_fsmap learned FSMap version 24 2024-05-22T23:59:48.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f268296a740 1 -- 172.21.15.103:0/3062498319 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564ab1ca10b0 con 0x564ab1c8ce40 2024-05-22T23:59:48.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f265ea00640 1 -- 172.21.15.103:0/3062498319 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2670003dd0 con 0x564ab1c8ce40 2024-05-22T23:59:48.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f265ea00640 1 -- 172.21.15.103:0/3062498319 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f266401c420 con 0x564ab1c8ce40 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f268296a740 10 client.25187 fetch_fsmap finished waiting for FSMap version 24 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f268296a740 10 client.25187 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f268296a740 20 client.25187 populate_metadata read hostname 'smithi103' 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f268296a740 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x564ab1ca3720 0x564ab1ca4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f268296a740 4 client.25187 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f268296a740 1 -- 172.21.15.103:0/3062498319 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ab1bfdbc0 con 0x564ab1ca3720 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.269+0000 7f267da00640 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x564ab1ca3720 0x564ab1ca4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.270+0000 7f267da00640 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x564ab1ca3720 0x564ab1ca4b90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cca8d2e80cbd117b server_cookie=690feabb83b980b6 in_seq=0 out_seq=0 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.270+0000 7f265ea00640 10 client.25187 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f265ea00640 1 -- 172.21.15.103:0/3062498319 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ab1bfdbc0 con 0x564ab1ca3720 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f265ea00640 10 client.25187 handle_command_reply: tid=0 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 client.25187 shutdown 2024-05-22T23:59:48.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 -- 172.21.15.103:0/3062498319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 msgr2=0x564ab1c96c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c96c90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f26640126e0 tx=0x7f2664012ba0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 -- 172.21.15.103:0/3062498319 shutdown_connections 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x564ab1ca3720 0x564ab1ca4b90 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c9dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c96c90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f265ea00640 0 client.25187 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 --2- 172.21.15.103:0/3062498319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c96750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 -- 172.21.15.103:0/3062498319 >> 172.21.15.103:0/3062498319 conn(0x564ab1c87ac0 msgr2=0x564ab1c86c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 -- 172.21.15.103:0/3062498319 shutdown_connections 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.271+0000 7f268296a740 1 -- 172.21.15.103:0/3062498319 wait complete. 2024-05-22T23:59:48.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.272+0000 7f268296a740 20 client.25187 trim_cache size 0 max 16384 2024-05-22T23:59:48.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.291+0000 7f268296a740 1 Processor -- start 2024-05-22T23:59:48.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.291+0000 7f268296a740 1 -- start start 2024-05-22T23:59:48.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.291+0000 7f268296a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c96360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.291+0000 7f268296a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8ce40 0x564ab1c9dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.291+0000 7f268296a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c968a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.291+0000 7f268296a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ab1ca2ed0 con 0x564ab1c8ce40 2024-05-22T23:59:48.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.291+0000 7f268296a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564ab1ca35a0 con 0x564ab1c8c450 2024-05-22T23:59:48.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.291+0000 7f268296a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564ab1ca3c70 con 0x564ab1c91fb0 2024-05-22T23:59:48.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f2677e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8ce40 0x564ab1c9dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f267da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c96360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f2677e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8ce40 0x564ab1c9dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35006/0 (socket says 172.21.15.103:35006) 2024-05-22T23:59:48.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f2677e00640 1 -- 172.21.15.103:0/2158006483 learned_addr learned my addr 172.21.15.103:0/2158006483 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:48.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f267da00640 1 --2- 172.21.15.103:0/2158006483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f2677e00640 1 -- 172.21.15.103:0/2158006483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 msgr2=0x564ab1c968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f2677e00640 1 --2- 172.21.15.103:0/2158006483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f2677e00640 1 -- 172.21.15.103:0/2158006483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 msgr2=0x564ab1c96360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f2677e00640 1 --2- 172.21.15.103:0/2158006483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c96360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.292+0000 7f2677e00640 1 -- 172.21.15.103:0/2158006483 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ab1c878f0 con 0x564ab1c8ce40 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.293+0000 7f2677e00640 1 --2- 172.21.15.103:0/2158006483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8ce40 0x564ab1c9dc90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2664000c00 tx=0x7f266400f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719162bdb98f4003 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.293+0000 7f2669600640 1 -- 172.21.15.103:0/2158006483 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266402f070 con 0x564ab1c8ce40 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.293+0000 7f2669600640 1 -- 172.21.15.103:0/2158006483 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26640055c0 con 0x564ab1c8ce40 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.293+0000 7f2669600640 1 -- 172.21.15.103:0/2158006483 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266400fc50 con 0x564ab1c8ce40 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 -- 172.21.15.103:0/2158006483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8ce40 msgr2=0x564ab1c9dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 --2- 172.21.15.103:0/2158006483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8ce40 0x564ab1c9dc90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2664000c00 tx=0x7f266400f810 comp rx=0 tx=0).stop 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 -- 172.21.15.103:0/2158006483 shutdown_connections 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 --2- 172.21.15.103:0/2158006483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 --2- 172.21.15.103:0/2158006483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8ce40 0x564ab1c9dc90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 --2- 172.21.15.103:0/2158006483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8c450 0x564ab1c96360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 -- 172.21.15.103:0/2158006483 >> 172.21.15.103:0/2158006483 conn(0x564ab1c87ac0 msgr2=0x564ab1ca2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 -- 172.21.15.103:0/2158006483 shutdown_connections 2024-05-22T23:59:48.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 -- 172.21.15.103:0/2158006483 wait complete. 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 -- start start 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.294+0000 7f268296a740 1 Processor -- start 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f268296a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c992d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f268296a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c99810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f268296a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c9c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f268296a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ab1ca4a10 con 0x564ab1c8c450 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f268296a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564ab1ca4340 con 0x564ab1c8ce40 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f268296a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564ab1ca50e0 con 0x564ab1c91fb0 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f268296a740 10 client.0 _ll_register_callbacks cb 0x7f2664009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-22T23:59:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f267da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f2677e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c99810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f267da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c992d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35016/0 (socket says 172.21.15.103:35016) 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f2677e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c99810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48644/0 (socket says 172.21.15.103:48644) 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f267da00640 1 -- 172.21.15.103:0/660804317 learned_addr learned my addr 172.21.15.103:0/660804317 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.295+0000 7f267da00640 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c9c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f2677e00640 1 -- 172.21.15.103:0/660804317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 msgr2=0x564ab1c9c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f2677e00640 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c9c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f2677e00640 1 -- 172.21.15.103:0/660804317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 msgr2=0x564ab1c992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f2677e00640 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f2677e00640 1 -- 172.21.15.103:0/660804317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ab1c9ce20 con 0x564ab1c8ce40 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f2677e00640 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c99810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f266401c8e0 tx=0x7f266401c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d867fb03c38509a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f265ea00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-22T23:59:48.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f265ea00640 1 -- 172.21.15.103:0/660804317 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f266402f030 con 0x564ab1c8ce40 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f268296a740 1 -- 172.21.15.103:0/660804317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26640055c0 con 0x564ab1c8ce40 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f265ea00640 1 -- 172.21.15.103:0/660804317 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2664018050 con 0x564ab1c8ce40 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.296+0000 7f265ea00640 1 -- 172.21.15.103:0/660804317 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f266400f990 con 0x564ab1c8ce40 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f268296a740 10 client.25193 fetch_fsmap learned FSMap version 24 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f268296a740 1 -- 172.21.15.103:0/660804317 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564ab1d11ee0 con 0x564ab1c8ce40 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f265ea00640 1 -- 172.21.15.103:0/660804317 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26640055c0 con 0x564ab1c8ce40 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f265ea00640 1 -- 172.21.15.103:0/660804317 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f2664003bd0 con 0x564ab1c8ce40 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f268296a740 10 client.25193 fetch_fsmap finished waiting for FSMap version 24 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f268296a740 10 client.25193 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f268296a740 20 client.25193 populate_metadata read hostname 'smithi103' 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f268296a740 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x564ab1d16670 0x564ab1d15380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:48.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f268296a740 4 client.25193 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f268296a740 1 -- 172.21.15.103:0/660804317 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "status"}) -- 0x564ab1bf7f10 con 0x564ab1d16670 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.297+0000 7f267da00640 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x564ab1d16670 0x564ab1d15380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f267da00640 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x564ab1d16670 0x564ab1d15380 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66b3c13f2bdacacd server_cookie=dd6a55b76a9d2126 in_seq=0 out_seq=0 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f265ea00640 10 client.25193 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f265ea00640 1 -- 172.21.15.103:0/660804317 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x564ab1bf7f10 con 0x564ab1d16670 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f265ea00640 10 client.25193 handle_command_reply: tid=0 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 client.25193 shutdown 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 -- 172.21.15.103:0/660804317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 msgr2=0x564ab1c99810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c99810 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f266401c8e0 tx=0x7f266401c650 comp rx=0 tx=0).stop 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 -- 172.21.15.103:0/660804317 shutdown_connections 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x564ab1d16670 0x564ab1d15380 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564ab1c91fb0 0x564ab1c9c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564ab1c8ce40 0x564ab1c99810 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f265ea00640 0 client.25193 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-22T23:59:48.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 --2- 172.21.15.103:0/660804317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ab1c8c450 0x564ab1c992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 -- 172.21.15.103:0/660804317 >> 172.21.15.103:0/660804317 conn(0x564ab1c87ac0 msgr2=0x564ab1c96360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 -- 172.21.15.103:0/660804317 shutdown_connections 2024-05-22T23:59:48.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 1 -- 172.21.15.103:0/660804317 wait complete. 2024-05-22T23:59:48.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.298+0000 7f268296a740 20 client.25193 trim_cache size 0 max 16384 2024-05-22T23:59:48.307 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "whoami": 0, 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 24575, 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "want_state": "up:active", 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "up:active", 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs", 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "rank_uptime": 134.58809028900001, 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "mdsmap_epoch": 24, 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch": 93, 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch_barrier": 91, 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "uptime": 141.35649888399999 2024-05-22T23:59:48.308 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-22T23:59:48.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 -- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26600a2b90 msgr2=0x7f26600a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26600a2b90 0x7f26600a5050 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f266c0ed3c0 tx=0x7f266c0f5550 comp rx=0 tx=0).stop 2024-05-22T23:59:48.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 -- 172.21.15.103:0/2132721315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26780742d0 msgr2=0x7f2678199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:48.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26780742d0 0x7f2678199c80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f266c009230 tx=0x7f266c0042f0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 -- 172.21.15.103:0/2132721315 shutdown_connections 2024-05-22T23:59:48.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f26600a2b90 0x7f26600a5050 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2678109640 0x7f26781a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26780758b0 0x7f267819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 --2- 172.21.15.103:0/2132721315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26780742d0 0x7f2678199c80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:48.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 -- 172.21.15.103:0/2132721315 >> 172.21.15.103:0/2132721315 conn(0x7f26780fcd10 msgr2=0x7f2678105540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:48.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 -- 172.21.15.103:0/2132721315 shutdown_connections 2024-05-22T23:59:48.310 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:48.299+0000 7f267e400640 1 -- 172.21.15.103:0/2132721315 wait complete. 2024-05-22T23:59:48.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:48 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/4241603822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:48.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:48 smithi103 ceph-mon[31370]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:48.322 INFO:tasks.cephadm:Setup complete, yielding 2024-05-22T23:59:48.323 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-22T23:59:48.333 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-05-22T23:59:48.333 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch status' 2024-05-22T23:59:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:48 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/4241603822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:48 smithi105 ceph-mon[36426]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:48 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/4241603822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-22T23:59:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:48 smithi174 ceph-mon[36179]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:48.772 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:50.542 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:50 smithi103 ceph-mon[31370]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:50 smithi105 ceph-mon[36426]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:50.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:50 smithi174 ceph-mon[36179]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-22T23:59:51.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.076+0000 7f2fb0c00700 1 -- 172.21.15.103:0/4242645111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fac102e50 msgr2=0x7f2fac105240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:51.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.076+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/4242645111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fac102e50 0x7f2fac105240 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009990 tx=0x7f2f9c009ca0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.076+0000 7f2fb0c00700 1 -- 172.21.15.103:0/4242645111 shutdown_connections 2024-05-22T23:59:51.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.076+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/4242645111 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fac102e50 0x7f2fac105240 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.076+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/4242645111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fac1080b0 0x7f2fac10a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.076+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/4242645111 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fac105780 0x7f2fac107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.076+0000 7f2fb0c00700 1 -- 172.21.15.103:0/4242645111 >> 172.21.15.103:0/4242645111 conn(0x7f2fac0fc860 msgr2=0x7f2fac0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:51.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.077+0000 7f2fb0c00700 1 -- 172.21.15.103:0/4242645111 shutdown_connections 2024-05-22T23:59:51.080 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.077+0000 7f2fb0c00700 1 -- 172.21.15.103:0/4242645111 wait complete. 2024-05-22T23:59:51.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.078+0000 7f2fb0c00700 1 Processor -- start 2024-05-22T23:59:51.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.078+0000 7f2fb0c00700 1 -- start start 2024-05-22T23:59:51.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fb0c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fac102e50 0x7f2fac19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:51.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fb0c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fac105780 0x7f2fac19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:51.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fb0c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fac1080b0 0x7f2fac1a1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:51.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fb0c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac10df90 con 0x7f2fac102e50 2024-05-22T23:59:51.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fb0c00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac10de10 con 0x7f2fac1080b0 2024-05-22T23:59:51.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fb0c00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2fac10e110 con 0x7f2fac105780 2024-05-22T23:59:51.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fa9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fac102e50 0x7f2fac19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:51.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fa1400700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fac105780 0x7f2fac19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fa9e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fac102e50 0x7f2fac19a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:35034/0 (socket says 172.21.15.103:35034) 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2fa9e00700 1 -- 172.21.15.103:0/1647335266 learned_addr learned my addr 172.21.15.103:0/1647335266 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.079+0000 7f2faa800700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fac1080b0 0x7f2fac1a1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa1400700 1 -- 172.21.15.103:0/1647335266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fac1080b0 msgr2=0x7f2fac1a1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa1400700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fac1080b0 0x7f2fac1a1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa1400700 1 -- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fac102e50 msgr2=0x7f2fac19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa1400700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fac102e50 0x7f2fac19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa1400700 1 -- 172.21.15.103:0/1647335266 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fac1a2470 con 0x7f2fac105780 2024-05-22T23:59:51.083 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa9e00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fac102e50 0x7f2fac19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-22T23:59:51.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa1400700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fac105780 0x7f2fac19ad50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400ea20 tx=0x7f2f9400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:51.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa3400700 1 -- 172.21.15.103:0/1647335266 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400cbd0 con 0x7f2fac105780 2024-05-22T23:59:51.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa3400700 1 -- 172.21.15.103:0/1647335266 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9400ee60 con 0x7f2fac105780 2024-05-22T23:59:51.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fac1a2760 con 0x7f2fac105780 2024-05-22T23:59:51.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fa3400700 1 -- 172.21.15.103:0/1647335266 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94018680 con 0x7f2fac105780 2024-05-22T23:59:51.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.080+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fac1a2ca0 con 0x7f2fac105780 2024-05-22T23:59:51.084 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.081+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fac069050 con 0x7f2fac105780 2024-05-22T23:59:51.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.084+0000 7f2fa3400700 1 -- 172.21.15.103:0/1647335266 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f2f94018820 con 0x7f2fac105780 2024-05-22T23:59:51.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.085+0000 7f2fa3400700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f900a2c20 0x7f2f900a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:51.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.085+0000 7f2fa3400700 1 -- 172.21.15.103:0/1647335266 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2f940cf830 con 0x7f2fac105780 2024-05-22T23:59:51.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.085+0000 7f2fa9e00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f900a2c20 0x7f2f900a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:51.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.086+0000 7f2fa9e00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f900a2c20 0x7f2f900a50e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00b7b0 tx=0x7f2f9c00b8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:51.089 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.087+0000 7f2fa3400700 1 -- 172.21.15.103:0/1647335266 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2f9409c3d0 con 0x7f2fac105780 2024-05-22T23:59:51.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.331+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f2fac0627a0 con 0x7f2f900a2c20 2024-05-22T23:59:51.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.334+0000 7f2fa3400700 1 -- 172.21.15.103:0/1647335266 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f2fac0627a0 con 0x7f2f900a2c20 2024-05-22T23:59:51.337 INFO:teuthology.orchestra.run.smithi103.stdout:Backend: cephadm 2024-05-22T23:59:51.337 INFO:teuthology.orchestra.run.smithi103.stdout:Available: Yes 2024-05-22T23:59:51.338 INFO:teuthology.orchestra.run.smithi103.stdout:Paused: No 2024-05-22T23:59:51.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.337+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f900a2c20 msgr2=0x7f2f900a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:51.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.337+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f900a2c20 0x7f2f900a50e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c00b7b0 tx=0x7f2f9c00b8a0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.337+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fac105780 msgr2=0x7f2fac19ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:51.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.337+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fac105780 0x7f2fac19ad50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400ea20 tx=0x7f2f9400ed30 comp rx=0 tx=0).stop 2024-05-22T23:59:51.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.337+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 shutdown_connections 2024-05-22T23:59:51.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.338+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2fac1080b0 0x7f2fac1a1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.338+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2f900a2c20 0x7f2f900a50e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.338+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fac102e50 0x7f2fac19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.338+0000 7f2fb0c00700 1 --2- 172.21.15.103:0/1647335266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2fac105780 0x7f2fac19ad50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:51.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.338+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 >> 172.21.15.103:0/1647335266 conn(0x7f2fac0fc860 msgr2=0x7f2fac1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:51.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.338+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 shutdown_connections 2024-05-22T23:59:51.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:51.338+0000 7f2fb0c00700 1 -- 172.21.15.103:0/1647335266 wait complete. 2024-05-22T23:59:52.255 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-22T23:59:52.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:52 smithi103 ceph-mon[31370]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:52.319 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:52 smithi103 ceph-mon[31370]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:52.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:52 smithi105 ceph-mon[36426]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:52.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:52 smithi105 ceph-mon[36426]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:52.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:52 smithi174 ceph-mon[36179]: from='client.25207 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:52.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:52 smithi174 ceph-mon[36179]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-22T23:59:52.687 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:54.409 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:54 smithi103 ceph-mon[31370]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:54.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:54 smithi105 ceph-mon[36426]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:54.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:54 smithi174 ceph-mon[36179]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-22T23:59:54.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 -- 172.21.15.103:0/1636676442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5201080b0 msgr2=0x7fe52010a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:54.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 --2- 172.21.15.103:0/1636676442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5201080b0 0x7fe52010a520 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe51800b0a0 tx=0x7fe51800b3b0 comp rx=0 tx=0).stop 2024-05-22T23:59:54.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 -- 172.21.15.103:0/1636676442 shutdown_connections 2024-05-22T23:59:54.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 --2- 172.21.15.103:0/1636676442 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5201080b0 0x7fe52010a520 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:54.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 --2- 172.21.15.103:0/1636676442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520102e50 0x7fe520105240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:54.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 --2- 172.21.15.103:0/1636676442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe520105780 0x7fe520107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:54.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 -- 172.21.15.103:0/1636676442 >> 172.21.15.103:0/1636676442 conn(0x7fe5200fc860 msgr2=0x7fe5200feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:54.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 -- 172.21.15.103:0/1636676442 shutdown_connections 2024-05-22T23:59:54.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.964+0000 7fe526c00700 1 -- 172.21.15.103:0/1636676442 wait complete. 2024-05-22T23:59:54.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.965+0000 7fe526c00700 1 Processor -- start 2024-05-22T23:59:54.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.966+0000 7fe526c00700 1 -- start start 2024-05-22T23:59:54.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.966+0000 7fe526c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe520102e50 0x7fe520198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:54.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.966+0000 7fe526c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520105780 0x7fe520198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:54.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.966+0000 7fe526c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5201080b0 0x7fe52019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:54.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.966+0000 7fe526c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe52010dfe0 con 0x7fe520105780 2024-05-22T23:59:54.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.966+0000 7fe526c00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe52010de60 con 0x7fe5201080b0 2024-05-22T23:59:54.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.966+0000 7fe526c00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe52010e160 con 0x7fe520102e50 2024-05-22T23:59:54.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe517400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520105780 0x7fe520198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:54.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe517400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520105780 0x7fe520198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54260/0 (socket says 172.21.15.103:54260) 2024-05-22T23:59:54.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe517400700 1 -- 172.21.15.103:0/42122144 learned_addr learned my addr 172.21.15.103:0/42122144 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:54.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe51fe00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe520102e50 0x7fe520198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:54.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe51fe00700 1 -- 172.21.15.103:0/42122144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5201080b0 msgr2=0x7fe52019fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe51fe00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5201080b0 0x7fe52019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe51fe00700 1 -- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520105780 msgr2=0x7fe520198b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe51fe00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520105780 0x7fe520198b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe51fe00700 1 -- 172.21.15.103:0/42122144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5201a0290 con 0x7fe520102e50 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.967+0000 7fe51fe00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe520102e50 0x7fe520198630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe51000ea20 tx=0x7fe51000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.968+0000 7fe51d600700 1 -- 172.21.15.103:0/42122144 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510004220 con 0x7fe520102e50 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.968+0000 7fe51d600700 1 -- 172.21.15.103:0/42122144 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe51000ce50 con 0x7fe520102e50 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.968+0000 7fe51d600700 1 -- 172.21.15.103:0/42122144 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe510010430 con 0x7fe520102e50 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.968+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5201a0580 con 0x7fe520102e50 2024-05-22T23:59:54.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.968+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5201a0b10 con 0x7fe520102e50 2024-05-22T23:59:54.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.969+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe520069050 con 0x7fe520102e50 2024-05-22T23:59:54.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.971+0000 7fe51d600700 1 -- 172.21.15.103:0/42122144 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fe51000c970 con 0x7fe520102e50 2024-05-22T23:59:54.975 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.973+0000 7fe51d600700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe5040a2ca0 0x7fe5040a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:54.975 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.973+0000 7fe51d600700 1 -- 172.21.15.103:0/42122144 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe5100cf700 con 0x7fe520102e50 2024-05-22T23:59:54.975 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.973+0000 7fe517400700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe5040a2ca0 0x7fe5040a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:54.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.974+0000 7fe517400700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe5040a2ca0 0x7fe5040a5160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe508009d60 tx=0x7fe508009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:54.977 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:54.975+0000 7fe51d600700 1 -- 172.21.15.103:0/42122144 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe51009c330 con 0x7fe520102e50 2024-05-22T23:59:55.212 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.209+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe5200627a0 con 0x7fe5040a2ca0 2024-05-22T23:59:55.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.228+0000 7fe51d600700 1 -- 172.21.15.103:0/42122144 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fe5200627a0 con 0x7fe5040a2ca0 2024-05-22T23:59:55.301 INFO:teuthology.orchestra.run.smithi103.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-22T23:59:55.301 INFO:teuthology.orchestra.run.smithi103.stdout:mds.a smithi103 running (2m) 27s ago 2m 16.4M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 4cf31a575965 2024-05-22T23:59:55.301 INFO:teuthology.orchestra.run.smithi103.stdout:mds.b smithi105 running (2m) 79s ago 2m 28.2M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 76c9c4314656 2024-05-22T23:59:55.301 INFO:teuthology.orchestra.run.smithi103.stdout:mds.c smithi174 running (2m) 79s ago 2m 16.2M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 376efd3c5777 2024-05-22T23:59:55.301 INFO:teuthology.orchestra.run.smithi103.stdout:mds.d smithi103 running (2m) 27s ago 2m 12.5M - 19.0.0-2765-g2e1f7622 f0ae8441b76c ae06e094c8ba 2024-05-22T23:59:55.301 INFO:teuthology.orchestra.run.smithi103.stdout:mds.e smithi105 running (2m) 79s ago 2m 13.6M - 19.0.0-2765-g2e1f7622 f0ae8441b76c d880e172c126 2024-05-22T23:59:55.301 INFO:teuthology.orchestra.run.smithi103.stdout:mds.f smithi174 running (2m) 79s ago 2m 17.3M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 89db9aeff43f 2024-05-22T23:59:55.302 INFO:teuthology.orchestra.run.smithi103.stdout:mds.g smithi103 running (2m) 27s ago 2m 12.5M - 19.0.0-2765-g2e1f7622 f0ae8441b76c df066456361c 2024-05-22T23:59:55.302 INFO:teuthology.orchestra.run.smithi103.stdout:mds.h smithi105 running (2m) 79s ago 2m 15.5M - 19.0.0-2765-g2e1f7622 f0ae8441b76c bdac9b6bbd8f 2024-05-22T23:59:55.302 INFO:teuthology.orchestra.run.smithi103.stdout:mds.i smithi174 running (2m) 79s ago 2m 17.3M - 19.0.0-2765-g2e1f7622 f0ae8441b76c da829a48158a 2024-05-22T23:59:55.302 INFO:teuthology.orchestra.run.smithi103.stdout:mds.j smithi103 running (2m) 27s ago 2m 12.5M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 7bdcb80f9abd 2024-05-22T23:59:55.302 INFO:teuthology.orchestra.run.smithi103.stdout:mds.k smithi105 running (3m) 79s ago 3m 13.3M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 6b1c153f7aec 2024-05-22T23:59:55.302 INFO:teuthology.orchestra.run.smithi103.stdout:mds.l smithi174 running (3m) 79s ago 3m 14.2M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 3c1f3b8310c0 2024-05-22T23:59:55.302 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.x smithi103 *:9283,8765 running (11m) 27s ago 11m 517M - 19.0.0-2765-g2e1f7622 f0ae8441b76c d82c41646a04 2024-05-22T23:59:55.303 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.y smithi105 *:8443,8765 running (9m) 79s ago 9m 453M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 4c2a1c3611d0 2024-05-22T23:59:55.303 INFO:teuthology.orchestra.run.smithi103.stdout:mgr.z smithi174 *:8443,8765 running (9m) 79s ago 9m 456M - 19.0.0-2765-g2e1f7622 f0ae8441b76c 9e6c9e4eb710 2024-05-22T23:59:55.303 INFO:teuthology.orchestra.run.smithi103.stdout:mon.a smithi103 running (11m) 27s ago 11m 47.9M 2048M 19.0.0-2765-g2e1f7622 f0ae8441b76c 5991f9961eca 2024-05-22T23:59:55.303 INFO:teuthology.orchestra.run.smithi103.stdout:mon.b smithi105 running (9m) 79s ago 9m 42.9M 2048M 19.0.0-2765-g2e1f7622 f0ae8441b76c 7d4d62ba0f8d 2024-05-22T23:59:55.303 INFO:teuthology.orchestra.run.smithi103.stdout:mon.c smithi174 running (9m) 79s ago 9m 41.3M 2048M 19.0.0-2765-g2e1f7622 f0ae8441b76c bf01ac9e5176 2024-05-22T23:59:55.303 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 smithi103 running (8m) 27s ago 8m 44.7M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 577f5ffd4f07 2024-05-22T23:59:55.303 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 smithi105 running (8m) 79s ago 8m 47.1M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 4eafd9b0928f 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 smithi174 running (7m) 79s ago 7m 46.0M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 38aa374a3e9f 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 smithi103 running (7m) 27s ago 7m 52.9M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 090f8a531765 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 smithi105 running (6m) 79s ago 6m 43.3M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 75bd44b718d2 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 smithi174 running (6m) 79s ago 6m 56.7M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 5f375d34cb07 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6 smithi103 running (5m) 27s ago 5m 41.9M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 1e337aa6ef12 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7 smithi105 running (5m) 79s ago 5m 55.9M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 37eb69b1735b 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8 smithi174 running (4m) 79s ago 4m 46.8M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 42ef6b29fbd0 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9 smithi103 running (4m) 27s ago 4m 53.3M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 7febc895a62d 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10 smithi105 running (3m) 79s ago 3m 56.5M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c ddb2ee1f63da 2024-05-22T23:59:55.304 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11 smithi174 running (3m) 79s ago 3m 41.8M 4272M 19.0.0-2765-g2e1f7622 f0ae8441b76c 04e2fd0ef8cc 2024-05-22T23:59:55.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe5040a2ca0 msgr2=0x7fe5040a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:55.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe5040a2ca0 0x7fe5040a5160 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe508009d60 tx=0x7fe508009260 comp rx=0 tx=0).stop 2024-05-22T23:59:55.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe520102e50 msgr2=0x7fe520198630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:55.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe520102e50 0x7fe520198630 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe51000ea20 tx=0x7fe51000ed30 comp rx=0 tx=0).stop 2024-05-22T23:59:55.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 shutdown_connections 2024-05-22T23:59:55.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe5201080b0 0x7fe52019fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:55.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe5040a2ca0 0x7fe5040a5160 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:55.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe520102e50 0x7fe520198630 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:55.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 --2- 172.21.15.103:0/42122144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe520105780 0x7fe520198b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:55.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 >> 172.21.15.103:0/42122144 conn(0x7fe5200fc860 msgr2=0x7fe520103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:55.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.232+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 shutdown_connections 2024-05-22T23:59:55.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:55.233+0000 7fe526c00700 1 -- 172.21.15.103:0/42122144 wait complete. 2024-05-22T23:59:56.067 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-22T23:59:56.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:56 smithi103 ceph-mon[31370]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:56.538 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-22T23:59:56.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:56 smithi105 ceph-mon[36426]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:56.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:56 smithi174 ceph-mon[36179]: from='client.25213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:57.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:57 smithi105 ceph-mon[36426]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:57.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:57 smithi103 ceph-mon[31370]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:57.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:57 smithi174 ceph-mon[36179]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-22T23:59:58.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 -- 172.21.15.103:0/1688332388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec1060a0 msgr2=0x7fbaec10c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:58.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1688332388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec1060a0 0x7fbaec10c950 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbae400b0d0 tx=0x7fbae400b3e0 comp rx=0 tx=0).stop 2024-05-22T23:59:58.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 -- 172.21.15.103:0/1688332388 shutdown_connections 2024-05-22T23:59:58.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1688332388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec1060a0 0x7fbaec10c950 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:58.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1688332388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaec104500 0x7fbaec104920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:58.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1688332388 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbaec105700 0x7fbaec105b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:58.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 -- 172.21.15.103:0/1688332388 >> 172.21.15.103:0/1688332388 conn(0x7fbaec0ffaa0 msgr2=0x7fbaec101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:58.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 -- 172.21.15.103:0/1688332388 shutdown_connections 2024-05-22T23:59:58.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.838+0000 7fbaf2000700 1 -- 172.21.15.103:0/1688332388 wait complete. 2024-05-22T23:59:58.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.839+0000 7fbaf2000700 1 Processor -- start 2024-05-22T23:59:58.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.839+0000 7fbaf2000700 1 -- start start 2024-05-22T23:59:58.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf2000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec104500 0x7fbaec19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:58.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf2000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaec105700 0x7fbaec19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:58.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf2000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbaec1060a0 0x7fbaec1a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:58.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf2000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec1100d0 con 0x7fbaec104500 2024-05-22T23:59:58.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf2000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec10ff50 con 0x7fbaec1060a0 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf2000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec110250 con 0x7fbaec105700 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf0c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec104500 0x7fbaec19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf0c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec104500 0x7fbaec19a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:54276/0 (socket says 172.21.15.103:54276) 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf0c00700 1 -- 172.21.15.103:0/1064806592 learned_addr learned my addr 172.21.15.103:0/1064806592 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaebe00700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaec105700 0x7fbaec19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf1600700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbaec1060a0 0x7fbaec1a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf0c00700 1 -- 172.21.15.103:0/1064806592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaec105700 msgr2=0x7fbaec19ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.840+0000 7fbaf0c00700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaec105700 0x7fbaec19ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbaf0c00700 1 -- 172.21.15.103:0/1064806592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbaec1060a0 msgr2=0x7fbaec1a1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbaf0c00700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbaec1060a0 0x7fbaec1a1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbaf0c00700 1 -- 172.21.15.103:0/1064806592 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec1a2330 con 0x7fbaec104500 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbaf1600700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbaec1060a0 0x7fbaec1a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-22T23:59:58.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbaf0c00700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec104500 0x7fbaec19a6d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbae000e9a0 tx=0x7fbae000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:58.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbae9600700 1 -- 172.21.15.103:0/1064806592 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae000cbd0 con 0x7fbaec104500 2024-05-22T23:59:58.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbae9600700 1 -- 172.21.15.103:0/1064806592 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbae0004500 con 0x7fbaec104500 2024-05-22T23:59:58.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbae9600700 1 -- 172.21.15.103:0/1064806592 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0003e00 con 0x7fbaec104500 2024-05-22T23:59:58.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaec1a2620 con 0x7fbaec104500 2024-05-22T23:59:58.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.841+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaec1a2bb0 con 0x7fbaec104500 2024-05-22T23:59:58.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.842+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaec069050 con 0x7fbaec104500 2024-05-22T23:59:58.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.844+0000 7fbae9600700 1 -- 172.21.15.103:0/1064806592 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fbae0004000 con 0x7fbaec104500 2024-05-22T23:59:58.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.846+0000 7fbae9600700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbad40a2ca0 0x7fbad40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-22T23:59:58.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.846+0000 7fbae9600700 1 -- 172.21.15.103:0/1064806592 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fbae00cf6a0 con 0x7fbaec104500 2024-05-22T23:59:58.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.846+0000 7fbaebe00700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbad40a2ca0 0x7fbad40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-22T23:59:58.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.847+0000 7fbaebe00700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbad40a2ca0 0x7fbad40a5160 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009be0 tx=0x7fbad8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-22T23:59:58.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:58.850+0000 7fbae9600700 1 -- 172.21.15.103:0/1064806592 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fbae0016330 con 0x7fbaec104500 2024-05-22T23:59:59.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.089+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fbaec0627a0 con 0x7fbad40a2ca0 2024-05-22T23:59:59.101 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.099+0000 7fbae9600700 1 -- 172.21.15.103:0/1064806592 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fbaec0627a0 con 0x7fbad40a2ca0 2024-05-22T23:59:59.101 INFO:teuthology.orchestra.run.smithi103.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-22T23:59:59.102 INFO:teuthology.orchestra.run.smithi103.stdout:mds.all 12/12 83s ago 3m smithi103=a;smithi103=d;smithi103=g;smithi103=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi174=c;smithi174=f;smithi174=i;smithi174=l;count:12 2024-05-22T23:59:59.102 INFO:teuthology.orchestra.run.smithi103.stdout:mgr 3/3 83s ago 9m smithi103=x;smithi105=y;smithi174=z;count:3 2024-05-22T23:59:59.102 INFO:teuthology.orchestra.run.smithi103.stdout:mon 3/3 83s ago 9m smithi103:172.21.15.103=a;smithi105:172.21.15.105=b;smithi174:172.21.15.174=c;count:3 2024-05-22T23:59:59.102 INFO:teuthology.orchestra.run.smithi103.stdout:osd 12 83s ago - 2024-05-22T23:59:59.104 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbad40a2ca0 msgr2=0x7fbad40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:59.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbad40a2ca0 0x7fbad40a5160 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fbad8009be0 tx=0x7fbad8009260 comp rx=0 tx=0).stop 2024-05-22T23:59:59.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec104500 msgr2=0x7fbaec19a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-22T23:59:59.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec104500 0x7fbaec19a6d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbae000e9a0 tx=0x7fbae000ecb0 comp rx=0 tx=0).stop 2024-05-22T23:59:59.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 shutdown_connections 2024-05-22T23:59:59.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbaec1060a0 0x7fbaec1a1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:59.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fbad40a2ca0 0x7fbad40a5160 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:59.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbaec104500 0x7fbaec19a6d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:59.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 --2- 172.21.15.103:0/1064806592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbaec105700 0x7fbaec19ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-22T23:59:59.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.102+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 >> 172.21.15.103:0/1064806592 conn(0x7fbaec0ffaa0 msgr2=0x7fbaec108930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-22T23:59:59.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.103+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 shutdown_connections 2024-05-22T23:59:59.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-22T23:59:59.103+0000 7fbaf2000700 1 -- 172.21.15.103:0/1064806592 wait complete. 2024-05-22T23:59:59.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:59 smithi103 ceph-mon[31370]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:59.517 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 22 23:59:59 smithi103 ceph-mon[31370]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:59.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:59 smithi105 ceph-mon[36426]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:59.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 22 23:59:59 smithi105 ceph-mon[36426]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:59.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:59 smithi174 ceph-mon[36179]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-22T23:59:59.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 22 23:59:59 smithi174 ceph-mon[36179]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-22T23:59:59.926 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-23T00:00:00.371 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:00:00.519 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:00 smithi103 ceph-mon[31370]: overall HEALTH_OK 2024-05-23T00:00:00.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:00 smithi105 ceph-mon[36426]: overall HEALTH_OK 2024-05-23T00:00:00.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:00 smithi174 ceph-mon[36179]: overall HEALTH_OK 2024-05-23T00:00:01.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:01 smithi103 ceph-mon[31370]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:01.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:01 smithi105 ceph-mon[36426]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:01.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:01 smithi174 ceph-mon[36179]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:02.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 -- 172.21.15.103:0/243765771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14181015a0 msgr2=0x7f1418101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:02.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 --2- 172.21.15.103:0/243765771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14181015a0 0x7f1418101a20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1404009990 tx=0x7f1404009ca0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 -- 172.21.15.103:0/243765771 shutdown_connections 2024-05-23T00:00:02.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 --2- 172.21.15.103:0/243765771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418100c40 0x7f1418101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 --2- 172.21.15.103:0/243765771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1418101f60 0x7f14181023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 --2- 172.21.15.103:0/243765771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f14181015a0 0x7f1418101a20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 -- 172.21.15.103:0/243765771 >> 172.21.15.103:0/243765771 conn(0x7f14180fc860 msgr2=0x7f14180feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:02.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 -- 172.21.15.103:0/243765771 shutdown_connections 2024-05-23T00:00:02.726 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.722+0000 7f141e600700 1 -- 172.21.15.103:0/243765771 wait complete. 2024-05-23T00:00:02.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.723+0000 7f141e600700 1 Processor -- start 2024-05-23T00:00:02.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f141e600700 1 -- start start 2024-05-23T00:00:02.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f141e600700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418100c40 0x7f141819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:02.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f141e600700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14181015a0 0x7f141819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f141e600700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1418101f60 0x7f14181a1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f141e600700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f14181101c0 con 0x7f1418101f60 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f141e600700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1418110040 con 0x7f1418100c40 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f141e600700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1418110340 con 0x7f14181015a0 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f1417400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418100c40 0x7f141819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.724+0000 7f1416a00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14181015a0 0x7f141819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417400700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418100c40 0x7f141819a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49114/0 (socket says 172.21.15.103:49114) 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1416a00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14181015a0 0x7f141819ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41592/0 (socket says 172.21.15.103:41592) 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417400700 1 -- 172.21.15.103:0/3850472041 learned_addr learned my addr 172.21.15.103:0/3850472041 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417e00700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1418101f60 0x7f14181a1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417400700 1 -- 172.21.15.103:0/3850472041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14181015a0 msgr2=0x7f141819ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417400700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14181015a0 0x7f141819ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417400700 1 -- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1418101f60 msgr2=0x7f14181a1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417400700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1418101f60 0x7f14181a1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417400700 1 -- 172.21.15.103:0/3850472041 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14181a24c0 con 0x7f1418100c40 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417e00700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1418101f60 0x7f14181a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.725+0000 7f1417400700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418100c40 0x7f141819a860 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f140800e9a0 tx=0x7f140800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.726+0000 7f140fe00700 1 -- 172.21.15.103:0/3850472041 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140800cbd0 con 0x7f1418100c40 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.726+0000 7f140fe00700 1 -- 172.21.15.103:0/3850472041 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140800cd70 con 0x7f1418100c40 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.726+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14181a27b0 con 0x7f1418100c40 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.726+0000 7f140fe00700 1 -- 172.21.15.103:0/3850472041 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080187a0 con 0x7f1418100c40 2024-05-23T00:00:02.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.726+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14181a2cf0 con 0x7f1418100c40 2024-05-23T00:00:02.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.727+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1418069050 con 0x7f1418100c40 2024-05-23T00:00:02.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.729+0000 7f140fe00700 1 -- 172.21.15.103:0/3850472041 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f14080109a0 con 0x7f1418100c40 2024-05-23T00:00:02.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.731+0000 7f140fe00700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13fc0a2c20 0x7f13fc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:02.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.731+0000 7f140fe00700 1 -- 172.21.15.103:0/3850472041 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f14080cf9b0 con 0x7f1418100c40 2024-05-23T00:00:02.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.731+0000 7f1416a00700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13fc0a2c20 0x7f13fc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:02.734 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.732+0000 7f1416a00700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13fc0a2c20 0x7f13fc0a50e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f140400b2e0 tx=0x7f140400b910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:02.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.733+0000 7f140fe00700 1 -- 172.21.15.103:0/3850472041 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f140809c550 con 0x7f1418100c40 2024-05-23T00:00:02.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.963+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f14180627a0 con 0x7f13fc0a2c20 2024-05-23T00:00:02.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.965+0000 7f140fe00700 1 -- 172.21.15.103:0/3850472041 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f14180627a0 con 0x7f13fc0a2c20 2024-05-23T00:00:02.968 INFO:teuthology.orchestra.run.smithi103.stdout:HOST ADDR LABELS STATUS 2024-05-23T00:00:02.968 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 172.21.15.103 2024-05-23T00:00:02.968 INFO:teuthology.orchestra.run.smithi103.stdout:smithi105 172.21.15.105 2024-05-23T00:00:02.969 INFO:teuthology.orchestra.run.smithi103.stdout:smithi174 172.21.15.174 2024-05-23T00:00:02.969 INFO:teuthology.orchestra.run.smithi103.stdout:3 hosts in cluster 2024-05-23T00:00:02.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13fc0a2c20 msgr2=0x7f13fc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:02.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13fc0a2c20 0x7f13fc0a50e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f140400b2e0 tx=0x7f140400b910 comp rx=0 tx=0).stop 2024-05-23T00:00:02.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418100c40 msgr2=0x7f141819a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:02.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418100c40 0x7f141819a860 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f140800e9a0 tx=0x7f140800ecb0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 shutdown_connections 2024-05-23T00:00:02.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418100c40 0x7f141819a860 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f13fc0a2c20 0x7f13fc0a50e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1418101f60 0x7f14181a1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 --2- 172.21.15.103:0/3850472041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14181015a0 0x7f141819ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:02.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.968+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 >> 172.21.15.103:0/3850472041 conn(0x7f14180fc860 msgr2=0x7f14180fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:02.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.969+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 shutdown_connections 2024-05-23T00:00:02.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:02.969+0000 7f141e600700 1 -- 172.21.15.103:0/3850472041 wait complete. 2024-05-23T00:00:03.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:03 smithi105 ceph-mon[36426]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:03.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:03 smithi105 ceph-mon[36426]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:03.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:03 smithi103 ceph-mon[31370]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:03.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:03 smithi103 ceph-mon[31370]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:03.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:03 smithi174 ceph-mon[36179]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:03.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:03 smithi174 ceph-mon[36179]: from='client.25214 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:03.843 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-23T00:00:04.307 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:00:05.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:05 smithi105 ceph-mon[36426]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-23T00:00:05.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:05 smithi103 ceph-mon[31370]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-23T00:00:05.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:05 smithi174 ceph-mon[36179]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 -- 172.21.15.103:0/905893172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a38102e50 msgr2=0x7f8a38105240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/905893172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a38102e50 0x7f8a38105240 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009990 tx=0x7f8a28009ca0 comp rx=0 tx=0).stop 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 -- 172.21.15.103:0/905893172 shutdown_connections 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/905893172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a38102e50 0x7f8a38105240 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/905893172 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a381080b0 0x7f8a3810a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/905893172 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a38105780 0x7f8a38107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 -- 172.21.15.103:0/905893172 >> 172.21.15.103:0/905893172 conn(0x7f8a380fc860 msgr2=0x7f8a380feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 -- 172.21.15.103:0/905893172 shutdown_connections 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.843+0000 7f8a3ec00700 1 -- 172.21.15.103:0/905893172 wait complete. 2024-05-23T00:00:06.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.844+0000 7f8a3ec00700 1 Processor -- start 2024-05-23T00:00:06.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.844+0000 7f8a3ec00700 1 -- start start 2024-05-23T00:00:06.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a3ec00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a38102e50 0x7f8a38198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:06.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a3ec00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a38105780 0x7f8a38198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:06.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a3ec00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a381080b0 0x7f8a3819fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:06.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a3ec00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810dfe0 con 0x7f8a38102e50 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a3ec00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810de60 con 0x7f8a381080b0 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a3ec00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3810e160 con 0x7f8a38105780 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a37e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a38102e50 0x7f8a38198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a37e00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a38102e50 0x7f8a38198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51036/0 (socket says 172.21.15.103:51036) 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a37e00700 1 -- 172.21.15.103:0/3799458681 learned_addr learned my addr 172.21.15.103:0/3799458681 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a2f400700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a38105780 0x7f8a38198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.845+0000 7f8a3cc00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a381080b0 0x7f8a3819fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a2f400700 1 -- 172.21.15.103:0/3799458681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a381080b0 msgr2=0x7f8a3819fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a2f400700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a381080b0 0x7f8a3819fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a2f400700 1 -- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a38102e50 msgr2=0x7f8a38198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:06.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a2f400700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a38102e50 0x7f8a38198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a2f400700 1 -- 172.21.15.103:0/3799458681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a381a02e0 con 0x7f8a38105780 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a37e00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a38102e50 0x7f8a38198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a2f400700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a38105780 0x7f8a38198bc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000ea20 tx=0x7f8a2000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a35600700 1 -- 172.21.15.103:0/3799458681 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2000cbd0 con 0x7f8a38105780 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a381a05d0 con 0x7f8a38105780 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a35600700 1 -- 172.21.15.103:0/3799458681 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a2000ee60 con 0x7f8a38105780 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a35600700 1 -- 172.21.15.103:0/3799458681 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a20018680 con 0x7f8a38105780 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.846+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a381a0b60 con 0x7f8a38105780 2024-05-23T00:00:06.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.847+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a38069050 con 0x7f8a38105780 2024-05-23T00:00:06.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.850+0000 7f8a35600700 1 -- 172.21.15.103:0/3799458681 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f8a20018820 con 0x7f8a38105780 2024-05-23T00:00:06.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.851+0000 7f8a35600700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8a1c0a2c20 0x7f8a1c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:06.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.851+0000 7f8a35600700 1 -- 172.21.15.103:0/3799458681 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8a200cf550 con 0x7f8a38105780 2024-05-23T00:00:06.853 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.851+0000 7f8a37e00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8a1c0a2c20 0x7f8a1c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:06.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.852+0000 7f8a37e00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8a1c0a2c20 0x7f8a1c0a50e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800b7b0 tx=0x7f8a2800b8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:06.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:06.855+0000 7f8a35600700 1 -- 172.21.15.103:0/3799458681 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8a2009c0f0 con 0x7f8a38105780 2024-05-23T00:00:07.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.095+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 --> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f8a380627a0 con 0x7f8a1c0a2c20 2024-05-23T00:00:07.102 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.100+0000 7f8a35600700 1 -- 172.21.15.103:0/3799458681 <== mgr.14152 v2:172.21.15.103:6800/1744216601 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f8a380627a0 con 0x7f8a1c0a2c20 2024-05-23T00:00:07.103 INFO:teuthology.orchestra.run.smithi103.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-23T00:00:07.103 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204012Z400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-23T00:00:07.103 INFO:teuthology.orchestra.run.smithi103.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-23T00:00:07.103 INFO:teuthology.orchestra.run.smithi103.stdout:smithi174 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050074400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-23T00:00:07.105 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8a1c0a2c20 msgr2=0x7f8a1c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8a1c0a2c20 0x7f8a1c0a50e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800b7b0 tx=0x7f8a2800b8a0 comp rx=0 tx=0).stop 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a38105780 msgr2=0x7f8a38198bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a38105780 0x7f8a38198bc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8a2000ea20 tx=0x7f8a2000ed30 comp rx=0 tx=0).stop 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 shutdown_connections 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a381080b0 0x7f8a3819fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8a1c0a2c20 0x7f8a1c0a50e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a38102e50 0x7f8a38198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 --2- 172.21.15.103:0/3799458681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a38105780 0x7f8a38198bc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 >> 172.21.15.103:0/3799458681 conn(0x7f8a380fc860 msgr2=0x7f8a381038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 shutdown_connections 2024-05-23T00:00:07.106 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:07.103+0000 7f8a3ec00700 1 -- 172.21.15.103:0/3799458681 wait complete. 2024-05-23T00:00:07.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:07 smithi105 ceph-mon[36426]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-23T00:00:07.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:07 smithi105 ceph-mon[36426]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:07.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:07 smithi103 ceph-mon[31370]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-23T00:00:07.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:07 smithi103 ceph-mon[31370]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:07.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:07 smithi174 ceph-mon[36179]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-23T00:00:07.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:07 smithi174 ceph-mon[36179]: from='client.25231 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:08.166 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-23T00:00:08.179 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:00:08.179 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-23T00:00:08.624 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:00:09.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:09 smithi103 ceph-mon[31370]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:00:09.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:09 smithi105 ceph-mon[36426]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:00:09.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:09 smithi174 ceph-mon[36179]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:00:11.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 -- 172.21.15.103:0/1812925378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec1080a0 msgr2=0x7f2cec10a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:11.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1812925378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec1080a0 0x7f2cec10a510 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4009f90 tx=0x7f2ce400b330 comp rx=0 tx=0).stop 2024-05-23T00:00:11.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 -- 172.21.15.103:0/1812925378 shutdown_connections 2024-05-23T00:00:11.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1812925378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cec102e40 0x7f2cec105230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1812925378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec1080a0 0x7f2cec10a510 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1812925378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cec105770 0x7f2cec107b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 -- 172.21.15.103:0/1812925378 >> 172.21.15.103:0/1812925378 conn(0x7f2cec0fc850 msgr2=0x7f2cec0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:11.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 -- 172.21.15.103:0/1812925378 shutdown_connections 2024-05-23T00:00:11.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.003+0000 7f2cf2200700 1 -- 172.21.15.103:0/1812925378 wait complete. 2024-05-23T00:00:11.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.004+0000 7f2cf2200700 1 Processor -- start 2024-05-23T00:00:11.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.004+0000 7f2cf2200700 1 -- start start 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2cf2200700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec102e40 0x7f2cec198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2cf2200700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cec105770 0x7f2cec198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2cf2200700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cec1080a0 0x7f2cec19fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2cf2200700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec10df90 con 0x7f2cec102e40 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2cf2200700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec10de10 con 0x7f2cec1080a0 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2cf2200700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2cec10e110 con 0x7f2cec105770 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2ce2a00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cec105770 0x7f2cec198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2ce2a00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cec105770 0x7f2cec198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:44566/0 (socket says 172.21.15.103:44566) 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2ce2a00700 1 -- 172.21.15.103:0/1164155118 learned_addr learned my addr 172.21.15.103:0/1164155118 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2cebe00700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cec1080a0 0x7f2cec19fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.005+0000 7f2ceb400700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec102e40 0x7f2cec198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:11.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2ce2a00700 1 -- 172.21.15.103:0/1164155118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cec1080a0 msgr2=0x7f2cec19fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2ce2a00700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cec1080a0 0x7f2cec19fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2ce2a00700 1 -- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec102e40 msgr2=0x7f2cec198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2ce2a00700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec102e40 0x7f2cec198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2ce2a00700 1 -- 172.21.15.103:0/1164155118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cec1a0180 con 0x7f2cec105770 2024-05-23T00:00:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2cebe00700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cec1080a0 0x7f2cec19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:11.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2ce2a00700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cec105770 0x7f2cec198a60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400e9d0 tx=0x7f2cd400ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2ce8c00700 1 -- 172.21.15.103:0/1164155118 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd400cbd0 con 0x7f2cec105770 2024-05-23T00:00:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2ce8c00700 1 -- 172.21.15.103:0/1164155118 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd400ee70 con 0x7f2cec105770 2024-05-23T00:00:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.006+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cec1a0470 con 0x7f2cec105770 2024-05-23T00:00:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.007+0000 7f2ce8c00700 1 -- 172.21.15.103:0/1164155118 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4018660 con 0x7f2cec105770 2024-05-23T00:00:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.007+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cec1a09b0 con 0x7f2cec105770 2024-05-23T00:00:11.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.008+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cec069050 con 0x7f2cec105770 2024-05-23T00:00:11.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.010+0000 7f2ce8c00700 1 -- 172.21.15.103:0/1164155118 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f2cd4010430 con 0x7f2cec105770 2024-05-23T00:00:11.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.011+0000 7f2ce8c00700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2cd00a2c20 0x7f2cd00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:11.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.011+0000 7f2ce8c00700 1 -- 172.21.15.103:0/1164155118 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f2cd40cf960 con 0x7f2cec105770 2024-05-23T00:00:11.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.011+0000 7f2ceb400700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2cd00a2c20 0x7f2cd00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:11.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.012+0000 7f2ceb400700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2cd00a2c20 0x7f2cd00a50e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009750 tx=0x7f2cdc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:11.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.014+0000 7f2ce8c00700 1 -- 172.21.15.103:0/1164155118 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2cd409c500 con 0x7f2cec105770 2024-05-23T00:00:11.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.286+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f2cec02dbf0 con 0x7f2cec105770 2024-05-23T00:00:11.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.287+0000 7f2ce8c00700 1 -- 172.21.15.103:0/1164155118 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3068 (secure 0 0 0) 0x7f2cd400cd70 con 0x7f2cec105770 2024-05-23T00:00:11.290 INFO:teuthology.orchestra.run.smithi103.stdout:e24 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:legacy client fscid: 1 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:Filesystem 'cephfs' (1) 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:fs_name cephfs 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 24 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-05-22T23:57:32.804111+0000 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-05-22T23:58:05.260333+0000 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:tableserver 0 2024-05-23T00:00:11.291 INFO:teuthology.orchestra.run.smithi103.stdout:root 0 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:session_timeout 300 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:session_autoclose 300 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:max_file_size 1099511627776 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:max_xattr_size 65536 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:required_client_features {} 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure 0 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:last_failure_osd_epoch 0 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:max_mds 3 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:in 0,1,2 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:up {0=24575,1=24563,2=24559} 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:failed 2024-05-23T00:00:11.292 INFO:teuthology.orchestra.run.smithi103.stdout:damaged 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:stopped 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:data_pools [3,4] 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:metadata_pool 2 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:inline_data disabled 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:balancer 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:bal_rank_mask -1 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:standby_count_wanted 0 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:qdb_cluster leader: 24575 members: 24559,24563,24575 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.b{0:24575} state up:active seq 3 addr [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.c{0:24535} state up:standby-replay seq 1 addr [v2:172.21.15.174:6836/2641083628,v1:172.21.15.174:6837/2641083628] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.a{1:24563} state up:active seq 11 addr [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.f{1:24523} state up:standby-replay seq 1 addr [v2:172.21.15.174:6834/3839469517,v1:172.21.15.174:6835/3839469517] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.293 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.i{2:24559} state up:active seq 14 addr [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.h{2:24515} state up:standby-replay seq 1 addr [v2:172.21.15.105:6836/2433750399,v1:172.21.15.105:6837/2433750399] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout:Standby daemons: 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.d{-1:14406} state up:standby seq 1 addr [v2:172.21.15.103:6836/1586986651,v1:172.21.15.103:6837/1586986651] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.g{-1:14454} state up:standby seq 1 addr [v2:172.21.15.103:6838/2610863022,v1:172.21.15.103:6839/2610863022] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.k{-1:24440} state up:standby seq 1 addr [v2:172.21.15.105:6832/1828383607,v1:172.21.15.105:6833/1828383607] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.l{-1:24457} state up:standby seq 1 addr [v2:172.21.15.174:6832/1973931443,v1:172.21.15.174:6833/1973931443] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.j{-1:24458} state up:standby seq 1 addr [v2:172.21.15.103:6834/1622172738,v1:172.21.15.103:6835/1622172738] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.294 INFO:teuthology.orchestra.run.smithi103.stdout:[mds.e{-1:24488} state up:standby seq 1 addr [v2:172.21.15.105:6834/1500378287,v1:172.21.15.105:6835/1500378287] compat {c=[1],r=[1],i=[1fff]}] 2024-05-23T00:00:11.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2cd00a2c20 msgr2=0x7f2cd00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:11.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2cd00a2c20 0x7f2cd00a50e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc009750 tx=0x7f2cdc009310 comp rx=0 tx=0).stop 2024-05-23T00:00:11.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cec105770 msgr2=0x7f2cec198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:11.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cec105770 0x7f2cec198a60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400e9d0 tx=0x7f2cd400ece0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 shutdown_connections 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cec1080a0 0x7f2cec19fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f2cd00a2c20 0x7f2cd00a50e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cec102e40 0x7f2cec198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 --2- 172.21.15.103:0/1164155118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2cec105770 0x7f2cec198a60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 >> 172.21.15.103:0/1164155118 conn(0x7f2cec0fc850 msgr2=0x7f2cec103780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 shutdown_connections 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:11.290+0000 7f2cf2200700 1 -- 172.21.15.103:0/1164155118 wait complete. 2024-05-23T00:00:11.296 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 24 2024-05-23T00:00:11.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:11 smithi105 ceph-mon[36426]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:11.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:11 smithi103 ceph-mon[31370]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:11.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:11 smithi174 ceph-mon[36179]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:12.142 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-23T00:00:12.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:12 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1164155118' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-23T00:00:12.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:12 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1164155118' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-23T00:00:12.585 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:00:12.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:12 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1164155118' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-23T00:00:13.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:13 smithi105 ceph-mon[36426]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:13.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:13 smithi103 ceph-mon[31370]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:13.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:13 smithi174 ceph-mon[36179]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:14.975 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.971+0000 7f1292000700 1 -- 172.21.15.103:0/3760786456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c1061a0 msgr2=0x7f128c10ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.971+0000 7f1292000700 1 --2- 172.21.15.103:0/3760786456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c1061a0 0x7f128c10ca50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f12840099c0 tx=0x7f1284009cd0 comp rx=0 tx=0).stop 2024-05-23T00:00:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.972+0000 7f1292000700 1 -- 172.21.15.103:0/3760786456 shutdown_connections 2024-05-23T00:00:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.972+0000 7f1292000700 1 --2- 172.21.15.103:0/3760786456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c1061a0 0x7f128c10ca50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.972+0000 7f1292000700 1 --2- 172.21.15.103:0/3760786456 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f128c104600 0x7f128c104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.972+0000 7f1292000700 1 --2- 172.21.15.103:0/3760786456 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f128c105800 0x7f128c105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.972+0000 7f1292000700 1 -- 172.21.15.103:0/3760786456 >> 172.21.15.103:0/3760786456 conn(0x7f128c0ffbc0 msgr2=0x7f128c101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.972+0000 7f1292000700 1 -- 172.21.15.103:0/3760786456 shutdown_connections 2024-05-23T00:00:14.976 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.972+0000 7f1292000700 1 -- 172.21.15.103:0/3760786456 wait complete. 2024-05-23T00:00:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.973+0000 7f1292000700 1 Processor -- start 2024-05-23T00:00:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.973+0000 7f1292000700 1 -- start start 2024-05-23T00:00:14.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.973+0000 7f1292000700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f128c104600 0x7f128c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.973+0000 7f1292000700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c105800 0x7f128c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.973+0000 7f1292000700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f128c1061a0 0x7f128c1a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.973+0000 7f1292000700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f128c110230 con 0x7f128c1061a0 2024-05-23T00:00:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.973+0000 7f1292000700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f128c1100b0 con 0x7f128c105800 2024-05-23T00:00:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.973+0000 7f1292000700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f128c1103b0 con 0x7f128c104600 2024-05-23T00:00:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f1282a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c105800 0x7f128c19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f1282a00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c105800 0x7f128c19acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:59242/0 (socket says 172.21.15.103:59242) 2024-05-23T00:00:14.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f1282a00700 1 -- 172.21.15.103:0/3283832246 learned_addr learned my addr 172.21.15.103:0/3283832246 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f128be00700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f128c1061a0 0x7f128c1a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f128b400700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f128c104600 0x7f128c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f128b400700 1 -- 172.21.15.103:0/3283832246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c105800 msgr2=0x7f128c19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f128b400700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c105800 0x7f128c19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f128b400700 1 -- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f128c1061a0 msgr2=0x7f128c1a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f128b400700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f128c1061a0 0x7f128c1a1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f128b400700 1 -- 172.21.15.103:0/3283832246 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f128c1a23f0 con 0x7f128c104600 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.974+0000 7f1282a00700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c105800 0x7f128c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.975+0000 7f128b400700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f128c104600 0x7f128c19a790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f127c00efc0 tx=0x7f127c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:14.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.975+0000 7f1288c00700 1 -- 172.21.15.103:0/3283832246 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127c003a00 con 0x7f128c104600 2024-05-23T00:00:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.975+0000 7f1288c00700 1 -- 172.21.15.103:0/3283832246 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f127c003ba0 con 0x7f128c104600 2024-05-23T00:00:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.975+0000 7f1288c00700 1 -- 172.21.15.103:0/3283832246 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127c004880 con 0x7f128c104600 2024-05-23T00:00:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.975+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f128c1a26e0 con 0x7f128c104600 2024-05-23T00:00:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.975+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f128c1a2c20 con 0x7f128c104600 2024-05-23T00:00:14.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.976+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f128c069050 con 0x7f128c104600 2024-05-23T00:00:14.982 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.978+0000 7f1288c00700 1 -- 172.21.15.103:0/3283832246 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f127c012050 con 0x7f128c104600 2024-05-23T00:00:14.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.979+0000 7f1288c00700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f12700a2ca0 0x7f12700a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:14.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.980+0000 7f1288c00700 1 -- 172.21.15.103:0/3283832246 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f127c0cf3e0 con 0x7f128c104600 2024-05-23T00:00:14.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.980+0000 7f1282a00700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f12700a2ca0 0x7f12700a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:14.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.981+0000 7f1282a00700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f12700a2ca0 0x7f12700a5160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1274011550 tx=0x7f1274009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:14.985 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:14.982+0000 7f1288c00700 1 -- 172.21.15.103:0/3283832246 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f127c010040 con 0x7f128c104600 2024-05-23T00:00:15.210 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.206+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f128c062f70 con 0x7f128c104600 2024-05-23T00:00:15.211 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.207+0000 7f1288c00700 1 -- 172.21.15.103:0/3283832246 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5629 (secure 0 0 0) 0x7f127c09bf80 con 0x7f128c104600 2024-05-23T00:00:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:epoch 93 2024-05-23T00:00:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-23T00:00:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:created 2024-05-22T23:48:21.338539+0000 2024-05-23T00:00:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:modified 2024-05-22T23:58:31.088381+0000 2024-05-23T00:00:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-23T00:00:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:crush_version 27 2024-05-23T00:00:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:full_ratio 0.95 2024-05-23T00:00:15.212 INFO:teuthology.orchestra.run.smithi103.stdout:backfillfull_ratio 0.9 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:nearfull_ratio 0.85 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:require_min_compat_client luminous 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:min_compat_client luminous 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:require_osd_release squid 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:stretch_mode_enabled false 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 87 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 91 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:max_osd 12 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:osd.0 up in weight 1 up_from 8 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6802/2549120661,v1:172.21.15.103:6803/2549120661] [v2:172.21.15.103:6804/2549120661,v1:172.21.15.103:6805/2549120661] exists,up 2976753d-f629-42a4-ab40-e6c3bee4d852 2024-05-23T00:00:15.213 INFO:teuthology.orchestra.run.smithi103.stdout:osd.1 up in weight 1 up_from 13 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6800/732590607,v1:172.21.15.105:6801/732590607] [v2:172.21.15.105:6802/732590607,v1:172.21.15.105:6803/732590607] exists,up 5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.2 up in weight 1 up_from 18 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6800/3252167839,v1:172.21.15.174:6801/3252167839] [v2:172.21.15.174:6802/3252167839,v1:172.21.15.174:6803/3252167839] exists,up 63567395-a0a0-48f9-9f29-d65443b5c4ce 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.3 up in weight 1 up_from 26 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6810/29866944,v1:172.21.15.103:6811/29866944] [v2:172.21.15.103:6812/29866944,v1:172.21.15.103:6813/29866944] exists,up eea36383-d6a9-49c6-a3cb-9d55f4275529 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.4 up in weight 1 up_from 34 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6808/4091773686,v1:172.21.15.105:6809/4091773686] [v2:172.21.15.105:6810/4091773686,v1:172.21.15.105:6811/4091773686] exists,up 71550539-dfa3-4158-b4b9-4357ae474da2 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.5 up in weight 1 up_from 40 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6808/1001493350,v1:172.21.15.174:6809/1001493350] [v2:172.21.15.174:6810/1001493350,v1:172.21.15.174:6811/1001493350] exists,up d76fc01a-a243-4277-bbdc-1c31965446db 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.6 up in weight 1 up_from 48 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6818/1312276577,v1:172.21.15.103:6819/1312276577] [v2:172.21.15.103:6820/1312276577,v1:172.21.15.103:6821/1312276577] exists,up 9494aa47-b660-41dd-af98-e7c55899525c 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.7 up in weight 1 up_from 53 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6816/4084427396,v1:172.21.15.105:6817/4084427396] [v2:172.21.15.105:6818/4084427396,v1:172.21.15.105:6819/4084427396] exists,up 8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.8 up in weight 1 up_from 61 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6816/1749666610,v1:172.21.15.174:6817/1749666610] [v2:172.21.15.174:6818/1749666610,v1:172.21.15.174:6819/1749666610] exists,up ad862c3c-61c3-46da-adc1-008a4d5f5fce 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.9 up in weight 1 up_from 69 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6826/3886855673,v1:172.21.15.103:6827/3886855673] [v2:172.21.15.103:6828/3886855673,v1:172.21.15.103:6829/3886855673] exists,up 91405846-87e9-40b3-a158-ffcc2b762c2d 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.10 up in weight 1 up_from 74 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6824/3074362984,v1:172.21.15.105:6825/3074362984] [v2:172.21.15.105:6826/3074362984,v1:172.21.15.105:6827/3074362984] exists,up 32c228c2-bec7-4f25-99a9-137e0514381d 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:osd.11 up in weight 1 up_from 82 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.174:6824/1551576781,v1:172.21.15.174:6825/1551576781] [v2:172.21.15.174:6826/1551576781,v1:172.21.15.174:6827/1551576781] exists,up 6c93124e-d53a-4091-bc85-bc3dd328a43c 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 2.2 [7,11] 2024-05-23T00:00:15.214 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 2.6 [8,11] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 2.16 [2,4] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 2.1a [8,1] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 2.1b [2,4] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 3.7 [3,9] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 3.14 [4,6] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 3.1b [4,6] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 3.21 [4,8] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_upmap_items 3.37 [5,7] 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/3190009536 expires 2024-05-23T23:49:29.953234+0000 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/1448560264 expires 2024-05-23T23:49:29.953234+0000 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/274656845 expires 2024-05-23T23:48:54.926142+0000 2024-05-23T00:00:15.215 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/2141778982 expires 2024-05-23T23:48:54.926142+0000 2024-05-23T00:00:15.216 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/2592202413 expires 2024-05-23T23:48:54.926142+0000 2024-05-23T00:00:15.216 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:6801/3762323194 expires 2024-05-23T23:49:29.953234+0000 2024-05-23T00:00:15.216 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:6800/3495336128 expires 2024-05-23T23:48:54.926142+0000 2024-05-23T00:00:15.216 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:6800/3762323194 expires 2024-05-23T23:49:29.953234+0000 2024-05-23T00:00:15.216 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:6801/3495336128 expires 2024-05-23T23:48:54.926142+0000 2024-05-23T00:00:15.216 INFO:teuthology.orchestra.run.smithi103.stdout:blocklist 172.21.15.103:0/1425794392 expires 2024-05-23T23:49:29.953234+0000 2024-05-23T00:00:15.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.210+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f12700a2ca0 msgr2=0x7f12700a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:15.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.210+0000 7f1292000700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f12700a2ca0 0x7f12700a5160 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1274011550 tx=0x7f1274009260 comp rx=0 tx=0).stop 2024-05-23T00:00:15.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.210+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f128c104600 msgr2=0x7f128c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:15.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.210+0000 7f1292000700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f128c104600 0x7f128c19a790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f127c00efc0 tx=0x7f127c00c3a0 comp rx=0 tx=0).stop 2024-05-23T00:00:15.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.211+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 shutdown_connections 2024-05-23T00:00:15.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.211+0000 7f1292000700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f12700a2ca0 0x7f12700a5160 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:15.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.211+0000 7f1292000700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f128c1061a0 0x7f128c1a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:15.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.211+0000 7f1292000700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f128c104600 0x7f128c19a790 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.211+0000 7f1292000700 1 --2- 172.21.15.103:0/3283832246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f128c105800 0x7f128c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.211+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 >> 172.21.15.103:0/3283832246 conn(0x7f128c0ffbc0 msgr2=0x7f128c108aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.211+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 shutdown_connections 2024-05-23T00:00:15.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:15.211+0000 7f1292000700 1 -- 172.21.15.103:0/3283832246 wait complete. 2024-05-23T00:00:15.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:15 smithi105 ceph-mon[36426]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-23T00:00:15.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:15 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3283832246' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-23T00:00:15.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:15 smithi103 ceph-mon[31370]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-23T00:00:15.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:15 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3283832246' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-23T00:00:15.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:15 smithi174 ceph-mon[36179]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-23T00:00:15.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:15 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3283832246' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-23T00:00:16.091 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-23T00:00:16.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:00:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 -- 172.21.15.103:0/2129985022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b81018c0 msgr2=0x7f18b8101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 --2- 172.21.15.103:0/2129985022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b81018c0 0x7f18b8101ce0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009be0 tx=0x7f18b002a6c0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 -- 172.21.15.103:0/2129985022 shutdown_connections 2024-05-23T00:00:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 --2- 172.21.15.103:0/2129985022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18b8103470 0x7f18b8109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 --2- 172.21.15.103:0/2129985022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b8102ab0 0x7f18b8102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 --2- 172.21.15.103:0/2129985022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b81018c0 0x7f18b8101ce0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 -- 172.21.15.103:0/2129985022 >> 172.21.15.103:0/2129985022 conn(0x7f18b80fce50 msgr2=0x7f18b80ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:16.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 -- 172.21.15.103:0/2129985022 shutdown_connections 2024-05-23T00:00:16.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.330+0000 7f18c0400640 1 -- 172.21.15.103:0/2129985022 wait complete. 2024-05-23T00:00:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.331+0000 7f18c0400640 1 Processor -- start 2024-05-23T00:00:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.331+0000 7f18c0400640 1 -- start start 2024-05-23T00:00:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.331+0000 7f18c0400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18b81018c0 0x7f18b8199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.331+0000 7f18c0400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102ab0 0x7f18b819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18c0400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b8103470 0x7f18b81a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18c0400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810d530 con 0x7f18b81018c0 2024-05-23T00:00:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18c0400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810d3b0 con 0x7f18b8102ab0 2024-05-23T00:00:16.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18c0400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810d6b0 con 0x7f18b8103470 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bda00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18b81018c0 0x7f18b8199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bda00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18b81018c0 0x7f18b8199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57070/0 (socket says 172.21.15.103:57070) 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bda00640 1 -- 172.21.15.103:0/1792513217 learned_addr learned my addr 172.21.15.103:0/1792513217 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bd000640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102ab0 0x7f18b819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18be400640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b8103470 0x7f18b81a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bda00640 1 -- 172.21.15.103:0/1792513217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b8103470 msgr2=0x7f18b81a1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bda00640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b8103470 0x7f18b81a1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bda00640 1 -- 172.21.15.103:0/1792513217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102ab0 msgr2=0x7f18b819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bda00640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102ab0 0x7f18b819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bda00640 1 -- 172.21.15.103:0/1792513217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b81a1a90 con 0x7f18b81018c0 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.332+0000 7f18bd000640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102ab0 0x7f18b819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:16.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.333+0000 7f18bda00640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18b81018c0 0x7f18b8199e70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18b002acb0 tx=0x7f18b00305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:16.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.333+0000 7f18b6000640 1 -- 172.21.15.103:0/1792513217 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0041020 con 0x7f18b81018c0 2024-05-23T00:00:16.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.333+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b81a1d20 con 0x7f18b81018c0 2024-05-23T00:00:16.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.333+0000 7f18b6000640 1 -- 172.21.15.103:0/1792513217 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18b0031920 con 0x7f18b81018c0 2024-05-23T00:00:16.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.333+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b81a21b0 con 0x7f18b81018c0 2024-05-23T00:00:16.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.333+0000 7f18b6000640 1 -- 172.21.15.103:0/1792513217 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0036040 con 0x7f18b81018c0 2024-05-23T00:00:16.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.334+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b81a2410 con 0x7f18b81018c0 2024-05-23T00:00:16.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.336+0000 7f18b6000640 1 -- 172.21.15.103:0/1792513217 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f18b0044020 con 0x7f18b81018c0 2024-05-23T00:00:16.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.338+0000 7f18b6000640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18a00a2b90 0x7f18a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:16.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.338+0000 7f18b6000640 1 -- 172.21.15.103:0/1792513217 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f18b00efdd0 con 0x7f18b81018c0 2024-05-23T00:00:16.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.338+0000 7f18bd000640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18a00a2b90 0x7f18a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:16.342 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.339+0000 7f18bd000640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18a00a2b90 0x7f18a00a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f18b819b380 tx=0x7f18a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:16.345 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.342+0000 7f18b6000640 1 -- 172.21.15.103:0/1792513217 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f18b004b030 con 0x7f18b81018c0 2024-05-23T00:00:16.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.543+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f18b81a27a0 con 0x7f18b81018c0 2024-05-23T00:00:16.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.547+0000 7f18b6000640 1 -- 172.21.15.103:0/1792513217 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24658 (secure 0 0 0) 0x7f18b00bc970 con 0x7f18b81018c0 2024-05-23T00:00:16.551 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:16.551 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:58:31.088381+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:00:16.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18a00a2b90 msgr2=0x7f18a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18a00a2b90 0x7f18a00a5050 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f18b819b380 tx=0x7f18a8005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:16.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18b81018c0 msgr2=0x7f18b8199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18b81018c0 0x7f18b8199e70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18b002acb0 tx=0x7f18b00305d0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 shutdown_connections 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f18a00a2b90 0x7f18a00a5050 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18b8103470 0x7f18b81a1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18b8102ab0 0x7f18b819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 --2- 172.21.15.103:0/1792513217 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18b81018c0 0x7f18b8199e70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 >> 172.21.15.103:0/1792513217 conn(0x7f18b80fce50 msgr2=0x7f18b8073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 shutdown_connections 2024-05-23T00:00:16.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.550+0000 7f18c0400640 1 -- 172.21.15.103:0/1792513217 wait complete. 2024-05-23T00:00:16.574 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-23T00:00:16.574 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-23T00:00:16.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.777+0000 7faf07600640 1 -- 172.21.15.103:0/1650019602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf00103ad0 msgr2=0x7faf00103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.777+0000 7faf07600640 1 --2- 172.21.15.103:0/1650019602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf00103ad0 0x7faf00103ef0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faef4009900 tx=0x7faef402a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.777+0000 7faf07600640 1 -- 172.21.15.103:0/1650019602 shutdown_connections 2024-05-23T00:00:16.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.777+0000 7faf07600640 1 --2- 172.21.15.103:0/1650019602 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf00105680 0x7faf0010bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.777+0000 7faf07600640 1 --2- 172.21.15.103:0/1650019602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf00104cc0 0x7faf00105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.777+0000 7faf07600640 1 --2- 172.21.15.103:0/1650019602 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf00103ad0 0x7faf00103ef0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.777+0000 7faf07600640 1 -- 172.21.15.103:0/1650019602 >> 172.21.15.103:0/1650019602 conn(0x7faf000ff060 msgr2=0x7faf00101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:16.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.778+0000 7faf07600640 1 -- 172.21.15.103:0/1650019602 shutdown_connections 2024-05-23T00:00:16.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.778+0000 7faf07600640 1 -- 172.21.15.103:0/1650019602 wait complete. 2024-05-23T00:00:16.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.778+0000 7faf07600640 1 Processor -- start 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf07600640 1 -- start start 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf07600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf00103ad0 0x7faf00199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf07600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf00104cc0 0x7faf0019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf07600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf00105680 0x7faf001a13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf07600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faf0010f720 con 0x7faf00103ad0 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf07600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf0010f5a0 con 0x7faf00105680 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf07600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf0010f8a0 con 0x7faf00104cc0 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf04c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf00103ad0 0x7faf00199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf04c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf00103ad0 0x7faf00199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57088/0 (socket says 172.21.15.103:57088) 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf04c00640 1 -- 172.21.15.103:0/1472469310 learned_addr learned my addr 172.21.15.103:0/1472469310 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faf05600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf00105680 0x7faf001a13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:16.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.779+0000 7faeffe00640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf00104cc0 0x7faf0019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faf05600640 1 -- 172.21.15.103:0/1472469310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf00104cc0 msgr2=0x7faf0019a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faf05600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf00104cc0 0x7faf0019a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faf05600640 1 -- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf00103ad0 msgr2=0x7faf00199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faf05600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf00103ad0 0x7faf00199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faf05600640 1 -- 172.21.15.103:0/1472469310 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf001a1a60 con 0x7faf00105680 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faf05600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf00105680 0x7faf001a13f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faef800ec50 tx=0x7faef800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faefd600640 1 -- 172.21.15.103:0/1472469310 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8003b70 con 0x7faf00105680 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf001a1d50 con 0x7faf00105680 2024-05-23T00:00:16.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faefd600640 1 -- 172.21.15.103:0/1472469310 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef8003d10 con 0x7faf00105680 2024-05-23T00:00:16.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faefd600640 1 -- 172.21.15.103:0/1472469310 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8010890 con 0x7faf00105680 2024-05-23T00:00:16.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.780+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf001a21f0 con 0x7faf00105680 2024-05-23T00:00:16.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.782+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf0010af10 con 0x7faf00105680 2024-05-23T00:00:16.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.783+0000 7faefd600640 1 -- 172.21.15.103:0/1472469310 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7faef8003fe0 con 0x7faf00105680 2024-05-23T00:00:16.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.785+0000 7faefd600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faee80a2b90 0x7faee80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:16.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.785+0000 7faefd600640 1 -- 172.21.15.103:0/1472469310 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7faef80cf9b0 con 0x7faf00105680 2024-05-23T00:00:16.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.785+0000 7faf04c00640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faee80a2b90 0x7faee80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:16.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.786+0000 7faf04c00640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faee80a2b90 0x7faee80a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faef4000c00 tx=0x7faef40046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:16.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.789+0000 7faefd600640 1 -- 172.21.15.103:0/1472469310 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7faef809c550 con 0x7faf00105680 2024-05-23T00:00:16.994 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.990+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faf00062510 con 0x7faf00105680 2024-05-23T00:00:16.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.992+0000 7faefd600640 1 -- 172.21.15.103:0/1472469310 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7faef809be50 con 0x7faf00105680 2024-05-23T00:00:16.995 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-23T00:00:16.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faee80a2b90 msgr2=0x7faee80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faee80a2b90 0x7faee80a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faef4000c00 tx=0x7faef40046c0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf00105680 msgr2=0x7faf001a13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf00105680 0x7faf001a13f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7faef800ec50 tx=0x7faef800c6a0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 shutdown_connections 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7faee80a2b90 0x7faee80a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf00105680 0x7faf001a13f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf00104cc0 0x7faf0019a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 --2- 172.21.15.103:0/1472469310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf00103ad0 0x7faf00199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 >> 172.21.15.103:0/1472469310 conn(0x7faf000ff060 msgr2=0x7faf001080e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 shutdown_connections 2024-05-23T00:00:16.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:16.995+0000 7faf07600640 1 -- 172.21.15.103:0/1472469310 wait complete. 2024-05-23T00:00:17.019 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-23T00:00:17.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 -- 172.21.15.103:0/3644245831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 msgr2=0x7f1a481045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3644245831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 0x7f1a481045a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1a38009be0 tx=0x7f1a3802a6c0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.227 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 -- 172.21.15.103:0/3644245831 shutdown_connections 2024-05-23T00:00:17.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3644245831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a48107410 0x7f1a48109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3644245831 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a48104ae0 0x7f1a48106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3644245831 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 0x7f1a481045a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 -- 172.21.15.103:0/3644245831 >> 172.21.15.103:0/3644245831 conn(0x7f1a480fbb40 msgr2=0x7f1a480fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:17.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 -- 172.21.15.103:0/3644245831 shutdown_connections 2024-05-23T00:00:17.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.223+0000 7f1a4d000640 1 -- 172.21.15.103:0/3644245831 wait complete. 2024-05-23T00:00:17.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.224+0000 7f1a4d000640 1 Processor -- start 2024-05-23T00:00:17.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.224+0000 7f1a4d000640 1 -- start start 2024-05-23T00:00:17.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a4d000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 0x7f1a48101b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:17.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a4d000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a48104ae0 0x7f1a48100160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a4d000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a48107410 0x7f1a481006a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a4d000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4810d2e0 con 0x7f1a48107410 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a4d000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4810d160 con 0x7f1a481021b0 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a4d000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a4810d460 con 0x7f1a48104ae0 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a48107410 0x7f1a481006a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 0x7f1a48101b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a48107410 0x7f1a481006a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57110/0 (socket says 172.21.15.103:57110) 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47e00640 1 -- 172.21.15.103:0/3676893797 learned_addr learned my addr 172.21.15.103:0/3676893797 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a3ea00640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a48104ae0 0x7f1a48100160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:17.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47400640 1 -- 172.21.15.103:0/3676893797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a48104ae0 msgr2=0x7f1a48100160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47400640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a48104ae0 0x7f1a48100160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47400640 1 -- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a48107410 msgr2=0x7f1a481006a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47400640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a48107410 0x7f1a481006a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47400640 1 -- 172.21.15.103:0/3676893797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a48100f30 con 0x7f1a481021b0 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.225+0000 7f1a47e00640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a48107410 0x7f1a481006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.226+0000 7f1a47400640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 0x7f1a48101b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1a3802abe0 tx=0x7f1a380305b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.226+0000 7f1a44c00640 1 -- 172.21.15.103:0/3676893797 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38041020 con 0x7f1a481021b0 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.226+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a481a1a80 con 0x7f1a481021b0 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.226+0000 7f1a44c00640 1 -- 172.21.15.103:0/3676893797 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a38037040 con 0x7f1a481021b0 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.226+0000 7f1a44c00640 1 -- 172.21.15.103:0/3676893797 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a38036070 con 0x7f1a481021b0 2024-05-23T00:00:17.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.226+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a481a2020 con 0x7f1a481021b0 2024-05-23T00:00:17.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.227+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a480fd630 con 0x7f1a481021b0 2024-05-23T00:00:17.233 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.229+0000 7f1a44c00640 1 -- 172.21.15.103:0/3676893797 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1a38044020 con 0x7f1a481021b0 2024-05-23T00:00:17.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.231+0000 7f1a44c00640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a2c0a2b90 0x7f1a2c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:17.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.231+0000 7f1a44c00640 1 -- 172.21.15.103:0/3676893797 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1a380f0200 con 0x7f1a481021b0 2024-05-23T00:00:17.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.231+0000 7f1a3ea00640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a2c0a2b90 0x7f1a2c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:17.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.232+0000 7f1a3ea00640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a2c0a2b90 0x7f1a2c0a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1a48101890 tx=0x7f1a30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:17.236 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.233+0000 7f1a44c00640 1 -- 172.21.15.103:0/3676893797 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1a3804b020 con 0x7f1a481021b0 2024-05-23T00:00:17.438 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.435+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1a4802d390 con 0x7f1a481021b0 2024-05-23T00:00:17.439 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.436+0000 7f1a44c00640 1 -- 172.21.15.103:0/3676893797 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f1a380bcda0 con 0x7f1a481021b0 2024-05-23T00:00:17.439 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:17.442 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a2c0a2b90 msgr2=0x7f1a2c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a2c0a2b90 0x7f1a2c0a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1a48101890 tx=0x7f1a30009260 comp rx=0 tx=0).stop 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 msgr2=0x7f1a48101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 0x7f1a48101b20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1a3802abe0 tx=0x7f1a380305b0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 shutdown_connections 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f1a2c0a2b90 0x7f1a2c0a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a48107410 0x7f1a481006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a48104ae0 0x7f1a48100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 --2- 172.21.15.103:0/3676893797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a481021b0 0x7f1a48101b20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.443 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 >> 172.21.15.103:0/3676893797 conn(0x7f1a480fbb40 msgr2=0x7f1a48102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:17.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 shutdown_connections 2024-05-23T00:00:17.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.439+0000 7f1a4d000640 1 -- 172.21.15.103:0/3676893797 wait complete. 2024-05-23T00:00:17.461 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-23T00:00:17.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:17 smithi105 ceph-mon[36426]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-23T00:00:17.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:17 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1792513217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:17.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:17 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1472469310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:17.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:17 smithi103 ceph-mon[31370]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-23T00:00:17.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:17 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1792513217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:17.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:17 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1472469310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:17.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:17 smithi174 ceph-mon[36179]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-23T00:00:17.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:17 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1792513217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:17.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:17 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1472469310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:17.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 -- 172.21.15.103:0/3357842838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37100742d0 msgr2=0x7f3710074750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 --2- 172.21.15.103:0/3357842838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37100742d0 0x7f3710074750 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f36fc009900 tx=0x7f36fc02a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 -- 172.21.15.103:0/3357842838 shutdown_connections 2024-05-23T00:00:17.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 --2- 172.21.15.103:0/3357842838 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f371010dd50 0x7f3710110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 --2- 172.21.15.103:0/3357842838 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37100742d0 0x7f3710074750 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 --2- 172.21.15.103:0/3357842838 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37100758b0 0x7f3710073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 -- 172.21.15.103:0/3357842838 >> 172.21.15.103:0/3357842838 conn(0x7f37100fcd10 msgr2=0x7f37100ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:17.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 -- 172.21.15.103:0/3357842838 shutdown_connections 2024-05-23T00:00:17.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.662+0000 7f3716c00640 1 -- 172.21.15.103:0/3357842838 wait complete. 2024-05-23T00:00:17.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.663+0000 7f3716c00640 1 Processor -- start 2024-05-23T00:00:17.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.663+0000 7f3716c00640 1 -- start start 2024-05-23T00:00:17.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.663+0000 7f3716c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37100742d0 0x7f371019e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:17.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.663+0000 7f3716c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37100758b0 0x7f371019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:17.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.663+0000 7f3716c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f371010dd50 0x7f37101a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:17.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.663+0000 7f3716c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3710113a40 con 0x7f37100758b0 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.663+0000 7f3716c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f37101138c0 con 0x7f37100742d0 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.663+0000 7f3716c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3710113bc0 con 0x7f371010dd50 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37100758b0 0x7f371019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37100758b0 0x7f371019e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57124/0 (socket says 172.21.15.103:57124) 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 -- 172.21.15.103:0/1999783038 learned_addr learned my addr 172.21.15.103:0/1999783038 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3715800640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37100742d0 0x7f371019e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3716200640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f371010dd50 0x7f37101a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 -- 172.21.15.103:0/1999783038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f371010dd50 msgr2=0x7f37101a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f371010dd50 0x7f37101a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 -- 172.21.15.103:0/1999783038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37100742d0 msgr2=0x7f371019e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37100742d0 0x7f371019e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.669 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 -- 172.21.15.103:0/1999783038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37101a5cd0 con 0x7f37100758b0 2024-05-23T00:00:17.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.664+0000 7f3714e00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37100758b0 0x7f371019e5f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0315a0 tx=0x7f36fc032c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:17.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.665+0000 7f370e000640 1 -- 172.21.15.103:0/1999783038 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc03f030 con 0x7f37100758b0 2024-05-23T00:00:17.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.665+0000 7f370e000640 1 -- 172.21.15.103:0/1999783038 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36fc003ce0 con 0x7f37100758b0 2024-05-23T00:00:17.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.665+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37101a5f60 con 0x7f37100758b0 2024-05-23T00:00:17.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.665+0000 7f370e000640 1 -- 172.21.15.103:0/1999783038 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36fc035070 con 0x7f37100758b0 2024-05-23T00:00:17.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.665+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37101a63c0 con 0x7f37100758b0 2024-05-23T00:00:17.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.666+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3710103e10 con 0x7f37100758b0 2024-05-23T00:00:17.671 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.668+0000 7f370e000640 1 -- 172.21.15.103:0/1999783038 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f36fc042030 con 0x7f37100758b0 2024-05-23T00:00:17.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.670+0000 7f370e000640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f36f80a2b90 0x7f36f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:17.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.670+0000 7f370e000640 1 -- 172.21.15.103:0/1999783038 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f36fc0ef740 con 0x7f37100758b0 2024-05-23T00:00:17.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.670+0000 7f3715800640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f36f80a2b90 0x7f36f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:17.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.671+0000 7f3715800640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f36f80a2b90 0x7f36f80a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3704009bd0 tx=0x7f3704005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:17.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.675+0000 7f370e000640 1 -- 172.21.15.103:0/1999783038 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f36fc0bc2e0 con 0x7f37100758b0 2024-05-23T00:00:17.885 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.881+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3710062470 con 0x7f37100758b0 2024-05-23T00:00:17.885 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.882+0000 7f370e000640 1 -- 172.21.15.103:0/1999783038 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f36fc0bbbe0 con 0x7f37100758b0 2024-05-23T00:00:17.886 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-23T00:00:17.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f36f80a2b90 msgr2=0x7f36f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f36f80a2b90 0x7f36f80a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3704009bd0 tx=0x7f3704005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:17.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37100758b0 msgr2=0x7f371019e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:17.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37100758b0 0x7f371019e5f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f36fc0315a0 tx=0x7f36fc032c70 comp rx=0 tx=0).stop 2024-05-23T00:00:17.888 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 shutdown_connections 2024-05-23T00:00:17.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f36f80a2b90 0x7f36f80a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f371010dd50 0x7f37101a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f37100758b0 0x7f371019e5f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 --2- 172.21.15.103:0/1999783038 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f37100742d0 0x7f371019e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:17.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 >> 172.21.15.103:0/1999783038 conn(0x7f37100fcd10 msgr2=0x7f3710100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:17.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 shutdown_connections 2024-05-23T00:00:17.889 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:17.885+0000 7f3716c00640 1 -- 172.21.15.103:0/1999783038 wait complete. 2024-05-23T00:00:17.907 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-23T00:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 -- 172.21.15.103:0/136541555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341016b0 msgr2=0x7f4d34101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 --2- 172.21.15.103:0/136541555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341016b0 0x7f4d34101ad0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009ca0 tx=0x7f4d2c02a8c0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 -- 172.21.15.103:0/136541555 shutdown_connections 2024-05-23T00:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 --2- 172.21.15.103:0/136541555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d34103260 0x7f4d34109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 --2- 172.21.15.103:0/136541555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d341028a0 0x7f4d34102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.113 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 --2- 172.21.15.103:0/136541555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341016b0 0x7f4d34101ad0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 -- 172.21.15.103:0/136541555 >> 172.21.15.103:0/136541555 conn(0x7f4d340fcc20 msgr2=0x7f4d340ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:18.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 -- 172.21.15.103:0/136541555 shutdown_connections 2024-05-23T00:00:18.114 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.109+0000 7f4d39400640 1 -- 172.21.15.103:0/136541555 wait complete. 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.110+0000 7f4d39400640 1 Processor -- start 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d39400640 1 -- start start 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d39400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d341016b0 0x7f4d34075620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d39400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341028a0 0x7f4d34073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d39400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d34103260 0x7f4d340741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d39400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3410d380 con 0x7f4d341028a0 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d39400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3410d200 con 0x7f4d341016b0 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d39400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3410d500 con 0x7f4d34103260 2024-05-23T00:00:18.116 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d33400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341028a0 0x7f4d34073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d33400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341028a0 0x7f4d34073c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57148/0 (socket says 172.21.15.103:57148) 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d33400640 1 -- 172.21.15.103:0/2319950938 learned_addr learned my addr 172.21.15.103:0/2319950938 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d33e00640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d341016b0 0x7f4d34075620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.111+0000 7f4d38a00640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d34103260 0x7f4d340741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d33400640 1 -- 172.21.15.103:0/2319950938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d34103260 msgr2=0x7f4d340741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d33400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d34103260 0x7f4d340741a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d33400640 1 -- 172.21.15.103:0/2319950938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d341016b0 msgr2=0x7f4d34075620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d33400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d341016b0 0x7f4d34075620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d33400640 1 -- 172.21.15.103:0/2319950938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d34074a30 con 0x7f4d341028a0 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d33400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341028a0 0x7f4d34073c60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00ea60 tx=0x7f4d1c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d30c00640 1 -- 172.21.15.103:0/2319950938 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00cde0 con 0x7f4d341028a0 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d341a1a90 con 0x7f4d341028a0 2024-05-23T00:00:18.117 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d30c00640 1 -- 172.21.15.103:0/2319950938 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d1c0035f0 con 0x7f4d341028a0 2024-05-23T00:00:18.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d30c00640 1 -- 172.21.15.103:0/2319950938 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c010670 con 0x7f4d341028a0 2024-05-23T00:00:18.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.112+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d341a1fe0 con 0x7f4d341028a0 2024-05-23T00:00:18.118 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.113+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d341089d0 con 0x7f4d341028a0 2024-05-23T00:00:18.119 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.116+0000 7f4d30c00640 1 -- 172.21.15.103:0/2319950938 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f4d1c0037a0 con 0x7f4d341028a0 2024-05-23T00:00:18.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.117+0000 7f4d30c00640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4d180a2b90 0x7f4d180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:18.120 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.117+0000 7f4d30c00640 1 -- 172.21.15.103:0/2319950938 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f4d1c0cf7e0 con 0x7f4d341028a0 2024-05-23T00:00:18.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.117+0000 7f4d33e00640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4d180a2b90 0x7f4d180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:18.121 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.118+0000 7f4d33e00640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4d180a2b90 0x7f4d180a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c000f80 tx=0x7f4d2c005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:18.122 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.119+0000 7f4d30c00640 1 -- 172.21.15.103:0/2319950938 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4d1c09c410 con 0x7f4d341028a0 2024-05-23T00:00:18.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.323+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4d3402d390 con 0x7f4d341028a0 2024-05-23T00:00:18.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.324+0000 7f4d30c00640 1 -- 172.21.15.103:0/2319950938 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f4d1c09bd10 con 0x7f4d341028a0 2024-05-23T00:00:18.327 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:18.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.326+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4d180a2b90 msgr2=0x7f4d180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.326+0000 7f4d39400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4d180a2b90 0x7f4d180a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c000f80 tx=0x7f4d2c005f30 comp rx=0 tx=0).stop 2024-05-23T00:00:18.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.326+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341028a0 msgr2=0x7f4d34073c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.326+0000 7f4d39400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341028a0 0x7f4d34073c60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c00ea60 tx=0x7f4d1c00ef20 comp rx=0 tx=0).stop 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.327+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 shutdown_connections 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.327+0000 7f4d39400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f4d180a2b90 0x7f4d180a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.327+0000 7f4d39400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4d34103260 0x7f4d340741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.327+0000 7f4d39400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4d341028a0 0x7f4d34073c60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.327+0000 7f4d39400640 1 --2- 172.21.15.103:0/2319950938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d341016b0 0x7f4d34075620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.327+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 >> 172.21.15.103:0/2319950938 conn(0x7f4d340fcc20 msgr2=0x7f4d34105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.327+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 shutdown_connections 2024-05-23T00:00:18.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.327+0000 7f4d39400640 1 -- 172.21.15.103:0/2319950938 wait complete. 2024-05-23T00:00:18.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:00:18.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.550+0000 7fb810400640 1 -- 172.21.15.103:0/471168540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb808108870 msgr2=0x7fb808108c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.550+0000 7fb810400640 1 --2- 172.21.15.103:0/471168540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb808108870 0x7fb808108c90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009930 tx=0x7fb7fc02a440 comp rx=0 tx=0).stop 2024-05-23T00:00:18.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.550+0000 7fb810400640 1 -- 172.21.15.103:0/471168540 shutdown_connections 2024-05-23T00:00:18.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.550+0000 7fb810400640 1 --2- 172.21.15.103:0/471168540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8081100f0 0x7fb8081124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.550+0000 7fb810400640 1 --2- 172.21.15.103:0/471168540 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8081091d0 0x7fb80810fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.550+0000 7fb810400640 1 --2- 172.21.15.103:0/471168540 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb808108870 0x7fb808108c90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.550+0000 7fb810400640 1 -- 172.21.15.103:0/471168540 >> 172.21.15.103:0/471168540 conn(0x7fb80806eab0 msgr2=0x7fb808070ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:18.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.551+0000 7fb810400640 1 -- 172.21.15.103:0/471168540 shutdown_connections 2024-05-23T00:00:18.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.551+0000 7fb810400640 1 -- 172.21.15.103:0/471168540 wait complete. 2024-05-23T00:00:18.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb810400640 1 Processor -- start 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb810400640 1 -- start start 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb810400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb808108870 0x7fb8081a2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb810400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8081091d0 0x7fb8081a2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb810400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8081100f0 0x7fb8081a9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb810400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb808115db0 con 0x7fb8081100f0 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb810400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb808115c30 con 0x7fb8081091d0 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb810400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb808115f30 con 0x7fb808108870 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb80da00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb808108870 0x7fb8081a2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:18.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb80da00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb808108870 0x7fb8081a2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:35922/0 (socket says 172.21.15.103:35922) 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb80e400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8081100f0 0x7fb8081a9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb80da00640 1 -- 172.21.15.103:0/1973129849 learned_addr learned my addr 172.21.15.103:0/1973129849 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.552+0000 7fb80d000640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8081091d0 0x7fb8081a2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.553+0000 7fb80da00640 1 -- 172.21.15.103:0/1973129849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8081091d0 msgr2=0x7fb8081a2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.553+0000 7fb80da00640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8081091d0 0x7fb8081a2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.553+0000 7fb80da00640 1 -- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8081100f0 msgr2=0x7fb8081a9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.553+0000 7fb80da00640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8081100f0 0x7fb8081a9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.553+0000 7fb80da00640 1 -- 172.21.15.103:0/1973129849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8081aa260 con 0x7fb808108870 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.553+0000 7fb80e400640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8081100f0 0x7fb8081a9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.553+0000 7fb80da00640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb808108870 0x7fb8081a2690 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc031440 tx=0x7fb7fc032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.553+0000 7fb806000640 1 -- 172.21.15.103:0/1973129849 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc03f030 con 0x7fb808108870 2024-05-23T00:00:18.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.554+0000 7fb806000640 1 -- 172.21.15.103:0/1973129849 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7fc003ce0 con 0x7fb808108870 2024-05-23T00:00:18.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.554+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8081aa4f0 con 0x7fb808108870 2024-05-23T00:00:18.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.554+0000 7fb806000640 1 -- 172.21.15.103:0/1973129849 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc035070 con 0x7fb808108870 2024-05-23T00:00:18.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.554+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8081aaa30 con 0x7fb808108870 2024-05-23T00:00:18.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:18 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3676893797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:18.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:18 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1999783038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:18.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.555+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8081aac90 con 0x7fb808108870 2024-05-23T00:00:18.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.557+0000 7fb806000640 1 -- 172.21.15.103:0/1973129849 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb7fc042030 con 0x7fb808108870 2024-05-23T00:00:18.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.558+0000 7fb806000640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:18.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.558+0000 7fb806000640 1 -- 172.21.15.103:0/1973129849 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb7fc0ef790 con 0x7fb808108870 2024-05-23T00:00:18.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.558+0000 7fb80d000640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:18.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.559+0000 7fb80d000640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb7f40095f0 tx=0x7fb7f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:18.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.561+0000 7fb806000640 1 -- 172.21.15.103:0/1973129849 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb7fc0bbce0 con 0x7fb808108870 2024-05-23T00:00:18.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:18 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3676893797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:18.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:18 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1999783038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:18.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:18 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3676893797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:18.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:18 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1999783038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:18.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.803+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8081aae60 con 0x7fb808108870 2024-05-23T00:00:18.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.804+0000 7fb806000640 1 -- 172.21.15.103:0/1973129849 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11896 (secure 0 0 0) 0x7fb7fc0bffe0 con 0x7fb808108870 2024-05-23T00:00:18.808 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:18.808 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-22T23:58:05.260333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24559,24563,24575]},"id":1}]} 2024-05-23T00:00:18.808 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 24 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb7f00a2c10 msgr2=0x7fb7f00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb7f40095f0 tx=0x7fb7f4005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb808108870 msgr2=0x7fb8081a2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb808108870 0x7fb8081a2690 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc031440 tx=0x7fb7fc032ce0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 shutdown_connections 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb7f00a2c10 0x7fb7f00a50d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb8081100f0 0x7fb8081a9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8081091d0 0x7fb8081a2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 --2- 172.21.15.103:0/1973129849 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb808108870 0x7fb8081a2690 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.807+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 >> 172.21.15.103:0/1973129849 conn(0x7fb80806eab0 msgr2=0x7fb808070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.808+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 shutdown_connections 2024-05-23T00:00:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:18.808+0000 7fb810400640 1 -- 172.21.15.103:0/1973129849 wait complete. 2024-05-23T00:00:18.830 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:00:19.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.028+0000 7fb45a000640 1 -- 172.21.15.103:0/1051207156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454103af0 msgr2=0x7fb454103f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.028+0000 7fb45a000640 1 --2- 172.21.15.103:0/1051207156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454103af0 0x7fb454103f10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb444009be0 tx=0x7fb44402a6c0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.029+0000 7fb45a000640 1 -- 172.21.15.103:0/1051207156 shutdown_connections 2024-05-23T00:00:19.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.029+0000 7fb45a000640 1 --2- 172.21.15.103:0/1051207156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb4541056a0 0x7fb45410bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.029+0000 7fb45a000640 1 --2- 172.21.15.103:0/1051207156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb454104ce0 0x7fb454105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.029+0000 7fb45a000640 1 --2- 172.21.15.103:0/1051207156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454103af0 0x7fb454103f10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.029+0000 7fb45a000640 1 -- 172.21.15.103:0/1051207156 >> 172.21.15.103:0/1051207156 conn(0x7fb4540ff060 msgr2=0x7fb4541014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:19.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.029+0000 7fb45a000640 1 -- 172.21.15.103:0/1051207156 shutdown_connections 2024-05-23T00:00:19.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.029+0000 7fb45a000640 1 -- 172.21.15.103:0/1051207156 wait complete. 2024-05-23T00:00:19.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb45a000640 1 Processor -- start 2024-05-23T00:00:19.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb45a000640 1 -- start start 2024-05-23T00:00:19.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb45a000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb454103af0 0x7fb454199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb45a000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454104ce0 0x7fb45419a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb45a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4541056a0 0x7fb4541a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb45a000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb45410f720 con 0x7fb4541056a0 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb45a000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fb45410f5a0 con 0x7fb454103af0 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb45a000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb45410f8a0 con 0x7fb454104ce0 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb453400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb454103af0 0x7fb454199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb44aa00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454104ce0 0x7fb45419a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb453e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4541056a0 0x7fb4541a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.030+0000 7fb453400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb454103af0 0x7fb454199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:59356/0 (socket says 172.21.15.103:59356) 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.031+0000 7fb453400640 1 -- 172.21.15.103:0/3251005287 learned_addr learned my addr 172.21.15.103:0/3251005287 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.031+0000 7fb453400640 1 -- 172.21.15.103:0/3251005287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454104ce0 msgr2=0x7fb45419a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.031+0000 7fb453400640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454104ce0 0x7fb45419a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.031+0000 7fb453400640 1 -- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4541056a0 msgr2=0x7fb4541a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.031+0000 7fb453400640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4541056a0 0x7fb4541a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.031+0000 7fb453400640 1 -- 172.21.15.103:0/3251005287 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4541a1a10 con 0x7fb454103af0 2024-05-23T00:00:19.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.031+0000 7fb453e00640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4541056a0 0x7fb4541a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:19.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.031+0000 7fb453400640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb454103af0 0x7fb454199df0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb444030960 tx=0x7fb4440305b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:19.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.032+0000 7fb450c00640 1 -- 172.21.15.103:0/3251005287 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444041020 con 0x7fb454103af0 2024-05-23T00:00:19.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.032+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4541a1ca0 con 0x7fb454103af0 2024-05-23T00:00:19.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.032+0000 7fb450c00640 1 -- 172.21.15.103:0/3251005287 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb444037040 con 0x7fb454103af0 2024-05-23T00:00:19.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.032+0000 7fb450c00640 1 -- 172.21.15.103:0/3251005287 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444036070 con 0x7fb454103af0 2024-05-23T00:00:19.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.032+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4541a2190 con 0x7fb454103af0 2024-05-23T00:00:19.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.033+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb45410aeb0 con 0x7fb454103af0 2024-05-23T00:00:19.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.035+0000 7fb450c00640 1 -- 172.21.15.103:0/3251005287 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fb444044020 con 0x7fb454103af0 2024-05-23T00:00:19.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.036+0000 7fb450c00640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb4380a2b90 0x7fb4380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.036+0000 7fb450c00640 1 -- 172.21.15.103:0/3251005287 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fb4440f0330 con 0x7fb454103af0 2024-05-23T00:00:19.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.037+0000 7fb44aa00640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb4380a2b90 0x7fb4380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.037+0000 7fb44aa00640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb4380a2b90 0x7fb4380a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb45419b300 tx=0x7fb43c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:19.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.039+0000 7fb450c00640 1 -- 172.21.15.103:0/3251005287 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fb44404b020 con 0x7fb454103af0 2024-05-23T00:00:19.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.245+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb45402d450 con 0x7fb454103af0 2024-05-23T00:00:19.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.247+0000 7fb450c00640 1 -- 172.21.15.103:0/3251005287 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24658 (secure 0 0 0) 0x7fb4440bced0 con 0x7fb454103af0 2024-05-23T00:00:19.251 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:19.251 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:58:31.088381+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:00:19.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb4380a2b90 msgr2=0x7fb4380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb4380a2b90 0x7fb4380a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb45419b300 tx=0x7fb43c005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:19.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb454103af0 msgr2=0x7fb454199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb454103af0 0x7fb454199df0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb444030960 tx=0x7fb4440305b0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 shutdown_connections 2024-05-23T00:00:19.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fb4380a2b90 0x7fb4380a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4541056a0 0x7fb4541a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb454104ce0 0x7fb45419a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 --2- 172.21.15.103:0/3251005287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb454103af0 0x7fb454199df0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.249+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 >> 172.21.15.103:0/3251005287 conn(0x7fb4540ff060 msgr2=0x7fb454108080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:19.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.250+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 shutdown_connections 2024-05-23T00:00:19.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.250+0000 7fb45a000640 1 -- 172.21.15.103:0/3251005287 wait complete. 2024-05-23T00:00:19.271 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-23T00:00:19.271 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-23T00:00:19.476 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 -- 172.21.15.103:0/3492355720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 msgr2=0x7fa9e010bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/3492355720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 0x7fa9e010bc00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009900 tx=0x7fa9d802a410 comp rx=0 tx=0).stop 2024-05-23T00:00:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 -- 172.21.15.103:0/3492355720 shutdown_connections 2024-05-23T00:00:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/3492355720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 0x7fa9e010bc00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/3492355720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e00742d0 0x7fa9e0074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/3492355720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e00758b0 0x7fa9e0073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 -- 172.21.15.103:0/3492355720 >> 172.21.15.103:0/3492355720 conn(0x7fa9e00fce20 msgr2=0x7fa9e00ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 -- 172.21.15.103:0/3492355720 shutdown_connections 2024-05-23T00:00:19.477 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.473+0000 7fa9e7c00640 1 -- 172.21.15.103:0/3492355720 wait complete. 2024-05-23T00:00:19.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.474+0000 7fa9e7c00640 1 Processor -- start 2024-05-23T00:00:19.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.474+0000 7fa9e7c00640 1 -- start start 2024-05-23T00:00:19.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.474+0000 7fa9e7c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e00742d0 0x7fa9e0199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.480 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e7c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e00758b0 0x7fa9e019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e7c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 0x7fa9e01a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e7c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e010f710 con 0x7fa9e0109770 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e7c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e010f590 con 0x7fa9e00758b0 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e7c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e010f890 con 0x7fa9e00742d0 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 0x7fa9e01a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 0x7fa9e01a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57192/0 (socket says 172.21.15.103:57192) 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5c00640 1 -- 172.21.15.103:0/4167368321 learned_addr learned my addr 172.21.15.103:0/4167368321 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5200640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e00742d0 0x7fa9e0199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9dfe00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e00758b0 0x7fa9e019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5c00640 1 -- 172.21.15.103:0/4167368321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e00742d0 msgr2=0x7fa9e0199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.481 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e00742d0 0x7fa9e0199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5c00640 1 -- 172.21.15.103:0/4167368321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e00758b0 msgr2=0x7fa9e019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e00758b0 0x7fa9e019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.475+0000 7fa9e5c00640 1 -- 172.21.15.103:0/4167368321 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e01a1990 con 0x7fa9e0109770 2024-05-23T00:00:19.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.476+0000 7fa9e5c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 0x7fa9e01a1320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa9d80314e0 tx=0x7fa9d80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:19.482 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.476+0000 7fa9dd600640 1 -- 172.21.15.103:0/4167368321 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d803f030 con 0x7fa9e0109770 2024-05-23T00:00:19.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.476+0000 7fa9dd600640 1 -- 172.21.15.103:0/4167368321 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d8003ce0 con 0x7fa9e0109770 2024-05-23T00:00:19.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.476+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e01a1c20 con 0x7fa9e0109770 2024-05-23T00:00:19.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.476+0000 7fa9dd600640 1 -- 172.21.15.103:0/4167368321 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8035070 con 0x7fa9e0109770 2024-05-23T00:00:19.483 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.476+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e01a2110 con 0x7fa9e0109770 2024-05-23T00:00:19.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.477+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e01084a0 con 0x7fa9e0109770 2024-05-23T00:00:19.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.479+0000 7fa9dd600640 1 -- 172.21.15.103:0/4167368321 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fa9d8042030 con 0x7fa9e0109770 2024-05-23T00:00:19.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.481+0000 7fa9dd600640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9c40a2b90 0x7fa9c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.481+0000 7fa9dd600640 1 -- 172.21.15.103:0/4167368321 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fa9d80ef840 con 0x7fa9e0109770 2024-05-23T00:00:19.484 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.481+0000 7fa9e5200640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9c40a2b90 0x7fa9c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.485 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.482+0000 7fa9e5200640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9c40a2b90 0x7fa9c40a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0005fd0 tx=0x7fa9d0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:19.488 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.485+0000 7fa9dd600640 1 -- 172.21.15.103:0/4167368321 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa9d80bbd90 con 0x7fa9e0109770 2024-05-23T00:00:19.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:19 smithi105 ceph-mon[36426]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:00:19.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:19 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2319950938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:19.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:19 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1973129849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:19.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:19 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3251005287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:19.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:19 smithi103 ceph-mon[31370]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:00:19.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:19 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2319950938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:19.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:19 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1973129849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:19.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:19 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3251005287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:19.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:19 smithi174 ceph-mon[36179]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:00:19.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:19 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2319950938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:19.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:19 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1973129849' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:19.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:19 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3251005287' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:19.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.685+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa9e0062510 con 0x7fa9e0109770 2024-05-23T00:00:19.689 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.686+0000 7fa9dd600640 1 -- 172.21.15.103:0/4167368321 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fa9d80c0090 con 0x7fa9e0109770 2024-05-23T00:00:19.690 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-23T00:00:19.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9c40a2b90 msgr2=0x7fa9c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9c40a2b90 0x7fa9c40a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0005fd0 tx=0x7fa9d0009500 comp rx=0 tx=0).stop 2024-05-23T00:00:19.692 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 msgr2=0x7fa9e01a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 0x7fa9e01a1320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa9d80314e0 tx=0x7fa9d80042f0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 shutdown_connections 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fa9c40a2b90 0x7fa9c40a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e0109770 0x7fa9e01a1320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e00758b0 0x7fa9e019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 --2- 172.21.15.103:0/4167368321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e00742d0 0x7fa9e0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 >> 172.21.15.103:0/4167368321 conn(0x7fa9e00fce20 msgr2=0x7fa9e0105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 shutdown_connections 2024-05-23T00:00:19.693 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.689+0000 7fa9e7c00640 1 -- 172.21.15.103:0/4167368321 wait complete. 2024-05-23T00:00:19.712 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-23T00:00:19.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.912+0000 7f33c2000640 1 -- 172.21.15.103:0/776361234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33bc103ab0 msgr2=0x7f33bc103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.912+0000 7f33c2000640 1 --2- 172.21.15.103:0/776361234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33bc103ab0 0x7f33bc103ed0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f33ac009900 tx=0x7f33ac02a410 comp rx=0 tx=0).stop 2024-05-23T00:00:19.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.913+0000 7f33c2000640 1 -- 172.21.15.103:0/776361234 shutdown_connections 2024-05-23T00:00:19.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.913+0000 7f33c2000640 1 --2- 172.21.15.103:0/776361234 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc105660 0x7f33bc10bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.920 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.913+0000 7f33c2000640 1 --2- 172.21.15.103:0/776361234 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33bc104ca0 0x7f33bc105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.913+0000 7f33c2000640 1 --2- 172.21.15.103:0/776361234 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33bc103ab0 0x7f33bc103ed0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.913+0000 7f33c2000640 1 -- 172.21.15.103:0/776361234 >> 172.21.15.103:0/776361234 conn(0x7f33bc0ff060 msgr2=0x7f33bc101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:19.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.913+0000 7f33c2000640 1 -- 172.21.15.103:0/776361234 shutdown_connections 2024-05-23T00:00:19.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.913+0000 7f33c2000640 1 -- 172.21.15.103:0/776361234 wait complete. 2024-05-23T00:00:19.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33c2000640 1 Processor -- start 2024-05-23T00:00:19.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33c2000640 1 -- start start 2024-05-23T00:00:19.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33c2000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33bc103ab0 0x7f33bc199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33c2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33bc104ca0 0x7f33bc19a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33c2000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc105660 0x7f33bc1a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33c2000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc10f700 con 0x7f33bc104ca0 2024-05-23T00:00:19.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33c2000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc10f580 con 0x7f33bc105660 2024-05-23T00:00:19.922 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33c2000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f33bc10f880 con 0x7f33bc103ab0 2024-05-23T00:00:19.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.914+0000 7f33baa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33bc104ca0 0x7f33bc19a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33baa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33bc104ca0 0x7f33bc19a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57202/0 (socket says 172.21.15.103:57202) 2024-05-23T00:00:19.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33baa00640 1 -- 172.21.15.103:0/1718524583 learned_addr learned my addr 172.21.15.103:0/1718524583 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:19.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33bbe00640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc105660 0x7f33bc1a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33bb400640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33bc103ab0 0x7f33bc199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33baa00640 1 -- 172.21.15.103:0/1718524583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33bc103ab0 msgr2=0x7f33bc199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33baa00640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33bc103ab0 0x7f33bc199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33baa00640 1 -- 172.21.15.103:0/1718524583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc105660 msgr2=0x7f33bc1a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:19.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33baa00640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc105660 0x7f33bc1a1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:19.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33baa00640 1 -- 172.21.15.103:0/1718524583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc1a19f0 con 0x7f33bc104ca0 2024-05-23T00:00:19.924 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33bbe00640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc105660 0x7f33bc1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:19.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.915+0000 7f33baa00640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33bc104ca0 0x7f33bc19a310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f33a400e9f0 tx=0x7f33a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:19.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.916+0000 7f33b3e00640 1 -- 172.21.15.103:0/1718524583 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a400cd80 con 0x7f33bc104ca0 2024-05-23T00:00:19.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.916+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33bc1a1ce0 con 0x7f33bc104ca0 2024-05-23T00:00:19.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.916+0000 7f33b3e00640 1 -- 172.21.15.103:0/1718524583 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33a4010910 con 0x7f33bc104ca0 2024-05-23T00:00:19.925 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.916+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33bc1a21d0 con 0x7f33bc104ca0 2024-05-23T00:00:19.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.916+0000 7f33b3e00640 1 -- 172.21.15.103:0/1718524583 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4018910 con 0x7f33bc104ca0 2024-05-23T00:00:19.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.917+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33bc10ae50 con 0x7f33bc104ca0 2024-05-23T00:00:19.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.920+0000 7f33b3e00640 1 -- 172.21.15.103:0/1718524583 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f33a4010ac0 con 0x7f33bc104ca0 2024-05-23T00:00:19.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.921+0000 7f33b3e00640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f33a00a2b90 0x7f33a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:19.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.921+0000 7f33b3e00640 1 -- 172.21.15.103:0/1718524583 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f33a40cf780 con 0x7f33bc104ca0 2024-05-23T00:00:19.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.921+0000 7f33bb400640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f33a00a2b90 0x7f33a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:19.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.922+0000 7f33bb400640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f33a00a2b90 0x7f33a00a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f33ac005ab0 tx=0x7f33ac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:19.929 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:19.925+0000 7f33b3e00640 1 -- 172.21.15.103:0/1718524583 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f33a409bcd0 con 0x7f33bc104ca0 2024-05-23T00:00:20.129 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.126+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f33bc062510 con 0x7f33bc104ca0 2024-05-23T00:00:20.130 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.127+0000 7f33b3e00640 1 -- 172.21.15.103:0/1718524583 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f33a409ffd0 con 0x7f33bc104ca0 2024-05-23T00:00:20.130 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:20.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.129+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f33a00a2b90 msgr2=0x7f33a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.129+0000 7f33c2000640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f33a00a2b90 0x7f33a00a5050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f33ac005ab0 tx=0x7f33ac000bc0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.133 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.129+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33bc104ca0 msgr2=0x7f33bc19a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.129+0000 7f33c2000640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33bc104ca0 0x7f33bc19a310 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f33a400e9f0 tx=0x7f33a400eeb0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.130+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 shutdown_connections 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.130+0000 7f33c2000640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f33a00a2b90 0x7f33a00a5050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.130+0000 7f33c2000640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33bc105660 0x7f33bc1a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.130+0000 7f33c2000640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33bc104ca0 0x7f33bc19a310 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.130+0000 7f33c2000640 1 --2- 172.21.15.103:0/1718524583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f33bc103ab0 0x7f33bc199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.130+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 >> 172.21.15.103:0/1718524583 conn(0x7f33bc0ff060 msgr2=0x7f33bc108020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.130+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 shutdown_connections 2024-05-23T00:00:20.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.130+0000 7f33c2000640 1 -- 172.21.15.103:0/1718524583 wait complete. 2024-05-23T00:00:20.152 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-23T00:00:20.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.360+0000 7f6ebc200640 1 -- 172.21.15.103:0/967794858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb41018c0 msgr2=0x7f6eb4101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.360+0000 7f6ebc200640 1 --2- 172.21.15.103:0/967794858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb41018c0 0x7f6eb4101ce0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009900 tx=0x7f6eac02a410 comp rx=0 tx=0).stop 2024-05-23T00:00:20.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.360+0000 7f6ebc200640 1 -- 172.21.15.103:0/967794858 shutdown_connections 2024-05-23T00:00:20.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.360+0000 7f6ebc200640 1 --2- 172.21.15.103:0/967794858 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb4103470 0x7f6eb4109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.360+0000 7f6ebc200640 1 --2- 172.21.15.103:0/967794858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb4102ab0 0x7f6eb4102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.360+0000 7f6ebc200640 1 --2- 172.21.15.103:0/967794858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb41018c0 0x7f6eb4101ce0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.360+0000 7f6ebc200640 1 -- 172.21.15.103:0/967794858 >> 172.21.15.103:0/967794858 conn(0x7f6eb40fce50 msgr2=0x7f6eb40ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:20.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.360+0000 7f6ebc200640 1 -- 172.21.15.103:0/967794858 shutdown_connections 2024-05-23T00:00:20.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.361+0000 7f6ebc200640 1 -- 172.21.15.103:0/967794858 wait complete. 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.361+0000 7f6ebc200640 1 Processor -- start 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6ebc200640 1 -- start start 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6ebc200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb41018c0 0x7f6eb4197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6ebc200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb4102ab0 0x7f6eb4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6ebc200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb4103470 0x7f6eb419f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6ebc200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410d540 con 0x7f6eb4102ab0 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6ebc200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410d3c0 con 0x7f6eb41018c0 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6ebc200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb410d6c0 con 0x7f6eb4103470 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6eb9800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb41018c0 0x7f6eb4197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:20.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6eb8e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb4102ab0 0x7f6eb4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6eb9800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb41018c0 0x7f6eb4197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:59398/0 (socket says 172.21.15.103:59398) 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6eb9800640 1 -- 172.21.15.103:0/496005307 learned_addr learned my addr 172.21.15.103:0/496005307 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6eb8e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb4102ab0 0x7f6eb4198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57218/0 (socket says 172.21.15.103:57218) 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.362+0000 7f6eba200640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb4103470 0x7f6eb419f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb9800640 1 -- 172.21.15.103:0/496005307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb4103470 msgr2=0x7f6eb419f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb9800640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb4103470 0x7f6eb419f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb9800640 1 -- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb4102ab0 msgr2=0x7f6eb4198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb9800640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb4102ab0 0x7f6eb4198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb9800640 1 -- 172.21.15.103:0/496005307 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb419f850 con 0x7f6eb41018c0 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb9800640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb41018c0 0x7f6eb4197c40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6eac031610 tx=0x7f6eac004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb2000640 1 -- 172.21.15.103:0/496005307 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac03f030 con 0x7f6eb41018c0 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb2000640 1 -- 172.21.15.103:0/496005307 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eac003ad0 con 0x7f6eb41018c0 2024-05-23T00:00:20.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb4073aa0 con 0x7f6eb41018c0 2024-05-23T00:00:20.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6eb2000640 1 -- 172.21.15.103:0/496005307 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac035070 con 0x7f6eb41018c0 2024-05-23T00:00:20.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.363+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb4073fe0 con 0x7f6eb41018c0 2024-05-23T00:00:20.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.365+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb4108c00 con 0x7f6eb41018c0 2024-05-23T00:00:20.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.366+0000 7f6eb2000640 1 -- 172.21.15.103:0/496005307 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f6eac042030 con 0x7f6eb41018c0 2024-05-23T00:00:20.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.368+0000 7f6eb2000640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6e980a2c10 0x7f6e980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:20.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.368+0000 7f6eb2000640 1 -- 172.21.15.103:0/496005307 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f6eac0ef6c0 con 0x7f6eb41018c0 2024-05-23T00:00:20.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.368+0000 7f6eb8e00640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6e980a2c10 0x7f6e980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:20.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.369+0000 7f6eb8e00640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6e980a2c10 0x7f6e980a50d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00099a0 tx=0x7f6ea0012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:20.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.372+0000 7f6eb2000640 1 -- 172.21.15.103:0/496005307 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6eac0bbc10 con 0x7f6eb41018c0 2024-05-23T00:00:20.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.573+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6eb402c080 con 0x7f6eb41018c0 2024-05-23T00:00:20.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.574+0000 7f6eb2000640 1 -- 172.21.15.103:0/496005307 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6eac0bff10 con 0x7f6eb41018c0 2024-05-23T00:00:20.577 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-23T00:00:20.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:20 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/4167368321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:20.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:20 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1718524583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:20.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6e980a2c10 msgr2=0x7f6e980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6e980a2c10 0x7f6e980a50d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6ea00099a0 tx=0x7f6ea0012040 comp rx=0 tx=0).stop 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb41018c0 msgr2=0x7f6eb4197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb41018c0 0x7f6eb4197c40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6eac031610 tx=0x7f6eac004270 comp rx=0 tx=0).stop 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 shutdown_connections 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f6e980a2c10 0x7f6e980a50d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6eb4103470 0x7f6eb419f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6eb4102ab0 0x7f6eb4198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 --2- 172.21.15.103:0/496005307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6eb41018c0 0x7f6eb4197c40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 >> 172.21.15.103:0/496005307 conn(0x7f6eb40fce50 msgr2=0x7f6eb4105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 shutdown_connections 2024-05-23T00:00:20.581 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.577+0000 7f6ebc200640 1 -- 172.21.15.103:0/496005307 wait complete. 2024-05-23T00:00:20.600 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-23T00:00:20.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:20 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/4167368321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:20.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:20 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1718524583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 -- 172.21.15.103:0/302992788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9858105660 msgr2=0x7f985810bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 --2- 172.21.15.103:0/302992788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9858105660 0x7f985810bef0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9850009be0 tx=0x7f985002a6c0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 -- 172.21.15.103:0/302992788 shutdown_connections 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 --2- 172.21.15.103:0/302992788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9858105660 0x7f985810bef0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 --2- 172.21.15.103:0/302992788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9858104ca0 0x7f9858105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 --2- 172.21.15.103:0/302992788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9858103ab0 0x7f9858103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 -- 172.21.15.103:0/302992788 >> 172.21.15.103:0/302992788 conn(0x7f98580ff060 msgr2=0x7f9858101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 -- 172.21.15.103:0/302992788 shutdown_connections 2024-05-23T00:00:20.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.804+0000 7f985e800640 1 -- 172.21.15.103:0/302992788 wait complete. 2024-05-23T00:00:20.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:20 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/4167368321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:20.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:20 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1718524583' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.805+0000 7f985e800640 1 Processor -- start 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f985e800640 1 -- start start 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f985e800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9858103ab0 0x7f9858199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f985e800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9858104ca0 0x7f985819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f985e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9858105660 0x7f98581a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f985e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f985810f6e0 con 0x7f9858105660 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f985e800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f985810f560 con 0x7f9858103ab0 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f985e800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f985810f860 con 0x7f9858104ca0 2024-05-23T00:00:20.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f9857e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9858105660 0x7f98581a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f9857e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9858105660 0x7f98581a1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57232/0 (socket says 172.21.15.103:57232) 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f9857e00640 1 -- 172.21.15.103:0/953281043 learned_addr learned my addr 172.21.15.103:0/953281043 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f9857400640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9858103ab0 0x7f9858199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.806+0000 7f9856a00640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9858104ca0 0x7f985819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f9857e00640 1 -- 172.21.15.103:0/953281043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9858104ca0 msgr2=0x7f985819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f9857e00640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9858104ca0 0x7f985819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f9857e00640 1 -- 172.21.15.103:0/953281043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9858103ab0 msgr2=0x7f9858199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f9857e00640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9858103ab0 0x7f9858199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f9857e00640 1 -- 172.21.15.103:0/953281043 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98581a1970 con 0x7f9858105660 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f9857400640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9858103ab0 0x7f9858199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f9857e00640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9858105660 0x7f98581a1300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f98500305b0 tx=0x7f98500305e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f984fe00640 1 -- 172.21.15.103:0/953281043 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850041020 con 0x7f9858105660 2024-05-23T00:00:20.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f984fe00640 1 -- 172.21.15.103:0/953281043 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9850037040 con 0x7f9858105660 2024-05-23T00:00:20.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f984fe00640 1 -- 172.21.15.103:0/953281043 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850036070 con 0x7f9858105660 2024-05-23T00:00:20.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.807+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98581a1c00 con 0x7f9858105660 2024-05-23T00:00:20.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.808+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98581a20f0 con 0x7f9858105660 2024-05-23T00:00:20.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.809+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f985810add0 con 0x7f9858105660 2024-05-23T00:00:20.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.811+0000 7f984fe00640 1 -- 172.21.15.103:0/953281043 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f9850044020 con 0x7f9858105660 2024-05-23T00:00:20.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.812+0000 7f984fe00640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f983c0a2b90 0x7f983c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:20.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.812+0000 7f984fe00640 1 -- 172.21.15.103:0/953281043 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f98500eff90 con 0x7f9858105660 2024-05-23T00:00:20.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.812+0000 7f9857400640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f983c0a2b90 0x7f983c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:20.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.813+0000 7f9857400640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f983c0a2b90 0x7f983c0a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9858104b00 tx=0x7f9848005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:20.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:20.816+0000 7f984fe00640 1 -- 172.21.15.103:0/953281043 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f985004b020 con 0x7f9858105660 2024-05-23T00:00:21.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.016+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9858062510 con 0x7f9858105660 2024-05-23T00:00:21.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.017+0000 7f984fe00640 1 -- 172.21.15.103:0/953281043 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f98500bcb90 con 0x7f9858105660 2024-05-23T00:00:21.021 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:21.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f983c0a2b90 msgr2=0x7f983c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f983c0a2b90 0x7f983c0a5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9858104b00 tx=0x7f9848005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:21.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9858105660 msgr2=0x7f98581a1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9858105660 0x7f98581a1300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f98500305b0 tx=0x7f98500305e0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 shutdown_connections 2024-05-23T00:00:21.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f983c0a2b90 0x7f983c0a5050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9858105660 0x7f98581a1300 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9858104ca0 0x7f985819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 --2- 172.21.15.103:0/953281043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9858103ab0 0x7f9858199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 >> 172.21.15.103:0/953281043 conn(0x7f98580ff060 msgr2=0x7f9858107fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:21.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 shutdown_connections 2024-05-23T00:00:21.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.020+0000 7f985e800640 1 -- 172.21.15.103:0/953281043 wait complete. 2024-05-23T00:00:21.044 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:00:21.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 -- 172.21.15.103:0/2302837010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c4102390 msgr2=0x7f43c4104780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 --2- 172.21.15.103:0/2302837010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c4102390 0x7f43c4104780 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f43b4009900 tx=0x7f43b402a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 -- 172.21.15.103:0/2302837010 shutdown_connections 2024-05-23T00:00:21.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 --2- 172.21.15.103:0/2302837010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43c41075f0 0x7f43c41099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 --2- 172.21.15.103:0/2302837010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43c4104cc0 0x7f43c41070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 --2- 172.21.15.103:0/2302837010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c4102390 0x7f43c4104780 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 -- 172.21.15.103:0/2302837010 >> 172.21.15.103:0/2302837010 conn(0x7f43c40fbd60 msgr2=0x7f43c40fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:21.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 -- 172.21.15.103:0/2302837010 shutdown_connections 2024-05-23T00:00:21.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.248+0000 7f43c8e00640 1 -- 172.21.15.103:0/2302837010 wait complete. 2024-05-23T00:00:21.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.249+0000 7f43c8e00640 1 Processor -- start 2024-05-23T00:00:21.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.249+0000 7f43c8e00640 1 -- start start 2024-05-23T00:00:21.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.249+0000 7f43c8e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43c4102390 0x7f43c4101d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:21.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.249+0000 7f43c8e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43c4104cc0 0x7f43c4100340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:21.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.249+0000 7f43c8e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c41075f0 0x7f43c4100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:21.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.249+0000 7f43c8e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f43c410d520 con 0x7f43c41075f0 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.249+0000 7f43c8e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f43c410d3a0 con 0x7f43c4104cc0 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.249+0000 7f43c8e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43c410d6a0 con 0x7f43c4102390 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c41075f0 0x7f43c4100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c41075f0 0x7f43c4100880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57250/0 (socket says 172.21.15.103:57250) 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 -- 172.21.15.103:0/3467025808 learned_addr learned my addr 172.21.15.103:0/3467025808 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c1e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43c4102390 0x7f43c4101d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43b9400640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43c4104cc0 0x7f43c4100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 -- 172.21.15.103:0/3467025808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43c4102390 msgr2=0x7f43c4101d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43c4102390 0x7f43c4101d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 -- 172.21.15.103:0/3467025808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43c4104cc0 msgr2=0x7f43c4100340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43c4104cc0 0x7f43c4100340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 -- 172.21.15.103:0/3467025808 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c4101110 con 0x7f43c41075f0 2024-05-23T00:00:21.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c1e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43c4102390 0x7f43c4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:21.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.250+0000 7f43c2800640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c41075f0 0x7f43c4100880 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00efd0 tx=0x7f43bc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:21.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.251+0000 7f43bb400640 1 -- 172.21.15.103:0/3467025808 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43bc003b70 con 0x7f43c41075f0 2024-05-23T00:00:21.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.251+0000 7f43bb400640 1 -- 172.21.15.103:0/3467025808 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43bc003d10 con 0x7f43c41075f0 2024-05-23T00:00:21.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.251+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c41a1ce0 con 0x7f43c41075f0 2024-05-23T00:00:21.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.251+0000 7f43bb400640 1 -- 172.21.15.103:0/3467025808 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43bc0109f0 con 0x7f43c41075f0 2024-05-23T00:00:21.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.251+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c41a2180 con 0x7f43c41075f0 2024-05-23T00:00:21.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.252+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c40fd850 con 0x7f43c41075f0 2024-05-23T00:00:21.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.254+0000 7f43bb400640 1 -- 172.21.15.103:0/3467025808 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f43bc004020 con 0x7f43c41075f0 2024-05-23T00:00:21.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.256+0000 7f43bb400640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f43a80a2c10 0x7f43a80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:21.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.256+0000 7f43bb400640 1 -- 172.21.15.103:0/3467025808 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f43bc0cf900 con 0x7f43c41075f0 2024-05-23T00:00:21.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.256+0000 7f43c1e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f43a80a2c10 0x7f43a80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:21.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.257+0000 7f43c1e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f43a80a2c10 0x7f43a80a50d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f43b4000c00 tx=0x7f43b40095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:21.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.260+0000 7f43bb400640 1 -- 172.21.15.103:0/3467025808 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f43bc09c4a0 con 0x7f43c41075f0 2024-05-23T00:00:21.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.510+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43c4062470 con 0x7f43c41075f0 2024-05-23T00:00:21.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.511+0000 7f43bb400640 1 -- 172.21.15.103:0/3467025808 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11896 (secure 0 0 0) 0x7f43bc09bda0 con 0x7f43c41075f0 2024-05-23T00:00:21.515 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:21.515 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-22T23:58:05.260333+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24559,24563,24575]},"id":1}]} 2024-05-23T00:00:21.515 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 24 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f43a80a2c10 msgr2=0x7f43a80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f43a80a2c10 0x7f43a80a50d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f43b4000c00 tx=0x7f43b40095f0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c41075f0 msgr2=0x7f43c4100880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c41075f0 0x7f43c4100880 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f43bc00efd0 tx=0x7f43bc00c6a0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 shutdown_connections 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f43a80a2c10 0x7f43a80a50d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f43c41075f0 0x7f43c4100880 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f43c4104cc0 0x7f43c4100340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 --2- 172.21.15.103:0/3467025808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43c4102390 0x7f43c4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 >> 172.21.15.103:0/3467025808 conn(0x7f43c40fbd60 msgr2=0x7f43c4102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:21.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 shutdown_connections 2024-05-23T00:00:21.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.515+0000 7f43c8e00640 1 -- 172.21.15.103:0/3467025808 wait complete. 2024-05-23T00:00:21.538 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:00:21.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:21 smithi103 ceph-mon[31370]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:21.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:21 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/496005307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:21.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:21 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/953281043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:21.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:21 smithi174 ceph-mon[36179]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:21.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:21 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/496005307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:21.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:21 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/953281043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:21.747 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 -- 172.21.15.103:0/1527988647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460101430 msgr2=0x7fd4601018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 --2- 172.21.15.103:0/1527988647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460101430 0x7fd4601018b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd450009900 tx=0x7fd45002a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 -- 172.21.15.103:0/1527988647 shutdown_connections 2024-05-23T00:00:21.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 --2- 172.21.15.103:0/1527988647 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460101430 0x7fd4601018b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 --2- 172.21.15.103:0/1527988647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd460100a90 0x7fd460100ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 --2- 172.21.15.103:0/1527988647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd460100130 0x7fd460100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 -- 172.21.15.103:0/1527988647 >> 172.21.15.103:0/1527988647 conn(0x7fd4600fbd50 msgr2=0x7fd4600fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:21.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 -- 172.21.15.103:0/1527988647 shutdown_connections 2024-05-23T00:00:21.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.744+0000 7fd465e00640 1 -- 172.21.15.103:0/1527988647 wait complete. 2024-05-23T00:00:21.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd465e00640 1 Processor -- start 2024-05-23T00:00:21.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd465e00640 1 -- start start 2024-05-23T00:00:21.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd465e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd460100130 0x7fd460199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:21.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd465e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460100a90 0x7fd46019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:21.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd465e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd460101430 0x7fd4601a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd465e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd46010f710 con 0x7fd460100a90 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd465e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd46010f590 con 0x7fd460100130 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd465e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd46010f890 con 0x7fd460101430 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.745+0000 7fd45e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460100a90 0x7fd46019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd460100130 0x7fd460199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460100a90 0x7fd46019a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57258/0 (socket says 172.21.15.103:57258) 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd460100130 0x7fd460199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:59432/0 (socket says 172.21.15.103:59432) 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45e000640 1 -- 172.21.15.103:0/844296004 learned_addr learned my addr 172.21.15.103:0/844296004 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45f400640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd460101430 0x7fd4601a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45e000640 1 -- 172.21.15.103:0/844296004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd460101430 msgr2=0x7fd4601a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45e000640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd460101430 0x7fd4601a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45e000640 1 -- 172.21.15.103:0/844296004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd460100130 msgr2=0x7fd460199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45e000640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd460100130 0x7fd460199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45e000640 1 -- 172.21.15.103:0/844296004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4601a1960 con 0x7fd460100a90 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45ea00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd460100130 0x7fd460199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.746+0000 7fd45e000640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460100a90 0x7fd46019a290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd448009d50 tx=0x7fd44800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.747+0000 7fd464a00640 1 -- 172.21.15.103:0/844296004 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448003b70 con 0x7fd460100a90 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.747+0000 7fd464a00640 1 -- 172.21.15.103:0/844296004 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd448003d10 con 0x7fd460100a90 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.747+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4601a1c50 con 0x7fd460100a90 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.747+0000 7fd464a00640 1 -- 172.21.15.103:0/844296004 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd448010a20 con 0x7fd460100a90 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.747+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4601a21a0 con 0x7fd460100a90 2024-05-23T00:00:21.752 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.748+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd46010afd0 con 0x7fd460100a90 2024-05-23T00:00:21.753 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.750+0000 7fd464a00640 1 -- 172.21.15.103:0/844296004 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fd448003ec0 con 0x7fd460100a90 2024-05-23T00:00:21.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.752+0000 7fd464a00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4440a2b90 0x7fd4440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:21.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.752+0000 7fd464a00640 1 -- 172.21.15.103:0/844296004 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fd4480cfae0 con 0x7fd460100a90 2024-05-23T00:00:21.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.752+0000 7fd45ea00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4440a2b90 0x7fd4440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:21.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.753+0000 7fd45ea00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4440a2b90 0x7fd4440a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd46019b220 tx=0x7fd458005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:21.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.757+0000 7fd464a00640 1 -- 172.21.15.103:0/844296004 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd44809c6e0 con 0x7fd460100a90 2024-05-23T00:00:21.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:21 smithi105 ceph-mon[36426]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-05-23T00:00:21.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:21 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/496005307' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:21.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:21 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/953281043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:21.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.961+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd460062470 con 0x7fd460100a90 2024-05-23T00:00:21.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.965+0000 7fd464a00640 1 -- 172.21.15.103:0/844296004 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24658 (secure 0 0 0) 0x7fd44809bfe0 con 0x7fd460100a90 2024-05-23T00:00:21.969 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:21.970 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:58:31.088381+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:00:21.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.967+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4440a2b90 msgr2=0x7fd4440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.967+0000 7fd465e00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4440a2b90 0x7fd4440a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd46019b220 tx=0x7fd458005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:21.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.967+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460100a90 msgr2=0x7fd46019a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:21.972 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.967+0000 7fd465e00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460100a90 0x7fd46019a290 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd448009d50 tx=0x7fd44800c6a0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.968+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 shutdown_connections 2024-05-23T00:00:21.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.968+0000 7fd465e00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fd4440a2b90 0x7fd4440a5050 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.968+0000 7fd465e00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd460101430 0x7fd4601a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.968+0000 7fd465e00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd460100a90 0x7fd46019a290 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.968+0000 7fd465e00640 1 --2- 172.21.15.103:0/844296004 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd460100130 0x7fd460199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:21.973 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.968+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 >> 172.21.15.103:0/844296004 conn(0x7fd4600fbd50 msgr2=0x7fd4600fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:21.974 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.968+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 shutdown_connections 2024-05-23T00:00:21.974 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:21.968+0000 7fd465e00640 1 -- 172.21.15.103:0/844296004 wait complete. 2024-05-23T00:00:21.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-22T23:57:32.804111+0000', 'modified': '2024-05-22T23:58:05.260333+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24575, 'mds_1': 24563, 'mds_2': 24559}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6837/2433750399', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 2433750399}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 2433750399}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6835/3839469517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6834', 'nonce': 3839469517}, {'type': 'v1', 'addr': '172.21.15.174:6835', 'nonce': 3839469517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.174:6837/2641083628', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 2641083628}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 2641083628}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24559': {'gid': 24559, 'name': 'i', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.174:6839/3021454884', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6838', 'nonce': 3021454884}, {'type': 'v1', 'addr': '172.21.15.174:6839', 'nonce': 3021454884}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24563': {'gid': 24563, 'name': 'a', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.103:6841/4258722678', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6840', 'nonce': 4258722678}, {'type': 'v1', 'addr': '172.21.15.103:6841', 'nonce': 4258722678}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.105:6839/160741513', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 160741513}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 160741513}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24575, 'qdb_cluster': [24559, 24563, 24575]} 2024-05-23T00:00:21.992 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-23T00:00:21.992 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-23T00:00:22.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 -- 172.21.15.103:0/3282556638 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c1039b0 msgr2=0x7f109c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 --2- 172.21.15.103:0/3282556638 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c1039b0 0x7f109c103dd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1090009900 tx=0x7f109002a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 -- 172.21.15.103:0/3282556638 shutdown_connections 2024-05-23T00:00:22.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 --2- 172.21.15.103:0/3282556638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f109c105560 0x7f109c10bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 --2- 172.21.15.103:0/3282556638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f109c104ba0 0x7f109c105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 --2- 172.21.15.103:0/3282556638 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c1039b0 0x7f109c103dd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 -- 172.21.15.103:0/3282556638 >> 172.21.15.103:0/3282556638 conn(0x7f109c0fef20 msgr2=0x7f109c101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 -- 172.21.15.103:0/3282556638 shutdown_connections 2024-05-23T00:00:22.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.240+0000 7f10a2400640 1 -- 172.21.15.103:0/3282556638 wait complete. 2024-05-23T00:00:22.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.241+0000 7f10a2400640 1 Processor -- start 2024-05-23T00:00:22.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.241+0000 7f10a2400640 1 -- start start 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.241+0000 7f10a2400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f109c1039b0 0x7f109c199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.241+0000 7f10a2400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f109c104ba0 0x7f109c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a2400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c105560 0x7f109c1a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a2400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f109c10f630 con 0x7f109c1039b0 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a2400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f109c10f4b0 con 0x7f109c105560 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a2400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f109c10f7b0 con 0x7f109c104ba0 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f109c1039b0 0x7f109c199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f109c1039b0 0x7f109c199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57268/0 (socket says 172.21.15.103:57268) 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1000640 1 -- 172.21.15.103:0/4272955374 learned_addr learned my addr 172.21.15.103:0/4272955374 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1a00640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c105560 0x7f109c1a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f109be00640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f109c104ba0 0x7f109c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1a00640 1 -- 172.21.15.103:0/4272955374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f109c104ba0 msgr2=0x7f109c19a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1a00640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f109c104ba0 0x7f109c19a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1a00640 1 -- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f109c1039b0 msgr2=0x7f109c199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1a00640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f109c1039b0 0x7f109c199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1a00640 1 -- 172.21.15.103:0/4272955374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109c1a18e0 con 0x7f109c105560 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.242+0000 7f10a1000640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f109c1039b0 0x7f109c199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.243+0000 7f10a1a00640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c105560 0x7f109c1a1240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1094009fd0 tx=0x7f109400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.243+0000 7f1099600640 1 -- 172.21.15.103:0/4272955374 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109400ce20 con 0x7f109c105560 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.243+0000 7f10a2400640 1 -- 172.21.15.103:0/4272955374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109c1a1bd0 con 0x7f109c105560 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.243+0000 7f1099600640 1 -- 172.21.15.103:0/4272955374 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1094005020 con 0x7f109c105560 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.243+0000 7f1099600640 1 -- 172.21.15.103:0/4272955374 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1094010720 con 0x7f109c105560 2024-05-23T00:00:22.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.243+0000 7f10a2400640 1 -- 172.21.15.103:0/4272955374 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109c1a2170 con 0x7f109c105560 2024-05-23T00:00:22.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.246+0000 7f10a67dc740 1 Processor -- start 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f1099600640 1 -- 172.21.15.103:0/4272955374 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f1094005540 con 0x7f109c105560 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a67dc740 1 -- start start 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60450 0x562d5ef60870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef65940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef65fb0 0x562d5ef68440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a67dc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562d5ef27db0 con 0x562d5ef65fb0 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a67dc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562d5eec4490 con 0x562d5ef60450 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a67dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562d5ee6c100 con 0x562d5ef60e40 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a1000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef65940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a1000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef65940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:36070/0 (socket says 172.21.15.103:36070) 2024-05-23T00:00:22.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f109be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60450 0x562d5ef60870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f10a1000640 1 -- 172.21.15.103:0/1994025842 learned_addr learned my addr 172.21.15.103:0/1994025842 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.247+0000 7f109be00640 1 --2- 172.21.15.103:0/1994025842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef65fb0 0x562d5ef68440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f10a1000640 1 -- 172.21.15.103:0/1994025842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60450 msgr2=0x562d5ef60870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f10a1000640 1 --2- 172.21.15.103:0/1994025842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60450 0x562d5ef60870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f10a1000640 1 -- 172.21.15.103:0/1994025842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef65fb0 msgr2=0x562d5ef68440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f10a1000640 1 --2- 172.21.15.103:0/1994025842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef65fb0 0x562d5ef68440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f10a1000640 1 -- 172.21.15.103:0/1994025842 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d5ebd82d0 con 0x562d5ef60e40 2024-05-23T00:00:22.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f10a1000640 1 --2- 172.21.15.103:0/1994025842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef65940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1090000c00 tx=0x7f10900316b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c9a01db7379311f server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f108e000640 1 -- 172.21.15.103:0/1994025842 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090035040 con 0x562d5ef60e40 2024-05-23T00:00:22.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f1099600640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f10840a2c10 0x7f10840a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f108e000640 1 -- 172.21.15.103:0/1994025842 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10900035f0 con 0x562d5ef60e40 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.248+0000 7f108e000640 1 -- 172.21.15.103:0/1994025842 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f109003c640 con 0x562d5ef60e40 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f1099600640 1 -- 172.21.15.103:0/4272955374 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f10940cf990 con 0x7f109c105560 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f109be00640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f10840a2c10 0x7f10840a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 -- 172.21.15.103:0/1994025842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 msgr2=0x562d5ef65940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 --2- 172.21.15.103:0/1994025842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef65940 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f1090000c00 tx=0x7f10900316b0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 -- 172.21.15.103:0/1994025842 shutdown_connections 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 --2- 172.21.15.103:0/1994025842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef65fb0 0x562d5ef68440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 --2- 172.21.15.103:0/1994025842 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef65940 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 --2- 172.21.15.103:0/1994025842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60450 0x562d5ef60870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 -- 172.21.15.103:0/1994025842 >> 172.21.15.103:0/1994025842 conn(0x562d5ef5bac0 msgr2=0x562d5ef5bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 -- 172.21.15.103:0/1994025842 shutdown_connections 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f109be00640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f10840a2c10 0x7f10840a50d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x562d5ef616a0 tx=0x7f108800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.249+0000 7f10a67dc740 1 -- 172.21.15.103:0/1994025842 wait complete. 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.250+0000 7f10a67dc740 1 -- start start 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.250+0000 7f10a67dc740 1 Processor -- start 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.250+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.250+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef6ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.250+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef71c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.250+0000 7f10a67dc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562d5ee7f520 con 0x562d5ef60450 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.250+0000 7f10a67dc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562d5eec0450 con 0x562d5ef65fb0 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.250+0000 7f10a67dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562d5ef68f00 con 0x562d5ef60e40 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a67dc740 10 client.0 _ll_register_callbacks cb 0x562d5ebd7ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:22.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57282/0 (socket says 172.21.15.103:57282) 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1000640 1 -- 172.21.15.103:0/2643891489 learned_addr learned my addr 172.21.15.103:0/2643891489 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f109be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef6ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1a00640 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef71c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1000640 1 -- 172.21.15.103:0/2643891489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 msgr2=0x562d5ef6ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1000640 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef6ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1000640 1 -- 172.21.15.103:0/2643891489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 msgr2=0x562d5ef71c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1000640 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef71c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1000640 1 -- 172.21.15.103:0/2643891489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d5eca26b0 con 0x562d5ef60450 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.251+0000 7f10a1a00640 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef71c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.252+0000 7f10a1000640 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6a720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1090005ab0 tx=0x7f1090004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53dede97ca557c91 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.252+0000 7f1082a00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.252+0000 7f1082a00640 1 -- 172.21.15.103:0/2643891489 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090035040 con 0x562d5ef60450 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.252+0000 7f1082a00640 1 -- 172.21.15.103:0/2643891489 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1090004480 con 0x562d5ef60450 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.252+0000 7f10a67dc740 1 -- 172.21.15.103:0/2643891489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10900035f0 con 0x562d5ef60450 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.252+0000 7f1082a00640 1 -- 172.21.15.103:0/2643891489 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f10900039b0 con 0x562d5ef60450 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.252+0000 7f10a67dc740 10 client.15222 fetch_fsmap learned FSMap version 24 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.252+0000 7f10a67dc740 1 -- 172.21.15.103:0/2643891489 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562d5ef74fa0 con 0x562d5ef60450 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f1082a00640 1 -- 172.21.15.103:0/2643891489 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10900035f0 con 0x562d5ef60450 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f1082a00640 1 -- 172.21.15.103:0/2643891489 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1090046400 con 0x562d5ef60450 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f10a67dc740 10 client.15222 fetch_fsmap finished waiting for FSMap version 24 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f10a67dc740 10 client.15222 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f10a67dc740 20 client.15222 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f10a67dc740 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x562d5ef77120 0x562d5ef78590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f10a67dc740 4 client.15222 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:00:22.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f10a67dc740 1 -- 172.21.15.103:0/2643891489 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d5eed1bc0 con 0x562d5ef77120 2024-05-23T00:00:22.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.253+0000 7f109be00640 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x562d5ef77120 0x562d5ef78590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.254+0000 7f109be00640 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x562d5ef77120 0x562d5ef78590 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=47a07d12d465eeea server_cookie=34e7481024bbbe30 in_seq=0 out_seq=0 2024-05-23T00:00:22.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.254+0000 7f1082a00640 10 client.15222 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-23T00:00:22.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f1082a00640 1 -- 172.21.15.103:0/2643891489 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562d5eed1bc0 con 0x562d5ef77120 2024-05-23T00:00:22.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f1082a00640 10 client.15222 handle_command_reply: tid=0 2024-05-23T00:00:22.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 client.15222 shutdown 2024-05-23T00:00:22.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 -- 172.21.15.103:0/2643891489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 msgr2=0x562d5ef6a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6a720 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1090005ab0 tx=0x7f1090004290 comp rx=0 tx=0).stop 2024-05-23T00:00:22.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 -- 172.21.15.103:0/2643891489 shutdown_connections 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x562d5ef77120 0x562d5ef78590 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef71c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60e40 0x562d5ef6ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f1082a00640 0 client.15222 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 --2- 172.21.15.103:0/2643891489 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6a720 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 -- 172.21.15.103:0/2643891489 >> 172.21.15.103:0/2643891489 conn(0x562d5ef5bac0 msgr2=0x562d5ef5ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 -- 172.21.15.103:0/2643891489 shutdown_connections 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 1 -- 172.21.15.103:0/2643891489 wait complete. 2024-05-23T00:00:22.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.255+0000 7f10a67dc740 20 client.15222 trim_cache size 0 max 16384 2024-05-23T00:00:22.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a67dc740 1 Processor -- start 2024-05-23T00:00:22.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a67dc740 1 -- start start 2024-05-23T00:00:22.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60450 0x562d5ef5e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60e40 0x562d5ef71bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef5eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a67dc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562d5ef768d0 con 0x562d5ef60e40 2024-05-23T00:00:22.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a67dc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562d5ef76fa0 con 0x562d5ef65fb0 2024-05-23T00:00:22.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a67dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562d5ef77670 con 0x562d5ef60450 2024-05-23T00:00:22.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f109be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60e40 0x562d5ef71bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f109be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60e40 0x562d5ef71bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57296/0 (socket says 172.21.15.103:57296) 2024-05-23T00:00:22.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f109be00640 1 -- 172.21.15.103:0/3899994891 learned_addr learned my addr 172.21.15.103:0/3899994891 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.275+0000 7f10a1a00640 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef5eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f10a1000640 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60450 0x562d5ef5e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f109be00640 1 -- 172.21.15.103:0/3899994891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60450 msgr2=0x562d5ef5e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f109be00640 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60450 0x562d5ef5e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f109be00640 1 -- 172.21.15.103:0/3899994891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 msgr2=0x562d5ef5eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f109be00640 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef5eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f109be00640 1 -- 172.21.15.103:0/3899994891 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d5ef5b8f0 con 0x562d5ef60e40 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f10a1000640 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60450 0x562d5ef5e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f109be00640 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60e40 0x562d5ef71bf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f108800c010 tx=0x7f1088014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0f1e4ed8e39b75 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f108d600640 1 -- 172.21.15.103:0/3899994891 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1088031070 con 0x562d5ef60e40 2024-05-23T00:00:22.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f108d600640 1 -- 172.21.15.103:0/3899994891 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1088020d60 con 0x562d5ef60e40 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.276+0000 7f108d600640 1 -- 172.21.15.103:0/3899994891 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f108802a7f0 con 0x562d5ef60e40 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 -- 172.21.15.103:0/3899994891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60e40 msgr2=0x562d5ef71bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60e40 0x562d5ef71bf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f108800c010 tx=0x7f1088014040 comp rx=0 tx=0).stop 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 -- 172.21.15.103:0/3899994891 shutdown_connections 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef65fb0 0x562d5ef5eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60e40 0x562d5ef71bf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 --2- 172.21.15.103:0/3899994891 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef60450 0x562d5ef5e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 -- 172.21.15.103:0/3899994891 >> 172.21.15.103:0/3899994891 conn(0x562d5ef5bac0 msgr2=0x562d5ef5bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 -- 172.21.15.103:0/3899994891 shutdown_connections 2024-05-23T00:00:22.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.277+0000 7f10a67dc740 1 -- 172.21.15.103:0/3899994891 wait complete. 2024-05-23T00:00:22.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 1 -- start start 2024-05-23T00:00:22.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 1 Processor -- start 2024-05-23T00:00:22.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60e40 0x562d5ef6d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef65fb0 0x562d5ef70920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562d5ef78410 con 0x562d5ef60450 2024-05-23T00:00:22.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x562d5ef77d40 con 0x562d5ef60e40 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x562d5ef78ae0 con 0x562d5ef65fb0 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.278+0000 7f10a67dc740 10 client.0 _ll_register_callbacks cb 0x7f1090040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f109be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60e40 0x562d5ef6d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57300/0 (socket says 172.21.15.103:57300) 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 -- 172.21.15.103:0/227149279 learned_addr learned my addr 172.21.15.103:0/227149279 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1a00640 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef65fb0 0x562d5ef70920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 -- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef65fb0 msgr2=0x562d5ef70920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef65fb0 0x562d5ef70920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 -- 172.21.15.103:0/227149279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60e40 msgr2=0x562d5ef6d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60e40 0x562d5ef6d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 -- 172.21.15.103:0/227149279 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d5ef69070 con 0x562d5ef60450 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1a00640 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef65fb0 0x562d5ef70920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a1000640 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6d3c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1090031610 tx=0x7f10900045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a64acb72597e25 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f1082a00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f1082a00640 1 -- 172.21.15.103:0/227149279 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1090035040 con 0x562d5ef60450 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f1082a00640 1 -- 172.21.15.103:0/227149279 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f1090003b50 con 0x562d5ef60450 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.279+0000 7f10a67dc740 1 -- 172.21.15.103:0/227149279 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1088020d60 con 0x562d5ef60450 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f1082a00640 1 -- 172.21.15.103:0/227149279 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f10900035f0 con 0x562d5ef60450 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f10a67dc740 10 client.15228 fetch_fsmap learned FSMap version 24 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f10a67dc740 1 -- 172.21.15.103:0/227149279 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x562d5ef71fe0 con 0x562d5ef60450 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f1082a00640 1 -- 172.21.15.103:0/227149279 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1088020d60 con 0x562d5ef60450 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f1082a00640 1 -- 172.21.15.103:0/227149279 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f109003c740 con 0x562d5ef60450 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f10a67dc740 10 client.15228 fetch_fsmap finished waiting for FSMap version 24 2024-05-23T00:00:22.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f10a67dc740 10 client.15228 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f10a67dc740 20 client.15228 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f10a67dc740 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x562d5efe1b50 0x562d5efe0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f10a67dc740 4 client.15228 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.280+0000 7f10a67dc740 1 -- 172.21.15.103:0/227149279 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "status"}) -- 0x562d5eecbf10 con 0x562d5efe1b50 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.281+0000 7f109be00640 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x562d5efe1b50 0x562d5efe0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.281+0000 7f1082a00640 10 client.15228 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.281+0000 7f109be00640 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x562d5efe1b50 0x562d5efe0860 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=69bc9b86141c3b70 server_cookie=897e28f34d6f4cd in_seq=0 out_seq=0 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.281+0000 7f1082a00640 1 -- 172.21.15.103:0/227149279 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562d5eecbf10 con 0x562d5efe1b50 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.281+0000 7f1082a00640 10 client.15228 handle_command_reply: tid=0 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.281+0000 7f10a67dc740 1 client.15228 shutdown 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 -- 172.21.15.103:0/227149279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 msgr2=0x562d5ef6d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6d3c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1090031610 tx=0x7f10900045b0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 -- 172.21.15.103:0/227149279 shutdown_connections 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x562d5efe1b50 0x562d5efe0860 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x562d5ef65fb0 0x562d5ef70920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x562d5ef60e40 0x562d5ef6d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 --2- 172.21.15.103:0/227149279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562d5ef60450 0x562d5ef6d3c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f1082a00640 0 client.15228 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 -- 172.21.15.103:0/227149279 >> 172.21.15.103:0/227149279 conn(0x562d5ef5bac0 msgr2=0x562d5ef5a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 -- 172.21.15.103:0/227149279 shutdown_connections 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 1 -- 172.21.15.103:0/227149279 wait complete. 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.282+0000 7f10a67dc740 20 client.15228 trim_cache size 0 max 16384 2024-05-23T00:00:22.290 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "whoami": 2, 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 24559, 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "want_state": "up:active", 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "up:active", 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs", 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "rank_uptime": 138.038457897, 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "mdsmap_epoch": 24, 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch": 93, 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch_barrier": 91, 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout: "uptime": 186.32849200499999 2024-05-23T00:00:22.291 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:00:22.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 -- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f10840a2c10 msgr2=0x7f10840a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f10840a2c10 0x7f10840a50d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x562d5ef616a0 tx=0x7f108800b390 comp rx=0 tx=0).stop 2024-05-23T00:00:22.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 -- 172.21.15.103:0/4272955374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c105560 msgr2=0x7f109c1a1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c105560 0x7f109c1a1240 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1094009fd0 tx=0x7f109400ef10 comp rx=0 tx=0).stop 2024-05-23T00:00:22.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 -- 172.21.15.103:0/4272955374 shutdown_connections 2024-05-23T00:00:22.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f10840a2c10 0x7f10840a50d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f109c105560 0x7f109c1a1240 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f109c104ba0 0x7f109c19a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 --2- 172.21.15.103:0/4272955374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f109c1039b0 0x7f109c199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 -- 172.21.15.103:0/4272955374 >> 172.21.15.103:0/4272955374 conn(0x7f109c0fef20 msgr2=0x7f109c107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 -- 172.21.15.103:0/4272955374 shutdown_connections 2024-05-23T00:00:22.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.283+0000 7f10a2400640 1 -- 172.21.15.103:0/4272955374 wait complete. 2024-05-23T00:00:22.304 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-05-23T00:00:22.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 -- 172.21.15.103:0/1448688126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde18104cc0 msgr2=0x7fde18105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 --2- 172.21.15.103:0/1448688126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde18104cc0 0x7fde18105140 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fde00009900 tx=0x7fde0002a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 -- 172.21.15.103:0/1448688126 shutdown_connections 2024-05-23T00:00:22.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 --2- 172.21.15.103:0/1448688126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde18105680 0x7fde1810bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 --2- 172.21.15.103:0/1448688126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde18104cc0 0x7fde18105140 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 --2- 172.21.15.103:0/1448688126 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde18103ad0 0x7fde18103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 -- 172.21.15.103:0/1448688126 >> 172.21.15.103:0/1448688126 conn(0x7fde180ff060 msgr2=0x7fde18101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 -- 172.21.15.103:0/1448688126 shutdown_connections 2024-05-23T00:00:22.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.500+0000 7fde1de00640 1 -- 172.21.15.103:0/1448688126 wait complete. 2024-05-23T00:00:22.506 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.501+0000 7fde1de00640 1 Processor -- start 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.501+0000 7fde1de00640 1 -- start start 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde1de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde18103ad0 0x7fde18199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde1de00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde18104cc0 0x7fde1819a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde1de00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde18105680 0x7fde181a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde1de00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde1810f7a0 con 0x7fde18103ad0 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde1de00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fde1810f620 con 0x7fde18104cc0 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde1de00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fde1810f920 con 0x7fde18105680 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde16a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde18103ad0 0x7fde18199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde16a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde18103ad0 0x7fde18199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57306/0 (socket says 172.21.15.103:57306) 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde16a00640 1 -- 172.21.15.103:0/2074313257 learned_addr learned my addr 172.21.15.103:0/2074313257 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde0fe00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde18104cc0 0x7fde1819a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.502+0000 7fde17400640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde18105680 0x7fde181a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde16a00640 1 -- 172.21.15.103:0/2074313257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde18105680 msgr2=0x7fde181a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde16a00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde18105680 0x7fde181a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde16a00640 1 -- 172.21.15.103:0/2074313257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde18104cc0 msgr2=0x7fde1819a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde16a00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde18104cc0 0x7fde1819a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde16a00640 1 -- 172.21.15.103:0/2074313257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde181a1a30 con 0x7fde18103ad0 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde16a00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde18103ad0 0x7fde18199e10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fde0800e9c0 tx=0x7fde0800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde1ca00640 1 -- 172.21.15.103:0/2074313257 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0800ccc0 con 0x7fde18103ad0 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde1de00640 1 -- 172.21.15.103:0/2074313257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde181a1d20 con 0x7fde18103ad0 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde1ca00640 1 -- 172.21.15.103:0/2074313257 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde080035f0 con 0x7fde18103ad0 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde1ca00640 1 -- 172.21.15.103:0/2074313257 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0800ce60 con 0x7fde18103ad0 2024-05-23T00:00:22.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.503+0000 7fde1de00640 1 -- 172.21.15.103:0/2074313257 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde181a2190 con 0x7fde18103ad0 2024-05-23T00:00:22.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde20b64740 1 Processor -- start 2024-05-23T00:00:22.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde20b64740 1 -- start start 2024-05-23T00:00:22.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde1ca00640 1 -- 172.21.15.103:0/2074313257 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fde080037a0 con 0x7fde18103ad0 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde20b64740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde20b64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c77770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde20b64740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 0x55d7f6c7a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde20b64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6c39db0 con 0x55d7f6c72c70 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde20b64740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6bd6490 con 0x55d7f6c77de0 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde20b64740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6b7e100 con 0x55d7f6c72280 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.507+0000 7fde16a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c77770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde16a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c77770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57318/0 (socket says 172.21.15.103:57318) 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde17400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde16a00640 1 -- 172.21.15.103:0/1306554590 learned_addr learned my addr 172.21.15.103:0/1306554590 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde0fe00640 1 --2- 172.21.15.103:0/1306554590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 0x55d7f6c7a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde17400640 1 -- 172.21.15.103:0/1306554590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 msgr2=0x55d7f6c7a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde17400640 1 --2- 172.21.15.103:0/1306554590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 0x55d7f6c7a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde17400640 1 -- 172.21.15.103:0/1306554590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 msgr2=0x55d7f6c77770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde17400640 1 --2- 172.21.15.103:0/1306554590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c77770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde17400640 1 -- 172.21.15.103:0/1306554590 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f68ea2d0 con 0x55d7f6c72280 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.508+0000 7fde17400640 1 --2- 172.21.15.103:0/1306554590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c726a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fde10009f10 tx=0x7fde10011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcd98f97eb6118d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.509+0000 7fde1ca00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddfc0a2b90 0x7fddfc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.509+0000 7fde07e00640 1 -- 172.21.15.103:0/1306554590 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10004d20 con 0x55d7f6c72280 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.509+0000 7fde07e00640 1 -- 172.21.15.103:0/1306554590 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde10004030 con 0x55d7f6c72280 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.509+0000 7fde07e00640 1 -- 172.21.15.103:0/1306554590 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10005260 con 0x55d7f6c72280 2024-05-23T00:00:22.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.509+0000 7fde1ca00640 1 -- 172.21.15.103:0/2074313257 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fde080cf760 con 0x7fde18103ad0 2024-05-23T00:00:22.515 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.509+0000 7fde16a00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddfc0a2b90 0x7fddfc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.509+0000 7fde20b64740 1 -- 172.21.15.103:0/1306554590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 msgr2=0x55d7f6c726a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.509+0000 7fde20b64740 1 --2- 172.21.15.103:0/1306554590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c726a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fde10009f10 tx=0x7fde10011810 comp rx=0 tx=0).stop 2024-05-23T00:00:22.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 -- 172.21.15.103:0/1306554590 shutdown_connections 2024-05-23T00:00:22.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 --2- 172.21.15.103:0/1306554590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 0x55d7f6c7a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 --2- 172.21.15.103:0/1306554590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c77770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 --2- 172.21.15.103:0/1306554590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c726a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 -- 172.21.15.103:0/1306554590 >> 172.21.15.103:0/1306554590 conn(0x55d7f6c6d8f0 msgr2=0x55d7f6c6dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde16a00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddfc0a2b90 0x7fddfc0a5050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fde08004630 tx=0x7fde080d5530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 -- 172.21.15.103:0/1306554590 shutdown_connections 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 -- 172.21.15.103:0/1306554590 wait complete. 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 -- start start 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.510+0000 7fde20b64740 1 Processor -- start 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde20b64740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c7c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde20b64740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde20b64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c83b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde20b64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6b91520 con 0x55d7f6c77de0 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde20b64740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6bd2450 con 0x55d7f6c72280 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde20b64740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6c7ad90 con 0x55d7f6c72c70 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde20b64740 10 client.0 _ll_register_callbacks cb 0x55d7f68e9ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde17400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c83b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde0fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde17400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c83b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57330/0 (socket says 172.21.15.103:57330) 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde17400640 1 -- 172.21.15.103:0/1812879297 learned_addr learned my addr 172.21.15.103:0/1812879297 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.511+0000 7fde17400640 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c7c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.512+0000 7fde0fe00640 1 -- 172.21.15.103:0/1812879297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 msgr2=0x55d7f6c7c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.512+0000 7fde0fe00640 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c7c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.512+0000 7fde0fe00640 1 -- 172.21.15.103:0/1812879297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 msgr2=0x55d7f6c83b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.512+0000 7fde0fe00640 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c83b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.512+0000 7fde0fe00640 1 -- 172.21.15.103:0/1812879297 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f69b46b0 con 0x55d7f6c72c70 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.512+0000 7fde0fe00640 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7cac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fde0002a910 tx=0x7fde000043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d40cd4aedc8e7196 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.512+0000 7fde04c00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.512+0000 7fde04c00640 1 -- 172.21.15.103:0/1812879297 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00035040 con 0x55d7f6c72c70 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.513+0000 7fde20b64740 1 -- 172.21.15.103:0/1812879297 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde10004030 con 0x55d7f6c72c70 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.513+0000 7fde04c00640 1 -- 172.21.15.103:0/1812879297 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fde00005450 con 0x55d7f6c72c70 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.513+0000 7fde04c00640 1 -- 172.21.15.103:0/1812879297 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fde0004f050 con 0x55d7f6c72c70 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.513+0000 7fde20b64740 10 client.25348 fetch_fsmap learned FSMap version 24 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.513+0000 7fde20b64740 1 -- 172.21.15.103:0/1812879297 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d7f6c86f30 con 0x55d7f6c72c70 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.513+0000 7fde04c00640 1 -- 172.21.15.103:0/1812879297 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde10004030 con 0x55d7f6c72c70 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.513+0000 7fde04c00640 1 -- 172.21.15.103:0/1812879297 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fde00031a50 con 0x55d7f6c72c70 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.514+0000 7fde20b64740 10 client.25348 fetch_fsmap finished waiting for FSMap version 24 2024-05-23T00:00:22.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.514+0000 7fde20b64740 10 client.25348 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.514+0000 7fde20b64740 20 client.25348 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.514+0000 7fde20b64740 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55d7f6c89550 0x55d7f6c8a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.514+0000 7fde20b64740 4 client.25348 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.514+0000 7fde20b64740 1 -- 172.21.15.103:0/1812879297 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7f6be3bc0 con 0x55d7f6c89550 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.514+0000 7fde17400640 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55d7f6c89550 0x55d7f6c8a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.514+0000 7fde17400640 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55d7f6c89550 0x55d7f6c8a9c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24a9113d9cb21d2e server_cookie=f204c4a9e607a187 in_seq=0 out_seq=0 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.515+0000 7fde04c00640 10 client.25348 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.515+0000 7fde04c00640 1 -- 172.21.15.103:0/1812879297 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7f6be3bc0 con 0x55d7f6c89550 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.515+0000 7fde04c00640 10 client.25348 handle_command_reply: tid=0 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.515+0000 7fde20b64740 1 client.25348 shutdown 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 -- 172.21.15.103:0/1812879297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 msgr2=0x55d7f6c7cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7cac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fde0002a910 tx=0x7fde000043b0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 -- 172.21.15.103:0/1812879297 shutdown_connections 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c83b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55d7f6c89550 0x55d7f6c8a9c0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7cac0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 --2- 172.21.15.103:0/1812879297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c7c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 -- 172.21.15.103:0/1812879297 >> 172.21.15.103:0/1812879297 conn(0x55d7f6c6d8f0 msgr2=0x55d7f6c6cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde04c00640 0 client.25348 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 -- 172.21.15.103:0/1812879297 shutdown_connections 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 1 -- 172.21.15.103:0/1812879297 wait complete. 2024-05-23T00:00:22.524 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.516+0000 7fde20b64740 20 client.25348 trim_cache size 0 max 16384 2024-05-23T00:00:22.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.536+0000 7fde20b64740 1 Processor -- start 2024-05-23T00:00:22.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.536+0000 7fde20b64740 1 -- start start 2024-05-23T00:00:22.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.536+0000 7fde20b64740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c74070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.536+0000 7fde20b64740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c83ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.536+0000 7fde20b64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c745b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.536+0000 7fde20b64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6c88d00 con 0x55d7f6c77de0 2024-05-23T00:00:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.536+0000 7fde20b64740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6c893d0 con 0x55d7f6c72280 2024-05-23T00:00:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.536+0000 7fde20b64740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6c89aa0 con 0x55d7f6c72c70 2024-05-23T00:00:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c74070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde0fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c83ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c74070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:59534/0 (socket says 172.21.15.103:59534) 2024-05-23T00:00:22.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 -- 172.21.15.103:0/1668075977 learned_addr learned my addr 172.21.15.103:0/1668075977 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 -- 172.21.15.103:0/1668075977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 msgr2=0x55d7f6c83ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 --2- 172.21.15.103:0/1668075977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c83ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 -- 172.21.15.103:0/1668075977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 msgr2=0x55d7f6c745b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 --2- 172.21.15.103:0/1668075977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c745b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 -- 172.21.15.103:0/1668075977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7f6c7d340 con 0x55d7f6c72280 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde0fe00640 1 --2- 172.21.15.103:0/1668075977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c83ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.537+0000 7fde17400640 1 --2- 172.21.15.103:0/1668075977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c74070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fde10000f80 tx=0x7fde10009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e68d8a7212c61994 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde07400640 1 -- 172.21.15.103:0/1668075977 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1002e030 con 0x55d7f6c72280 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde07400640 1 -- 172.21.15.103:0/1668075977 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde10004030 con 0x55d7f6c72280 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde07400640 1 -- 172.21.15.103:0/1668075977 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde10031040 con 0x55d7f6c72280 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 -- 172.21.15.103:0/1668075977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 msgr2=0x55d7f6c74070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 --2- 172.21.15.103:0/1668075977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c74070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fde10000f80 tx=0x7fde10009500 comp rx=0 tx=0).stop 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 -- 172.21.15.103:0/1668075977 shutdown_connections 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 --2- 172.21.15.103:0/1668075977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c77de0 0x55d7f6c745b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 --2- 172.21.15.103:0/1668075977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72c70 0x55d7f6c83ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 --2- 172.21.15.103:0/1668075977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c72280 0x55d7f6c74070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 -- 172.21.15.103:0/1668075977 >> 172.21.15.103:0/1668075977 conn(0x55d7f6c6d8f0 msgr2=0x55d7f6c6dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 -- 172.21.15.103:0/1668075977 shutdown_connections 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.538+0000 7fde20b64740 1 -- 172.21.15.103:0/1668075977 wait complete. 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 1 -- start start 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 1 Processor -- start 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c7f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 0x55d7f6c829e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6c8a840 con 0x55d7f6c72c70 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6c8a170 con 0x55d7f6c77de0 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55d7f6c8af10 con 0x55d7f6c72280 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.539+0000 7fde20b64740 10 client.0 _ll_register_callbacks cb 0x7fde00009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57352/0 (socket says 172.21.15.103:57352) 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde17400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c7f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 -- 172.21.15.103:0/3153376437 learned_addr learned my addr 172.21.15.103:0/3153376437 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde17400640 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 0x55d7f6c829e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 -- 172.21.15.103:0/3153376437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 msgr2=0x55d7f6c7f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c7f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 -- 172.21.15.103:0/3153376437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 msgr2=0x55d7f6c829e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 0x55d7f6c829e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 -- 172.21.15.103:0/3153376437 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7f6c82fd0 con 0x55d7f6c72c70 2024-05-23T00:00:22.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde0fe00640 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7f9c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fde00005ec0 tx=0x7fde00003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895034a93c240c3e server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.540+0000 7fde04c00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:00:22.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.541+0000 7fde04c00640 1 -- 172.21.15.103:0/3153376437 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde00057070 con 0x55d7f6c72c70 2024-05-23T00:00:22.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.541+0000 7fde04c00640 1 -- 172.21.15.103:0/3153376437 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fde000316b0 con 0x55d7f6c72c70 2024-05-23T00:00:22.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.541+0000 7fde20b64740 1 -- 172.21.15.103:0/3153376437 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde10004030 con 0x55d7f6c72c70 2024-05-23T00:00:22.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.541+0000 7fde04c00640 1 -- 172.21.15.103:0/3153376437 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fde00063050 con 0x55d7f6c72c70 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.541+0000 7fde20b64740 10 client.15246 fetch_fsmap learned FSMap version 24 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.541+0000 7fde20b64740 1 -- 172.21.15.103:0/3153376437 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d7f6cef800 con 0x55d7f6c72c70 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.541+0000 7fde04c00640 1 -- 172.21.15.103:0/3153376437 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde10004030 con 0x55d7f6c72c70 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.542+0000 7fde04c00640 1 -- 172.21.15.103:0/3153376437 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7fde00003de0 con 0x55d7f6c72c70 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.542+0000 7fde20b64740 10 client.15246 fetch_fsmap finished waiting for FSMap version 24 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.542+0000 7fde20b64740 10 client.15246 resolve_mds: resolved name 'a' to daemon mds.a 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.542+0000 7fde20b64740 20 client.15246 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.542+0000 7fde20b64740 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55d7f6cf43b0 0x55d7f6cf30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.542+0000 7fde20b64740 4 client.15246 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.542+0000 7fde20b64740 1 -- 172.21.15.103:0/3153376437 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "status"}) -- 0x55d7f6bddf10 con 0x55d7f6cf43b0 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.542+0000 7fde17400640 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55d7f6cf43b0 0x55d7f6cf30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde17400640 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55d7f6cf43b0 0x55d7f6cf30c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=58b6ba50a62ab003 server_cookie=fa83ffe2f61cf3b8 in_seq=0 out_seq=0 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde04c00640 10 client.15246 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde04c00640 1 -- 172.21.15.103:0/3153376437 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55d7f6bddf10 con 0x55d7f6cf43b0 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde04c00640 10 client.15246 handle_command_reply: tid=0 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 client.15246 shutdown 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 -- 172.21.15.103:0/3153376437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 msgr2=0x55d7f6c7f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7f9c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fde00005ec0 tx=0x7fde00003820 comp rx=0 tx=0).stop 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 -- 172.21.15.103:0/3153376437 shutdown_connections 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55d7f6c77de0 0x55d7f6c829e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d7f6c72c70 0x55d7f6c7f9c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55d7f6cf43b0 0x55d7f6cf30c0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 --2- 172.21.15.103:0/3153376437 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55d7f6c72280 0x55d7f6c7f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 -- 172.21.15.103:0/3153376437 >> 172.21.15.103:0/3153376437 conn(0x55d7f6c6d8f0 msgr2=0x55d7f6c74070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde04c00640 0 client.15246 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-23T00:00:22.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 -- 172.21.15.103:0/3153376437 shutdown_connections 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.543+0000 7fde20b64740 1 -- 172.21.15.103:0/3153376437 wait complete. 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde20b64740 20 client.15246 trim_cache size 0 max 16384 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "whoami": 1, 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 24563, 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "want_state": "up:active", 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "up:active", 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs", 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "rank_uptime": 139.31451024200001, 2024-05-23T00:00:22.552 INFO:teuthology.orchestra.run.smithi103.stdout: "mdsmap_epoch": 24, 2024-05-23T00:00:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch": 93, 2024-05-23T00:00:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch_barrier": 91, 2024-05-23T00:00:22.553 INFO:teuthology.orchestra.run.smithi103.stdout: "uptime": 178.58409762700001 2024-05-23T00:00:22.553 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:00:22.553 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 -- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddfc0a2b90 msgr2=0x7fddfc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddfc0a2b90 0x7fddfc0a5050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fde08004630 tx=0x7fde080d5530 comp rx=0 tx=0).stop 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 -- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde18103ad0 msgr2=0x7fde18199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde18103ad0 0x7fde18199e10 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fde0800e9c0 tx=0x7fde0800edc0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 -- 172.21.15.103:0/2074313257 shutdown_connections 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fddfc0a2b90 0x7fddfc0a5050 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde18105680 0x7fde181a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde18104cc0 0x7fde1819a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 --2- 172.21.15.103:0/2074313257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde18103ad0 0x7fde18199e10 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.544+0000 7fde1de00640 1 -- 172.21.15.103:0/2074313257 >> 172.21.15.103:0/2074313257 conn(0x7fde180ff060 msgr2=0x7fde18107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.545+0000 7fde1de00640 1 -- 172.21.15.103:0/2074313257 shutdown_connections 2024-05-23T00:00:22.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.545+0000 7fde1de00640 1 -- 172.21.15.103:0/2074313257 wait complete. 2024-05-23T00:00:22.563 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:22 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3467025808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:22.563 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:22 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/844296004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:22.565 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-23T00:00:22.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:22 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3467025808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:22.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:22 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/844296004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:22.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.760+0000 7f06c2000640 1 -- 172.21.15.103:0/564255448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06bc109060 msgr2=0x7f06bc1094e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.760+0000 7f06c2000640 1 --2- 172.21.15.103:0/564255448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06bc109060 0x7f06bc1094e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f06a4009900 tx=0x7f06a402a410 comp rx=0 tx=0).stop 2024-05-23T00:00:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.761+0000 7f06c2000640 1 -- 172.21.15.103:0/564255448 shutdown_connections 2024-05-23T00:00:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.761+0000 7f06c2000640 1 --2- 172.21.15.103:0/564255448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06bc109a20 0x7f06bc110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.761+0000 7f06c2000640 1 --2- 172.21.15.103:0/564255448 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06bc109060 0x7f06bc1094e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.761+0000 7f06c2000640 1 --2- 172.21.15.103:0/564255448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06bc107e80 0x7f06bc1082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.761+0000 7f06c2000640 1 -- 172.21.15.103:0/564255448 >> 172.21.15.103:0/564255448 conn(0x7f06bc076e70 msgr2=0x7f06bc0792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.761+0000 7f06c2000640 1 -- 172.21.15.103:0/564255448 shutdown_connections 2024-05-23T00:00:22.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.761+0000 7f06c2000640 1 -- 172.21.15.103:0/564255448 wait complete. 2024-05-23T00:00:22.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06c2000640 1 Processor -- start 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06c2000640 1 -- start start 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06c2000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06bc107e80 0x7f06bc19e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06c2000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06bc109060 0x7f06bc19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06c2000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06bc109a20 0x7f06bc1a5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06c2000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc113ba0 con 0x7f06bc107e80 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06c2000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc113a20 con 0x7f06bc109060 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06c2000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06bc113d20 con 0x7f06bc109a20 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.762+0000 7f06bb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06bc107e80 0x7f06bc19e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06bc107e80 0x7f06bc19e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57372/0 (socket says 172.21.15.103:57372) 2024-05-23T00:00:22.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bb400640 1 -- 172.21.15.103:0/3787030786 learned_addr learned my addr 172.21.15.103:0/3787030786 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06b3e00640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06bc109060 0x7f06bc19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bbe00640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06bc109a20 0x7f06bc1a5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bbe00640 1 -- 172.21.15.103:0/3787030786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06bc109060 msgr2=0x7f06bc19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bbe00640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06bc109060 0x7f06bc19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bbe00640 1 -- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06bc107e80 msgr2=0x7f06bc19e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bbe00640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06bc107e80 0x7f06bc19e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bbe00640 1 -- 172.21.15.103:0/3787030786 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06bc1a5e00 con 0x7f06bc109a20 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bb400640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06bc107e80 0x7f06bc19e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.763+0000 7f06bbe00640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06bc109a20 0x7f06bc1a5790 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f06b400ec50 tx=0x7f06b400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.764+0000 7f06b8c00640 1 -- 172.21.15.103:0/3787030786 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4003b70 con 0x7f06bc109a20 2024-05-23T00:00:22.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.764+0000 7f06c2000640 1 -- 172.21.15.103:0/3787030786 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06bc1a60f0 con 0x7f06bc109a20 2024-05-23T00:00:22.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.764+0000 7f06b8c00640 1 -- 172.21.15.103:0/3787030786 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06b4003d10 con 0x7f06bc109a20 2024-05-23T00:00:22.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.764+0000 7f06b8c00640 1 -- 172.21.15.103:0/3787030786 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06b4010890 con 0x7f06bc109a20 2024-05-23T00:00:22.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.764+0000 7f06c2000640 1 -- 172.21.15.103:0/3787030786 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06bc1a6610 con 0x7f06bc109a20 2024-05-23T00:00:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06c4d6d740 1 Processor -- start 2024-05-23T00:00:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06b8c00640 1 -- 172.21.15.103:0/3787030786 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7f06b4003fc0 con 0x7f06bc109a20 2024-05-23T00:00:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06c4d6d740 1 -- start start 2024-05-23T00:00:22.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 0x55b9a02eb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02f3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b9a02b2db0 con 0x55b9a02eb280 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b9a024f490 con 0x55b9a02f0de0 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.767+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b9a01f7100 con 0x55b9a02ebc70 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06bb400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 0x55b9a02eb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06bb400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:36150/0 (socket says 172.21.15.103:36150) 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06bb400640 1 -- 172.21.15.103:0/4142847003 learned_addr learned my addr 172.21.15.103:0/4142847003 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06bb400640 1 -- 172.21.15.103:0/4142847003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 msgr2=0x55b9a02f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06b3e00640 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06bb400640 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06b8c00640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06a00a2c10 0x7f06a00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06bb400640 1 -- 172.21.15.103:0/4142847003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 msgr2=0x55b9a02eb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06bb400640 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 0x55b9a02eb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06bb400640 1 -- 172.21.15.103:0/4142847003 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b99ff632d0 con 0x55b9a02ebc70 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06b3e00640 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.768+0000 7f06b8c00640 1 -- 172.21.15.103:0/3787030786 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f06b40cf980 con 0x7f06bc109a20 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.769+0000 7f06bbe00640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06a00a2c10 0x7f06a00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.769+0000 7f06bb400640 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f0770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0115c0 tx=0x7f06ac011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b48a373eea3bfb97 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.769+0000 7f06abe00640 1 -- 172.21.15.103:0/4142847003 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac005020 con 0x55b9a02ebc70 2024-05-23T00:00:22.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.769+0000 7f06abe00640 1 -- 172.21.15.103:0/4142847003 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06ac0047a0 con 0x55b9a02ebc70 2024-05-23T00:00:22.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.769+0000 7f06abe00640 1 -- 172.21.15.103:0/4142847003 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac00fa00 con 0x55b9a02ebc70 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 -- 172.21.15.103:0/4142847003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 msgr2=0x55b9a02f0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06bbe00640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06a00a2c10 0x7f06a00a50d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f06b4098280 tx=0x7f06b400ce70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f0770 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0115c0 tx=0x7f06ac011a80 comp rx=0 tx=0).stop 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 -- 172.21.15.103:0/4142847003 shutdown_connections 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f0770 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/4142847003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 0x55b9a02eb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 -- 172.21.15.103:0/4142847003 >> 172.21.15.103:0/4142847003 conn(0x55b9a02e68f0 msgr2=0x55b9a02e6cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 -- 172.21.15.103:0/4142847003 shutdown_connections 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 -- 172.21.15.103:0/4142847003 wait complete. 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 -- start start 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.770+0000 7f06c4d6d740 1 Processor -- start 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 0x55b9a02f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b9a020a520 con 0x55b9a02eb280 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b9a024b450 con 0x55b9a02f0de0 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b9a02f3d90 con 0x55b9a02ebc70 2024-05-23T00:00:22.781 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06c4d6d740 10 client.0 _ll_register_callbacks cb 0x55b99ff62ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06bb400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 0x55b9a02f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06bb400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:36154/0 (socket says 172.21.15.103:36154) 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.771+0000 7f06bb400640 1 -- 172.21.15.103:0/2614143737 learned_addr learned my addr 172.21.15.103:0/2614143737 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06b3e00640 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06bb400640 1 -- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 msgr2=0x55b9a02fca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06bb400640 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06bb400640 1 -- 172.21.15.103:0/2614143737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 msgr2=0x55b9a02f54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06bb400640 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 0x55b9a02f54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06bb400640 1 -- 172.21.15.103:0/2614143737 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9a002d6b0 con 0x55b9a02ebc70 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06b3e00640 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06bb400640 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f59e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0058c0 tx=0x7f06ac005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e89319f380d0db96 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06a8c00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06a8c00640 1 -- 172.21.15.103:0/2614143737 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac028020 con 0x55b9a02ebc70 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2614143737 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06ac0047a0 con 0x55b9a02ebc70 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06a8c00640 1 -- 172.21.15.103:0/2614143737 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f06ac0053a0 con 0x55b9a02ebc70 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.772+0000 7f06a8c00640 1 -- 172.21.15.103:0/2614143737 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f06ac033050 con 0x55b9a02ebc70 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06c4d6d740 10 client.25366 fetch_fsmap learned FSMap version 24 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2614143737 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b9a02ffde0 con 0x55b9a02ebc70 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06a8c00640 1 -- 172.21.15.103:0/2614143737 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06ac0047a0 con 0x55b9a02ebc70 2024-05-23T00:00:22.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06a8c00640 1 -- 172.21.15.103:0/2614143737 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f06ac023630 con 0x55b9a02ebc70 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06c4d6d740 10 client.25366 fetch_fsmap finished waiting for FSMap version 24 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06c4d6d740 10 client.25366 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06c4d6d740 20 client.25366 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55b9a0302230 0x55b9a03036a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06c4d6d740 4 client.25366 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.773+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2614143737 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9a025cbc0 con 0x55b9a0302230 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.774+0000 7f06b3e00640 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55b9a0302230 0x55b9a03036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.774+0000 7f06b3e00640 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55b9a0302230 0x55b9a03036a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e88ae4d9cb453409 server_cookie=41ca4c19896c55f4 in_seq=0 out_seq=0 2024-05-23T00:00:22.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.774+0000 7f06a8c00640 10 client.25366 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06a8c00640 1 -- 172.21.15.103:0/2614143737 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9a025cbc0 con 0x55b9a0302230 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06a8c00640 10 client.25366 handle_command_reply: tid=0 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 client.25366 shutdown 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2614143737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 msgr2=0x55b9a02f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f59e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0058c0 tx=0x7f06ac005230 comp rx=0 tx=0).stop 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2614143737 shutdown_connections 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55b9a0302230 0x55b9a03036a0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02ebc70 0x55b9a02f59e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2614143737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02eb280 0x55b9a02f54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2614143737 >> 172.21.15.103:0/2614143737 conn(0x55b9a02e68f0 msgr2=0x55b9a02e5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06a8c00640 0 client.25366 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2614143737 shutdown_connections 2024-05-23T00:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2614143737 wait complete. 2024-05-23T00:00:22.786 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.775+0000 7f06c4d6d740 20 client.25366 trim_cache size 0 max 16384 2024-05-23T00:00:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.795+0000 7f06c4d6d740 1 Processor -- start 2024-05-23T00:00:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06c4d6d740 1 -- start start 2024-05-23T00:00:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02f5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02ebc70 0x55b9a02fc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02e6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b9a03019e0 con 0x55b9a02ebc70 2024-05-23T00:00:22.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b9a03020b0 con 0x55b9a02f0de0 2024-05-23T00:00:22.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b9a0302780 con 0x55b9a02eb280 2024-05-23T00:00:22.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02f5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06bb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02ebc70 0x55b9a02fc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02f5940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:36168/0 (socket says 172.21.15.103:36168) 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06bb400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02ebc70 0x55b9a02fc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57394/0 (socket says 172.21.15.103:57394) 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06b3e00640 1 -- 172.21.15.103:0/699936597 learned_addr learned my addr 172.21.15.103:0/699936597 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.796+0000 7f06b3e00640 1 --2- 172.21.15.103:0/699936597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02e6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06bb400640 1 -- 172.21.15.103:0/699936597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 msgr2=0x55b9a02f5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06bb400640 1 --2- 172.21.15.103:0/699936597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02f5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06bb400640 1 -- 172.21.15.103:0/699936597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 msgr2=0x55b9a02e6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06bb400640 1 --2- 172.21.15.103:0/699936597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02e6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06bb400640 1 -- 172.21.15.103:0/699936597 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9a02ec050 con 0x55b9a02ebc70 2024-05-23T00:00:22.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06bb400640 1 --2- 172.21.15.103:0/699936597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02ebc70 0x55b9a02fc9e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f06ac023bc0 tx=0x7f06ac009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ccf69de77432b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06ab400640 1 -- 172.21.15.103:0/699936597 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac028070 con 0x55b9a02ebc70 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06ab400640 1 -- 172.21.15.103:0/699936597 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06ac016030 con 0x55b9a02ebc70 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.797+0000 7f06ab400640 1 -- 172.21.15.103:0/699936597 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac01d040 con 0x55b9a02ebc70 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 -- 172.21.15.103:0/699936597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02ebc70 msgr2=0x55b9a02fc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/699936597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02ebc70 0x55b9a02fc9e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f06ac023bc0 tx=0x7f06ac009c10 comp rx=0 tx=0).stop 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 -- 172.21.15.103:0/699936597 shutdown_connections 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/699936597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02e6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/699936597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02ebc70 0x55b9a02fc9e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/699936597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02f5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 -- 172.21.15.103:0/699936597 >> 172.21.15.103:0/699936597 conn(0x55b9a02e68f0 msgr2=0x55b9a0301240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 -- 172.21.15.103:0/699936597 shutdown_connections 2024-05-23T00:00:22.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 -- 172.21.15.103:0/699936597 wait complete. 2024-05-23T00:00:22.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.798+0000 7f06c4d6d740 1 -- start start 2024-05-23T00:00:22.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06c4d6d740 1 Processor -- start 2024-05-23T00:00:22.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02fab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06c4d6d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02fb650 0x55b9a02fbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b9a0303520 con 0x55b9a02fb650 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55b9a0302e50 con 0x55b9a02f0de0 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06c4d6d740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b9a0303bf0 con 0x55b9a02eb280 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06c4d6d740 10 client.0 _ll_register_callbacks cb 0x7f06b40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02fab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06bb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02fb650 0x55b9a02fbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06b3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02fb650 0x55b9a02fbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:57398/0 (socket says 172.21.15.103:57398) 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06bb400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:59614/0 (socket says 172.21.15.103:59614) 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.799+0000 7f06b3e00640 1 -- 172.21.15.103:0/2821376091 learned_addr learned my addr 172.21.15.103:0/2821376091 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:22.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06bb400640 1 -- 172.21.15.103:0/2821376091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 msgr2=0x55b9a02fab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06bb400640 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02fab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06bb400640 1 -- 172.21.15.103:0/2821376091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02fb650 msgr2=0x55b9a02fbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06bb400640 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02fb650 0x55b9a02fbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06bb400640 1 -- 172.21.15.103:0/2821376091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9a02fc2e0 con 0x55b9a02f0de0 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06b3e00640 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02fb650 0x55b9a02fbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06bb400640 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fb0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55b9a02fddd0 tx=0x7f06ac009e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f257912b5c9d03cb server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06a8c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06a8c00640 1 -- 172.21.15.103:0/2821376091 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06ac01d070 con 0x55b9a02f0de0 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2821376091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06ac016030 con 0x55b9a02f0de0 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.800+0000 7f06a8c00640 1 -- 172.21.15.103:0/2821376091 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f06ac004d10 con 0x55b9a02f0de0 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06a8c00640 1 -- 172.21.15.103:0/2821376091 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f06ac028070 con 0x55b9a02f0de0 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06c4d6d740 10 client.25358 fetch_fsmap learned FSMap version 24 2024-05-23T00:00:22.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2821376091 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55b9a0368880 con 0x55b9a02f0de0 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06a8c00640 1 -- 172.21.15.103:0/2821376091 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f06ac016030 con 0x55b9a02f0de0 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06a8c00640 1 -- 172.21.15.103:0/2821376091 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f06ac00f660 con 0x55b9a02f0de0 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06c4d6d740 10 client.25358 fetch_fsmap finished waiting for FSMap version 24 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06c4d6d740 10 client.25358 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06c4d6d740 20 client.25358 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55b9a036d050 0x55b9a036bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06c4d6d740 4 client.25358 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.801+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2821376091 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "status"}) -- 0x55b9a0256f10 con 0x55b9a036d050 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06b3e00640 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55b9a036d050 0x55b9a036bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06b3e00640 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55b9a036d050 0x55b9a036bd60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2383d4faba1867a3 server_cookie=98fb9e6ffad184d7 in_seq=0 out_seq=0 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06a8c00640 10 client.25358 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:22.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06a8c00640 1 -- 172.21.15.103:0/2821376091 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b9a0256f10 con 0x55b9a036d050 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06a8c00640 10 client.25358 handle_command_reply: tid=0 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 client.25358 shutdown 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2821376091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 msgr2=0x55b9a02fb0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fb0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55b9a02fddd0 tx=0x7f06ac009e20 comp rx=0 tx=0).stop 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2821376091 shutdown_connections 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55b9a036d050 0x55b9a036bd60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9a02fb650 0x55b9a02fbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55b9a02f0de0 0x55b9a02fb0c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 --2- 172.21.15.103:0/2821376091 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b9a02eb280 0x55b9a02fab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06a8c00640 0 client.25358 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.802+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2821376091 >> 172.21.15.103:0/2821376091 conn(0x55b9a02e68f0 msgr2=0x55b9a02e6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2821376091 shutdown_connections 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c4d6d740 1 -- 172.21.15.103:0/2821376091 wait complete. 2024-05-23T00:00:22.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c4d6d740 20 client.25358 trim_cache size 0 max 16384 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout: "cluster_fsid": "9c0f3dc8-1895-11ef-bc9a-c7b262605968", 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout: "whoami": 0, 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 24575, 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout: "want_state": "up:active", 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout: "state": "up:active", 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout: "fs_name": "cephfs", 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout: "rank_uptime": 169.09242251000001, 2024-05-23T00:00:22.812 INFO:teuthology.orchestra.run.smithi103.stdout: "mdsmap_epoch": 24, 2024-05-23T00:00:22.813 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch": 93, 2024-05-23T00:00:22.813 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_epoch_barrier": 91, 2024-05-23T00:00:22.813 INFO:teuthology.orchestra.run.smithi103.stdout: "uptime": 175.86082927300001 2024-05-23T00:00:22.813 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:00:22.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 -- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06a00a2c10 msgr2=0x7f06a00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06a00a2c10 0x7f06a00a50d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f06b4098280 tx=0x7f06b400ce70 comp rx=0 tx=0).stop 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 -- 172.21.15.103:0/3787030786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06bc109a20 msgr2=0x7f06bc1a5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06bc109a20 0x7f06bc1a5790 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f06b400ec50 tx=0x7f06b400c6a0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 -- 172.21.15.103:0/3787030786 shutdown_connections 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f06a00a2c10 0x7f06a00a50d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f06bc109a20 0x7f06bc1a5790 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f06bc109060 0x7f06bc19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 --2- 172.21.15.103:0/3787030786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f06bc107e80 0x7f06bc19e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.803+0000 7f06c2000640 1 -- 172.21.15.103:0/3787030786 >> 172.21.15.103:0/3787030786 conn(0x7f06bc076e70 msgr2=0x7f06bc10c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.804+0000 7f06c2000640 1 -- 172.21.15.103:0/3787030786 shutdown_connections 2024-05-23T00:00:22.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:22.804+0000 7f06c2000640 1 -- 172.21.15.103:0/3787030786 wait complete. 2024-05-23T00:00:22.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:22 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3467025808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:22.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:22 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/844296004' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:22.824 INFO:teuthology.run_tasks:Running task sequential... 2024-05-23T00:00:22.834 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-23T00:00:22.834 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-23T00:00:22.834 INFO:teuthology.task.print:Enabling mgr modules 2024-05-23T00:00:22.834 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-05-23T00:00:22.835 INFO:teuthology.task.exec:Executing custom commands... 2024-05-23T00:00:22.835 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:00:22.836 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-05-23T00:00:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 -- 172.21.15.103:0/4235890464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd2c102900 msgr2=0x7fdd2c102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 --2- 172.21.15.103:0/4235890464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd2c102900 0x7fdd2c102d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdd18009900 tx=0x7fdd1802a410 comp rx=0 tx=0).stop 2024-05-23T00:00:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 -- 172.21.15.103:0/4235890464 shutdown_connections 2024-05-23T00:00:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 --2- 172.21.15.103:0/4235890464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd2c1032c0 0x7fdd2c109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 --2- 172.21.15.103:0/4235890464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd2c102900 0x7fdd2c102d80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 --2- 172.21.15.103:0/4235890464 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd2c101710 0x7fdd2c101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 -- 172.21.15.103:0/4235890464 >> 172.21.15.103:0/4235890464 conn(0x7fdd2c0fcc80 msgr2=0x7fdd2c0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:23.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 -- 172.21.15.103:0/4235890464 shutdown_connections 2024-05-23T00:00:23.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.027+0000 7fdd2be00640 1 -- 172.21.15.103:0/4235890464 wait complete. 2024-05-23T00:00:23.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.028+0000 7fdd2be00640 1 Processor -- start 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.028+0000 7fdd2be00640 1 -- start start 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd2c101710 0x7fdd2c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd2c102900 0x7fdd2c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd2c1032c0 0x7fdd2c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c10d370 con 0x7fdd2c1032c0 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c10d1f0 con 0x7fdd2c102900 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2be00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2c10d4f0 con 0x7fdd2c101710 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2aa00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd2c101710 0x7fdd2c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd2c1032c0 0x7fdd2c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2aa00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd2c101710 0x7fdd2c197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:36194/0 (socket says 172.21.15.103:36194) 2024-05-23T00:00:23.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2aa00640 1 -- 172.21.15.103:0/1693819614 learned_addr learned my addr 172.21.15.103:0/1693819614 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2a000640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd2c102900 0x7fdd2c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2aa00640 1 -- 172.21.15.103:0/1693819614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd2c102900 msgr2=0x7fdd2c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2aa00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd2c102900 0x7fdd2c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2aa00640 1 -- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd2c1032c0 msgr2=0x7fdd2c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2aa00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd2c1032c0 0x7fdd2c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.029+0000 7fdd2aa00640 1 -- 172.21.15.103:0/1693819614 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2c19f660 con 0x7fdd2c101710 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.030+0000 7fdd2aa00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd2c101710 0x7fdd2c197a40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00e9c0 tx=0x7fdd1c00edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.030+0000 7fdd23400640 1 -- 172.21.15.103:0/1693819614 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c00cd80 con 0x7fdd2c101710 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.030+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2c19f950 con 0x7fdd2c101710 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.030+0000 7fdd23400640 1 -- 172.21.15.103:0/1693819614 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd1c010910 con 0x7fdd2c101710 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.030+0000 7fdd23400640 1 -- 172.21.15.103:0/1693819614 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c018980 con 0x7fdd2c101710 2024-05-23T00:00:23.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.030+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd2c19fe70 con 0x7fdd2c101710 2024-05-23T00:00:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.031+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd2c108a80 con 0x7fdd2c101710 2024-05-23T00:00:23.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.033+0000 7fdd23400640 1 -- 172.21.15.103:0/1693819614 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 18) ==== 137834+0+0 (secure 0 0 0) 0x7fdd1c01f020 con 0x7fdd2c101710 2024-05-23T00:00:23.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.035+0000 7fdd23400640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdd100a2b80 0x7fdd100a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.035+0000 7fdd23400640 1 -- 172.21.15.103:0/1693819614 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fdd1c0cf4e0 con 0x7fdd2c101710 2024-05-23T00:00:23.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.035+0000 7fdd2a000640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdd100a2b80 0x7fdd100a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:23.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.036+0000 7fdd2a000640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdd100a2b80 0x7fdd100a5040 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd180314e0 tx=0x7fdd18000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:23.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.039+0000 7fdd23400640 1 -- 172.21.15.103:0/1693819614 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdd1c09c080 con 0x7fdd2c101710 2024-05-23T00:00:23.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.292+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7fdd2c062470 con 0x7fdd2c101710 2024-05-23T00:00:23.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.331+0000 7fdd23400640 1 -- 172.21.15.103:0/1693819614 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7fdd1c097ff0 con 0x7fdd2c101710 2024-05-23T00:00:23.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.332+0000 7fdd23400640 1 -- 172.21.15.103:0/1693819614 <== mon.1 v2:172.21.15.174:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7fdd1c0777d0 con 0x7fdd2c101710 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdd100a2b80 msgr2=0x7fdd100a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdd100a2b80 0x7fdd100a5040 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd180314e0 tx=0x7fdd18000bc0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd2c101710 msgr2=0x7fdd2c197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd2c101710 0x7fdd2c197a40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c00e9c0 tx=0x7fdd1c00edc0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 shutdown_connections 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fdd100a2b80 0x7fdd100a5040 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd2c1032c0 0x7fdd2c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd2c102900 0x7fdd2c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 --2- 172.21.15.103:0/1693819614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd2c101710 0x7fdd2c197a40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:23.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 >> 172.21.15.103:0/1693819614 conn(0x7fdd2c0fcc80 msgr2=0x7fdd2c105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:23.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 shutdown_connections 2024-05-23T00:00:23.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:23.335+0000 7fdd2be00640 1 -- 172.21.15.103:0/1693819614 wait complete. 2024-05-23T00:00:23.360 INFO:teuthology.run_tasks:Running task kclient... 2024-05-23T00:00:23.374 INFO:tasks.kclient:Mounting kernel clients... 2024-05-23T00:00:23.374 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-23T00:00:23.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:23.374 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-05-23T00:00:23.419 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-23T00:00:23.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:23.420 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-23T00:00:23.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:23.524 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-05-23T00:00:23.598 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:23 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: ignoring --setuser ceph since I am not root 2024-05-23T00:00:23.598 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:23 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: ignoring --setgroup ceph since I am not root 2024-05-23T00:00:23.598 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:23 smithi103 ceph-mon[31370]: pgmap v409: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:23.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:23 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1693819614' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-23T00:00:23.599 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:23 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-23T00:00:23.610 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-05-23T00:00:23.610 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-23T00:00:23.610 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-23T00:00:23.611 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph fs ls 2024-05-23T00:00:23.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:23 smithi174 ceph-mon[36179]: pgmap v409: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:23.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:23 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1693819614' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-23T00:00:23.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:23 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-23T00:00:23.617 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: ignoring --setuser ceph since I am not root 2024-05-23T00:00:23.618 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: ignoring --setgroup ceph since I am not root 2024-05-23T00:00:23.618 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:23 smithi174 ceph-mgr[38043]: -- 172.21.15.174:0/2423692826 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f9502a5a0 con 0x556f94fff000 2024-05-23T00:00:23.618 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:23.500+0000 7f562f76e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-23T00:00:23.627 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:23 smithi105 ceph-mon[36426]: pgmap v409: 129 pgs: 129 active+clean; 642 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:00:23.627 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:23 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1693819614' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-23T00:00:23.627 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:23 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-05-23T00:00:23.627 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: ignoring --setuser ceph since I am not root 2024-05-23T00:00:23.628 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: ignoring --setgroup ceph since I am not root 2024-05-23T00:00:23.628 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:23 smithi105 ceph-mgr[37806]: -- 172.21.15.105:0/667320114 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bb6b45e5a0 con 0x55bb6b432800 2024-05-23T00:00:23.628 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:23.504+0000 7fe31ea30200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-23T00:00:23.907 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:23 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:23.648+0000 7fd8df906200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-23T00:00:23.907 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:23 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:23.903+0000 7fd8df906200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-23T00:00:24.043 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:00:24.058 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:23.695+0000 7fe31ea30200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-23T00:00:24.058 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:23.798+0000 7fe31ea30200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-23T00:00:24.097 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:23.684+0000 7f562f76e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-23T00:00:24.097 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:23.787+0000 7f562f76e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-23T00:00:24.098 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:24.094+0000 7f562f76e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-23T00:00:24.189 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:24.045+0000 7fd8df906200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-23T00:00:24.339 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:24 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:24.104+0000 7fe31ea30200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-23T00:00:24.339 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:24 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:24.189+0000 7fe31ea30200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-23T00:00:24.367 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:24.179+0000 7f562f76e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-23T00:00:24.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:24 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-23T00:00:24.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:24 smithi174 ceph-mon[36179]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-23T00:00:24.448 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:24 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-23T00:00:24.449 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:24 smithi103 ceph-mon[31370]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-23T00:00:24.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:24 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-05-23T00:00:24.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:24 smithi105 ceph-mon[36426]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-05-23T00:00:24.829 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:24.454+0000 7fd8df906200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-23T00:00:24.830 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:24.571+0000 7fd8df906200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-23T00:00:25.617 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:25.353+0000 7f562f76e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-23T00:00:25.617 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:25.520+0000 7f562f76e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-23T00:00:25.737 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:25 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:25.379+0000 7fe31ea30200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-23T00:00:25.737 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:25 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:25.558+0000 7fe31ea30200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-23T00:00:25.982 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:25.691+0000 7f562f76e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-23T00:00:25.994 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:25.979+0000 7f562f76e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-23T00:00:26.033 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:25 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:25.734+0000 7fe31ea30200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.241+0000 7f8642c00700 1 -- 172.21.15.103:0/684326404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c1076d0 msgr2=0x7f863c10f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.241+0000 7f8642c00700 1 --2- 172.21.15.103:0/684326404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c1076d0 0x7f863c10f2a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f863400b0a0 tx=0x7f863400b3b0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.242+0000 7f8642c00700 1 -- 172.21.15.103:0/684326404 shutdown_connections 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.242+0000 7f8642c00700 1 --2- 172.21.15.103:0/684326404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c1076d0 0x7f863c10f2a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.242+0000 7f8642c00700 1 --2- 172.21.15.103:0/684326404 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c102610 0x7f863c1029f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.242+0000 7f8642c00700 1 --2- 172.21.15.103:0/684326404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f863c102fc0 0x7f863c106ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.242+0000 7f8642c00700 1 -- 172.21.15.103:0/684326404 >> 172.21.15.103:0/684326404 conn(0x7f863c079510 msgr2=0x7f863c079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.242+0000 7f8642c00700 1 -- 172.21.15.103:0/684326404 shutdown_connections 2024-05-23T00:00:26.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.242+0000 7f8642c00700 1 -- 172.21.15.103:0/684326404 wait complete. 2024-05-23T00:00:26.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.243+0000 7f8642c00700 1 Processor -- start 2024-05-23T00:00:26.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.243+0000 7f8642c00700 1 -- start start 2024-05-23T00:00:26.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.243+0000 7f8642c00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c102610 0x7f863c1a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:26.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.243+0000 7f8642c00700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c102fc0 0x7f863c1a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8642c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f863c1076d0 0x7f863c19ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8642c00700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f863c1126b0 con 0x7f863c102fc0 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8642c00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f863c112530 con 0x7f863c102610 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8642c00700 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f863c112830 con 0x7f863c1076d0 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f863be00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c102610 0x7f863c1a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8640c00700 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f863c1076d0 0x7f863c19ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f863b400700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c102fc0 0x7f863c1a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f863be00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c102610 0x7f863c1a0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48638/0 (socket says 172.21.15.103:48638) 2024-05-23T00:00:26.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f863be00700 1 -- 172.21.15.103:0/2583340943 learned_addr learned my addr 172.21.15.103:0/2583340943 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:26.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8640c00700 1 -- 172.21.15.103:0/2583340943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c102610 msgr2=0x7f863c1a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:26.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8640c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c102610 0x7f863c1a0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8640c00700 1 -- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c102fc0 msgr2=0x7f863c1a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:26.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8640c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c102fc0 0x7f863c1a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.244+0000 7f8640c00700 1 -- 172.21.15.103:0/2583340943 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863c19b280 con 0x7f863c1076d0 2024-05-23T00:00:26.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.245+0000 7f8640c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f863c1076d0 0x7f863c19ab20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f863400b0a0 tx=0x7f8634004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:26.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.245+0000 7f8638c00700 1 -- 172.21.15.103:0/2583340943 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634021070 con 0x7f863c1076d0 2024-05-23T00:00:26.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.245+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f863c19b510 con 0x7f863c1076d0 2024-05-23T00:00:26.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.245+0000 7f8638c00700 1 -- 172.21.15.103:0/2583340943 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8634007b80 con 0x7f863c1076d0 2024-05-23T00:00:26.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.245+0000 7f8638c00700 1 -- 172.21.15.103:0/2583340943 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634005040 con 0x7f863c1076d0 2024-05-23T00:00:26.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.245+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f863c1a77e0 con 0x7f863c1076d0 2024-05-23T00:00:26.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.246+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f863c069050 con 0x7f863c1076d0 2024-05-23T00:00:26.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.250+0000 7f8638c00700 1 -- 172.21.15.103:0/2583340943 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 19) ==== 137851+0+0 (secure 0 0 0) 0x7f8634024070 con 0x7f863c1076d0 2024-05-23T00:00:26.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.252+0000 7f8638c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f86200a2c70 0x7f86200a5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:26.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.252+0000 7f8638c00700 1 -- 172.21.15.103:0/2583340943 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f863400f020 con 0x7f863c1076d0 2024-05-23T00:00:26.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.252+0000 7f863be00700 1 -- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f86200a2c70 msgr2=0x7f86200a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:26.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.252+0000 7f863be00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f86200a2c70 0x7f86200a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-23T00:00:26.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.256+0000 7f8638c00700 1 -- 172.21.15.103:0/2583340943 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f863402b020 con 0x7f863c1076d0 2024-05-23T00:00:26.282 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:26.071+0000 7f562f76e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-23T00:00:26.283 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:26.280+0000 7f562f76e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-23T00:00:26.303 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:26 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:26.030+0000 7fe31ea30200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-23T00:00:26.303 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:26 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:26.124+0000 7fe31ea30200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-23T00:00:26.418 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:26.172+0000 7fd8df906200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-23T00:00:26.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.453+0000 7f863be00700 1 -- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f86200a2c70 msgr2=0x7f86200a5130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:26.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.453+0000 7f863be00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f86200a2c70 0x7f86200a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-23T00:00:26.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.532+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f863c19bfc0 con 0x7f863c1076d0 2024-05-23T00:00:26.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.533+0000 7f8638c00700 1 -- 172.21.15.103:0/2583340943 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f8634007d30 con 0x7f863c1076d0 2024-05-23T00:00:26.536 INFO:teuthology.orchestra.run.smithi103.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-23T00:00:26.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.536+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f86200a2c70 msgr2=0x7f86200a5130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-23T00:00:26.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.536+0000 7f8642c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f86200a2c70 0x7f86200a5130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.536+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f863c1076d0 msgr2=0x7f863c19ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:26.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.536+0000 7f8642c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f863c1076d0 0x7f863c19ab20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f863400b0a0 tx=0x7f8634004d00 comp rx=0 tx=0).stop 2024-05-23T00:00:26.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.537+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 shutdown_connections 2024-05-23T00:00:26.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.537+0000 7f8642c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f863c102610 0x7f863c1a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.537+0000 7f8642c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f86200a2c70 0x7f86200a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.537+0000 7f8642c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f863c1076d0 0x7f863c19ab20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.537+0000 7f8642c00700 1 --2- 172.21.15.103:0/2583340943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f863c102fc0 0x7f863c1a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:26.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.537+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 >> 172.21.15.103:0/2583340943 conn(0x7f863c079510 msgr2=0x7f863c100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:26.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.537+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 shutdown_connections 2024-05-23T00:00:26.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:26.537+0000 7f8642c00700 1 -- 172.21.15.103:0/2583340943 wait complete. 2024-05-23T00:00:26.558 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:26 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:26.333+0000 7fe31ea30200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-23T00:00:26.558 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:26 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:26.525+0000 7fe31ea30200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-23T00:00:26.589 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:26.473+0000 7f562f76e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-23T00:00:26.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:26 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2583340943' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-23T00:00:26.829 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:26.414+0000 7fd8df906200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-23T00:00:26.830 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:26.641+0000 7fd8df906200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-23T00:00:26.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:26 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2583340943' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-23T00:00:27.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:26 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2583340943' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-23T00:00:27.207 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:27.034+0000 7fd8df906200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-23T00:00:27.207 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:27.160+0000 7fd8df906200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-23T00:00:27.367 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:27.073+0000 7f562f76e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi103.front.sepia.ceph.com 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-23T00:00:27.458 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-23T00:00:27.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:27.459 DEBUG:teuthology.orchestra.run.smithi103:> ip addr 2024-05-23T00:00:27.504 INFO:teuthology.orchestra.run.smithi103.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: inet 127.0.0.1/8 scope host lo 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 ::1/128 scope host 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c4 brd ff:ff:ff:ff:ff:ff 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f0 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:d9:b5:c5 brd ff:ff:ff:ff:ff:ff 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: altname enp7s0f1 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-23T00:00:27.505 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0a brd ff:ff:ff:ff:ff:ff 2024-05-23T00:00:27.506 INFO:teuthology.orchestra.run.smithi103.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-23T00:00:27.506 INFO:teuthology.orchestra.run.smithi103.stdout: link/ether 0c:c4:7a:8f:cc:0b brd ff:ff:ff:ff:ff:ff 2024-05-23T00:00:27.506 INFO:teuthology.orchestra.run.smithi103.stdout: inet 172.21.15.103/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-05-23T00:00:27.506 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft 41896sec preferred_lft 41896sec 2024-05-23T00:00:27.506 INFO:teuthology.orchestra.run.smithi103.stdout: inet6 fe80::ec4:7aff:fe8f:cc0b/64 scope link 2024-05-23T00:00:27.506 INFO:teuthology.orchestra.run.smithi103.stdout: valid_lft forever preferred_lft forever 2024-05-23T00:00:27.506 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-23T00:00:27.506 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-23T00:00:27.506 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-23T00:00:27.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add name ceph-brx type bridge 2024-05-23T00:00:27.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr flush dev ceph-brx 2024-05-23T00:00:27.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set ceph-brx up 2024-05-23T00:00:27.507 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-23T00:00:27.507 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-23T00:00:27.550 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:27 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:27.125+0000 7fe31ea30200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-23T00:00:27.579 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:27.424+0000 7fd8df906200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-23T00:00:27.645 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-23T00:00:27.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:27.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:27.773 DEBUG:teuthology.orchestra.run.smithi103:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-23T00:00:27.786 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:27.489+0000 7f562f76e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-23T00:00:27.786 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:27.585+0000 7f562f76e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-23T00:00:27.786 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:27.783+0000 7f562f76e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-23T00:00:27.808 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:27 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:27.548+0000 7fe31ea30200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-23T00:00:27.808 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:27 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:27.645+0000 7fe31ea30200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-23T00:00:27.878 INFO:teuthology.orchestra.run.smithi103.stdout:1 2024-05-23T00:00:27.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:27.882 DEBUG:teuthology.orchestra.run.smithi103:> route 2024-05-23T00:00:27.923 INFO:teuthology.orchestra.run.smithi103.stdout:Kernel IP routing table 2024-05-23T00:00:27.923 INFO:teuthology.orchestra.run.smithi103.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-23T00:00:27.923 INFO:teuthology.orchestra.run.smithi103.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-23T00:00:27.923 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-23T00:00:27.923 INFO:teuthology.orchestra.run.smithi103.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-23T00:00:27.924 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-23T00:00:27.924 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-23T00:00:27.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-23T00:00:27.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-23T00:00:27.924 DEBUG:teuthology.orchestra.run.smithi103:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-23T00:00:27.924 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-23T00:00:28.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-23T00:00:28.079 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:27.694+0000 7fd8df906200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-23T00:00:28.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:28.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:28.157 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-05-23T00:00:28.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:28.197 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list-id 2024-05-23T00:00:28.290 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-23T00:00:28.290 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-23T00:00:28.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-23T00:00:28.290 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-23T00:00:28.290 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-23T00:00:28.308 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:27 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:27.844+0000 7fe31ea30200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-23T00:00:28.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-23T00:00:28.456 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:28.465 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-23T00:00:28.466 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-23T00:00:28.466 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-23T00:00:28.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-23T00:00:28.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-23T00:00:28.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-23T00:00:28.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-23T00:00:28.466 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-23T00:00:28.466 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-23T00:00:28.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-23T00:00:28.617 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:28.305+0000 7f562f76e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-23T00:00:28.617 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:28.399+0000 7f562f76e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-23T00:00:28.672 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:28.527+0000 7fd8df906200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-23T00:00:28.698 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:28 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:28.363+0000 7fe31ea30200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-23T00:00:28.698 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:28 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:28.454+0000 7fe31ea30200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-23T00:00:28.714 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:28.723 DEBUG:teuthology.orchestra.run.smithi103:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-23T00:00:28.723 DEBUG:teuthology.orchestra.run.smithi103:> set -e 2024-05-23T00:00:28.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set brx.0 up 2024-05-23T00:00:28.723 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link set dev brx.0 master ceph-brx 2024-05-23T00:00:28.723 DEBUG:teuthology.orchestra.run.smithi103:> ') 2024-05-23T00:00:28.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-23T00:00:28.901 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:28.910 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-23T00:00:28.910 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-23T00:00:28.948 INFO:teuthology.orchestra.run.smithi103.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-23T00:00:28.949 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-23T00:00:28.949 DEBUG:teuthology.orchestra.run.smithi103:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-23T00:00:28.983 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:28.640+0000 7f562f76e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-23T00:00:28.983 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:28.770+0000 7f562f76e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-23T00:00:28.983 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:28.864+0000 7f562f76e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-23T00:00:28.983 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:28.980+0000 7f562f76e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-23T00:00:29.033 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-23T00:00:29.034 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-23T00:00:29.034 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4 2024-05-23T00:00:29.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:29.034 DEBUG:teuthology.orchestra.run.smithi103:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-05-23T00:00:29.042 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:28 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:28.694+0000 7fe31ea30200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-23T00:00:29.042 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:28 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:28.828+0000 7fe31ea30200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-23T00:00:29.043 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:28 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:28.924+0000 7fe31ea30200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-23T00:00:29.043 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:29 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:29.038+0000 7fe31ea30200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-23T00:00:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-05-23T00:00:29.185 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-05-23T00:00:29.441 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:29.113+0000 7fd8df906200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-23T00:00:29.441 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:29.246+0000 7fd8df906200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-23T00:00:29.617 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:29 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:29.281+0000 7f562f76e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-23T00:00:29.808 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:29 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:29.339+0000 7fe31ea30200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-23T00:00:29.829 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:29.526+0000 7fd8df906200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-23T00:00:29.916 INFO:teuthology.orchestra.run.smithi103.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-05-23T00:00:29.917 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-05-23T00:00:29.917 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: resolved to: "172.21.15.103:3300,172.21.15.105:3300,172.21.15.174:3300" 2024-05-23T00:00:29.917 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: trying mount with new device syntax: 0@9c0f3dc8-1895-11ef-bc9a-c7b262605968.cephfs=/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4 2024-05-23T00:00:29.917 INFO:teuthology.orchestra.run.smithi103.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.103:3300/172.21.15.105:3300/172.21.15.174:3300" will pass to kernel 2024-05-23T00:00:29.919 INFO:teuthology.orchestra.run.smithi103.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-23T00:00:29.919 INFO:teuthology.orchestra.run.smithi103.stdout: You just mounted a file system that supports labels which does not 2024-05-23T00:00:29.920 INFO:teuthology.orchestra.run.smithi103.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-23T00:00:29.920 INFO:teuthology.orchestra.run.smithi103.stdout: applications will generate AVC messages and not be allowed access to 2024-05-23T00:00:29.920 INFO:teuthology.orchestra.run.smithi103.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-23T00:00:29.923 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-23T00:00:29.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:29.923 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-23T00:00:29.991 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-23T00:00:30.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-23T00:00:30.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:30.103 INFO:teuthology.orchestra.run.smithi103.stdout:client25399 2024-05-23T00:00:30.103 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-23T00:00:30.104 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-23T00:00:30.151 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-23T00:00:30.163 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:29 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:29.879+0000 7f562f76e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-23T00:00:30.163 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:00:30 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z[38039]: 2024-05-23T00:00:30.093+0000 7f562f76e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-23T00:00:30.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-23T00:00:30.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:30.304 INFO:teuthology.orchestra.run.smithi103.stdout:client25399 2024-05-23T00:00:30.304 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-23T00:00:30.308 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:29 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:29.936+0000 7fe31ea30200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-23T00:00:30.308 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:00:30 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y[37802]: 2024-05-23T00:00:30.154+0000 7fe31ea30200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-23T00:00:30.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:30 smithi105 ceph-mon[36426]: Standby manager daemon z restarted 2024-05-23T00:00:30.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:30 smithi105 ceph-mon[36426]: Standby manager daemon z started 2024-05-23T00:00:30.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:30 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-23T00:00:30.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:30 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-23T00:00:30.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:30 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-23T00:00:30.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:30 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-23T00:00:30.401 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-23T00:00:30.405 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:30.414 INFO:teuthology.orchestra.run.smithi103.stdout:client25399 2024-05-23T00:00:30.414 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968.client25399/status') 2024-05-23T00:00:30.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-23T00:00:30.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:30 smithi103 ceph-mon[31370]: Standby manager daemon z restarted 2024-05-23T00:00:30.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:30 smithi103 ceph-mon[31370]: Standby manager daemon z started 2024-05-23T00:00:30.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:30 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-23T00:00:30.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:30 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-23T00:00:30.531 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:30 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-23T00:00:30.532 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:30 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-23T00:00:30.532 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:30 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:30.207+0000 7fd8df906200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-23T00:00:30.532 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:30 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:30.336+0000 7fd8df906200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-23T00:00:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:instance: client.25399 (3)172.21.15.103:0/2939815310 2024-05-23T00:00:30.539 INFO:teuthology.orchestra.run.smithi103.stdout:blocklisted: false 2024-05-23T00:00:30.539 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records in 2024-05-23T00:00:30.539 INFO:teuthology.orchestra.run.smithi103.stderr:0+1 records out 2024-05-23T00:00:30.539 INFO:teuthology.orchestra.run.smithi103.stderr:72 bytes copied, 7.4372e-05 s, 968 kB/s 2024-05-23T00:00:30.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-23T00:00:30.551 INFO:teuthology.run_tasks:Running task exec... 2024-05-23T00:00:30.571 INFO:teuthology.task.exec:Executing custom commands... 2024-05-23T00:00:30.572 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:00:30.572 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-05-23T00:00:30.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:30 smithi174 ceph-mon[36179]: Standby manager daemon z restarted 2024-05-23T00:00:30.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:30 smithi174 ceph-mon[36179]: Standby manager daemon z started 2024-05-23T00:00:30.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:30 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-23T00:00:30.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:30 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-23T00:00:30.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:30 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-23T00:00:30.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:30 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.174:0/912046276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1440635020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba01032b0 msgr2=0x7f8ba0109b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1440635020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba01032b0 0x7f8ba0109b40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009ca0 tx=0x7f8b9802a8c0 comp rx=0 tx=0).stop 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1440635020 shutdown_connections 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1440635020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba01032b0 0x7f8ba0109b40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1440635020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba01028f0 0x7f8ba0102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1440635020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba0101700 0x7f8ba0101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1440635020 >> 172.21.15.103:0/1440635020 conn(0x7f8ba00fcc70 msgr2=0x7f8ba00ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1440635020 shutdown_connections 2024-05-23T00:00:30.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.752+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1440635020 wait complete. 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.753+0000 7f8b9fe00640 1 Processor -- start 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.753+0000 7f8b9fe00640 1 -- start start 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba0101700 0x7f8ba00755d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba01028f0 0x7f8ba0073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba01032b0 0x7f8ba00741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba010d3d0 con 0x7f8ba01032b0 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba010d250 con 0x7f8ba01028f0 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9fe00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba010d550 con 0x7f8ba0101700 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba01032b0 0x7f8ba00741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:30.758 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba01032b0 0x7f8ba00741a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40174/0 (socket says 172.21.15.103:40174) 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9ea00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba0101700 0x7f8ba00755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9f400640 1 -- 172.21.15.103:0/1496673312 learned_addr learned my addr 172.21.15.103:0/1496673312 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.754+0000 7f8b9e000640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba01028f0 0x7f8ba0073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9ea00640 1 -- 172.21.15.103:0/1496673312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba01028f0 msgr2=0x7f8ba0073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9ea00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba01028f0 0x7f8ba0073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9ea00640 1 -- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba01032b0 msgr2=0x7f8ba00741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9ea00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba01032b0 0x7f8ba00741a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9ea00640 1 -- 172.21.15.103:0/1496673312 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba0074a60 con 0x7f8ba0101700 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9f400640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba01032b0 0x7f8ba00741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:30.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9ea00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba0101700 0x7f8ba00755d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000e9c0 tx=0x7f8b9000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:30.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b97400640 1 -- 172.21.15.103:0/1496673312 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000ccc0 con 0x7f8ba0101700 2024-05-23T00:00:30.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b97400640 1 -- 172.21.15.103:0/1496673312 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b900035f0 con 0x7f8ba0101700 2024-05-23T00:00:30.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba01a1ad0 con 0x7f8ba0101700 2024-05-23T00:00:30.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b97400640 1 -- 172.21.15.103:0/1496673312 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9000ce60 con 0x7f8ba0101700 2024-05-23T00:00:30.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.755+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba01a1fa0 con 0x7f8ba0101700 2024-05-23T00:00:30.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.756+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba004fe80 con 0x7f8ba0101700 2024-05-23T00:00:30.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.758+0000 7f8b97400640 1 -- 172.21.15.103:0/1496673312 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 20) ==== 137851+0+0 (secure 0 0 0) 0x7f8b900037a0 con 0x7f8ba0101700 2024-05-23T00:00:30.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.759+0000 7f8b97400640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8b840a2bd0 0x7f8b840a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:30.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.760+0000 7f8b9e000640 1 -- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8b840a2bd0 msgr2=0x7f8b840a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:30.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.760+0000 7f8b97400640 1 -- 172.21.15.103:0/1496673312 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f8b900cf970 con 0x7f8ba0101700 2024-05-23T00:00:30.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.760+0000 7f8b9e000640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8b840a2bd0 0x7f8b840a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-23T00:00:30.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.764+0000 7f8b97400640 1 -- 172.21.15.103:0/1496673312 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8b9009c510 con 0x7f8ba0101700 2024-05-23T00:00:30.829 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:30 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:30.677+0000 7fd8df906200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-23T00:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.961+0000 7f8b9e000640 1 -- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8b840a2bd0 msgr2=0x7f8b840a5090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:30.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:30.961+0000 7f8b9e000640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8b840a2bd0 0x7f8b840a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-23T00:00:31.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.008+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f8ba0062470 con 0x7f8ba0101700 2024-05-23T00:00:31.160 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:30 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:30.866+0000 7fd8df906200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-23T00:00:31.160 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:30.998+0000 7fd8df906200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-23T00:00:31.160 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:31.155+0000 7fd8df906200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-23T00:00:31.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.171+0000 7f8b97400640 1 -- 172.21.15.103:0/1496673312 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 21) ==== 137851+0+0 (secure 0 0 0) 0x7f8b900d1d10 con 0x7f8ba0101700 2024-05-23T00:00:31.189 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.186+0000 7f8b97400640 1 -- 172.21.15.103:0/1496673312 <== mon.1 v2:172.21.15.174:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7f8b9008d3c0 con 0x7f8ba0101700 2024-05-23T00:00:31.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8b840a2bd0 msgr2=0x7f8b840a5090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-23T00:00:31.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8b840a2bd0 0x7f8b840a5090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.192 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba0101700 msgr2=0x7f8ba00755d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba0101700 0x7f8ba00755d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8b9000e9c0 tx=0x7f8b9000ee80 comp rx=0 tx=0).stop 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 shutdown_connections 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f8b840a2bd0 0x7f8b840a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ba01032b0 0x7f8ba00741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ba01028f0 0x7f8ba0073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 --2- 172.21.15.103:0/1496673312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ba0101700 0x7f8ba00755d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 >> 172.21.15.103:0/1496673312 conn(0x7f8ba00fcc70 msgr2=0x7f8ba0105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 shutdown_connections 2024-05-23T00:00:31.193 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.189+0000 7f8b9fe00640 1 -- 172.21.15.103:0/1496673312 wait complete. 2024-05-23T00:00:31.211 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-23T00:00:31.222 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-23T00:00:31.258 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:00:31.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 -- 172.21.15.103:0/3003036321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c105680 msgr2=0x7f7d8c10bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:31.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 --2- 172.21.15.103:0/3003036321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c105680 0x7f7d8c10bf10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d84009be0 tx=0x7f7d8402a6c0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 -- 172.21.15.103:0/3003036321 shutdown_connections 2024-05-23T00:00:31.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 --2- 172.21.15.103:0/3003036321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c105680 0x7f7d8c10bf10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 --2- 172.21.15.103:0/3003036321 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d8c104cc0 0x7f7d8c105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 --2- 172.21.15.103:0/3003036321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d8c103ad0 0x7f7d8c103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.457 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 -- 172.21.15.103:0/3003036321 >> 172.21.15.103:0/3003036321 conn(0x7f7d8c0ff060 msgr2=0x7f7d8c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:31.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 -- 172.21.15.103:0/3003036321 shutdown_connections 2024-05-23T00:00:31.458 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.453+0000 7f7d94200640 1 -- 172.21.15.103:0/3003036321 wait complete. 2024-05-23T00:00:31.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.454+0000 7f7d94200640 1 Processor -- start 2024-05-23T00:00:31.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.454+0000 7f7d94200640 1 -- start start 2024-05-23T00:00:31.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d94200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c103ad0 0x7f7d8c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:31.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d94200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d8c104cc0 0x7f7d8c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:31.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d94200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d8c105680 0x7f7d8c1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:31.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d94200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8c10f750 con 0x7f7d8c103ad0 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d94200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8c10f5d0 con 0x7f7d8c105680 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d94200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8c10f8d0 con 0x7f7d8c104cc0 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d91800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c103ad0 0x7f7d8c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d91800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c103ad0 0x7f7d8c199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40206/0 (socket says 172.21.15.103:40206) 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d91800640 1 -- 172.21.15.103:0/1343844300 learned_addr learned my addr 172.21.15.103:0/1343844300 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d92200640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d8c105680 0x7f7d8c1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.455+0000 7f7d90e00640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d8c104cc0 0x7f7d8c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d91800640 1 -- 172.21.15.103:0/1343844300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d8c104cc0 msgr2=0x7f7d8c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d91800640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d8c104cc0 0x7f7d8c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d91800640 1 -- 172.21.15.103:0/1343844300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d8c105680 msgr2=0x7f7d8c1a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d91800640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d8c105680 0x7f7d8c1a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d91800640 1 -- 172.21.15.103:0/1343844300 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8c1a19c0 con 0x7f7d8c103ad0 2024-05-23T00:00:31.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d92200640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d8c105680 0x7f7d8c1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:31.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d91800640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c103ad0 0x7f7d8c199db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000e9a0 tx=0x7f7d8000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:31.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d8a000640 1 -- 172.21.15.103:0/1343844300 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d8000cd80 con 0x7f7d8c103ad0 2024-05-23T00:00:31.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8c1a1cb0 con 0x7f7d8c103ad0 2024-05-23T00:00:31.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d8a000640 1 -- 172.21.15.103:0/1343844300 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d80010910 con 0x7f7d8c103ad0 2024-05-23T00:00:31.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.456+0000 7f7d8a000640 1 -- 172.21.15.103:0/1343844300 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d80018a40 con 0x7f7d8c103ad0 2024-05-23T00:00:31.462 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.457+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8c1a21a0 con 0x7f7d8c103ad0 2024-05-23T00:00:31.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.458+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d8c10ad90 con 0x7f7d8c103ad0 2024-05-23T00:00:31.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.460+0000 7f7d8a000640 1 -- 172.21.15.103:0/1343844300 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 21) ==== 137851+0+0 (secure 0 0 0) 0x7f7d80010ac0 con 0x7f7d8c103ad0 2024-05-23T00:00:31.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.462+0000 7f7d8a000640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7d740a2be0 0x7f7d740a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:31.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.463+0000 7f7d8a000640 1 -- 172.21.15.103:0/1343844300 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f7d800cf810 con 0x7f7d8c103ad0 2024-05-23T00:00:31.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.463+0000 7f7d90e00640 1 -- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7d740a2be0 msgr2=0x7f7d740a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:31.466 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.463+0000 7f7d90e00640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7d740a2be0 0x7f7d740a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-23T00:00:31.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.468+0000 7f7d8a000640 1 -- 172.21.15.103:0/1343844300 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7d80053850 con 0x7f7d8c103ad0 2024-05-23T00:00:31.548 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-05-23T00:00:31.548 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: Standby manager daemon y restarted 2024-05-23T00:00:31.548 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-23T00:00:31.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: Standby manager daemon y started 2024-05-23T00:00:31.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-23T00:00:31.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-23T00:00:31.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-23T00:00:31.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1496673312' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-23T00:00:31.549 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:31 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-23T00:00:31.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-05-23T00:00:31.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: Standby manager daemon y restarted 2024-05-23T00:00:31.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-23T00:00:31.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: Standby manager daemon y started 2024-05-23T00:00:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-23T00:00:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-23T00:00:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-23T00:00:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1496673312' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-23T00:00:31.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:31 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-23T00:00:31.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-05-23T00:00:31.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: Standby manager daemon y restarted 2024-05-23T00:00:31.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-23T00:00:31.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: Standby manager daemon y started 2024-05-23T00:00:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-23T00:00:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-23T00:00:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: from='mgr.? 172.21.15.105:0/2127277948' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-23T00:00:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1496673312' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-23T00:00:31.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:31 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-23T00:00:31.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.664+0000 7f7d90e00640 1 -- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7d740a2be0 msgr2=0x7f7d740a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:31.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.664+0000 7f7d90e00640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7d740a2be0 0x7f7d740a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-23T00:00:31.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.671+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d8c062510 con 0x7f7d8c103ad0 2024-05-23T00:00:31.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.673+0000 7f7d8a000640 1 -- 172.21.15.103:0/1343844300 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24658 (secure 0 0 0) 0x7f7d8009c3b0 con 0x7f7d8c103ad0 2024-05-23T00:00:31.676 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:31.677 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":93,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-22T23:58:31.088381+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:00:31.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7d740a2be0 msgr2=0x7f7d740a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-23T00:00:31.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7d740a2be0 0x7f7d740a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c103ad0 msgr2=0x7f7d8c199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:31.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c103ad0 0x7f7d8c199db0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7d8000e9a0 tx=0x7f7d8000ee60 comp rx=0 tx=0).stop 2024-05-23T00:00:31.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 shutdown_connections 2024-05-23T00:00:31.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f7d740a2be0 0x7f7d740a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d8c105680 0x7f7d8c1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d8c104cc0 0x7f7d8c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 --2- 172.21.15.103:0/1343844300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d8c103ad0 0x7f7d8c199db0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 >> 172.21.15.103:0/1343844300 conn(0x7f7d8c0ff060 msgr2=0x7f7d8c107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:31.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 shutdown_connections 2024-05-23T00:00:31.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.675+0000 7f7d94200640 1 -- 172.21.15.103:0/1343844300 wait complete. 2024-05-23T00:00:31.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-23T00:00:31.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-23T00:00:31.829 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:31.544+0000 7fd8df906200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-23T00:00:31.890 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 -- 172.21.15.103:0/3653416475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c103330 msgr2=0x7f036c109bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:31.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 --2- 172.21.15.103:0/3653416475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c103330 0x7f036c109bc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0368009ca0 tx=0x7f036802a8c0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 -- 172.21.15.103:0/3653416475 shutdown_connections 2024-05-23T00:00:31.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 --2- 172.21.15.103:0/3653416475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c103330 0x7f036c109bc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 --2- 172.21.15.103:0/3653416475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f036c102970 0x7f036c102df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 --2- 172.21.15.103:0/3653416475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c101780 0x7f036c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 -- 172.21.15.103:0/3653416475 >> 172.21.15.103:0/3653416475 conn(0x7f036c0fcd10 msgr2=0x7f036c0ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:31.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 -- 172.21.15.103:0/3653416475 shutdown_connections 2024-05-23T00:00:31.891 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.887+0000 7f0373200640 1 -- 172.21.15.103:0/3653416475 wait complete. 2024-05-23T00:00:31.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.888+0000 7f0373200640 1 Processor -- start 2024-05-23T00:00:31.893 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.888+0000 7f0373200640 1 -- start start 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.888+0000 7f0373200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f036c101780 0x7f036c197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.888+0000 7f0373200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c102970 0x7f036c197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.888+0000 7f0373200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c103330 0x7f036c19eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.888+0000 7f0373200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f036c10d430 con 0x7f036c102970 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.888+0000 7f0373200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f036c10d2b0 con 0x7f036c101780 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.888+0000 7f0373200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f036c10d5b0 con 0x7f036c103330 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c102970 0x7f036c197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f036c101780 0x7f036c197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f036c101780 0x7f036c197a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48692/0 (socket says 172.21.15.103:48692) 2024-05-23T00:00:31.894 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 -- 172.21.15.103:0/892881402 learned_addr learned my addr 172.21.15.103:0/892881402 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0372800640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c103330 0x7f036c19eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 -- 172.21.15.103:0/892881402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c103330 msgr2=0x7f036c19eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c103330 0x7f036c19eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 -- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c102970 msgr2=0x7f036c197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c102970 0x7f036c197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 -- 172.21.15.103:0/892881402 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f036c19f660 con 0x7f036c101780 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371400640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c102970 0x7f036c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.889+0000 7f0371e00640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f036c101780 0x7f036c197a40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f036000e9c0 tx=0x7f036000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.890+0000 7f0366a00640 1 -- 172.21.15.103:0/892881402 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f036000cde0 con 0x7f036c101780 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.890+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f036c19f950 con 0x7f036c101780 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.890+0000 7f0366a00640 1 -- 172.21.15.103:0/892881402 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0360010910 con 0x7f036c101780 2024-05-23T00:00:31.895 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.890+0000 7f0366a00640 1 -- 172.21.15.103:0/892881402 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0360018a10 con 0x7f036c101780 2024-05-23T00:00:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.890+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f036c19fef0 con 0x7f036c101780 2024-05-23T00:00:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.892+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f036c068a00 con 0x7f036c101780 2024-05-23T00:00:31.896 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.893+0000 7f0366a00640 1 -- 172.21.15.103:0/892881402 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 21) ==== 137851+0+0 (secure 0 0 0) 0x7f036001f020 con 0x7f036c101780 2024-05-23T00:00:31.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.894+0000 7f0366a00640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f03540a2c60 0x7f03540a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:31.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.895+0000 7f0366a00640 1 -- 172.21.15.103:0/892881402 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7f03600cf5f0 con 0x7f036c101780 2024-05-23T00:00:31.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.895+0000 7f0371400640 1 -- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f03540a2c60 msgr2=0x7f03540a5120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:31.898 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.895+0000 7f0371400640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f03540a2c60 0x7f03540a5120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-23T00:00:31.902 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:31.899+0000 7f0366a00640 1 -- 172.21.15.103:0/892881402 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f036009c190 con 0x7f036c101780 2024-05-23T00:00:32.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.096+0000 7f0371400640 1 -- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f03540a2c60 msgr2=0x7f03540a5120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:32.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.096+0000 7f0371400640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f03540a2c60 0x7f03540a5120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-23T00:00:32.103 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.100+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f036c108a40 con 0x7f036c101780 2024-05-23T00:00:32.104 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.101+0000 7f0366a00640 1 -- 172.21.15.103:0/892881402 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f036c108a40 con 0x7f036c101780 2024-05-23T00:00:32.105 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-23T00:00:32.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.103+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f03540a2c60 msgr2=0x7f03540a5120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-23T00:00:32.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.103+0000 7f0373200640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f03540a2c60 0x7f03540a5120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.107 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.103+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f036c101780 msgr2=0x7f036c197a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.103+0000 7f0373200640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f036c101780 0x7f036c197a40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f036000e9c0 tx=0x7f036000ee80 comp rx=0 tx=0).stop 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.104+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 shutdown_connections 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.104+0000 7f0373200640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7f03540a2c60 0x7f03540a5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.104+0000 7f0373200640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f036c103330 0x7f036c19eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.104+0000 7f0373200640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f036c102970 0x7f036c197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.104+0000 7f0373200640 1 --2- 172.21.15.103:0/892881402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f036c101780 0x7f036c197a40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.104+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 >> 172.21.15.103:0/892881402 conn(0x7f036c0fcd10 msgr2=0x7f036c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.104+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 shutdown_connections 2024-05-23T00:00:32.108 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.104+0000 7f0373200640 1 -- 172.21.15.103:0/892881402 wait complete. 2024-05-23T00:00:32.125 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 -- 172.21.15.103:0/1809586712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 msgr2=0x7fe9c8105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/1809586712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 0x7fe9c8105140 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4009900 tx=0x7fe9b402a410 comp rx=0 tx=0).stop 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 -- 172.21.15.103:0/1809586712 shutdown_connections 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/1809586712 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9c8105680 0x7fe9c810bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/1809586712 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 0x7fe9c8105140 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/1809586712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c8103ad0 0x7fe9c8103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 -- 172.21.15.103:0/1809586712 >> 172.21.15.103:0/1809586712 conn(0x7fe9c80ff060 msgr2=0x7fe9c8101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 -- 172.21.15.103:0/1809586712 shutdown_connections 2024-05-23T00:00:32.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.322+0000 7fe9cfa00640 1 -- 172.21.15.103:0/1809586712 wait complete. 2024-05-23T00:00:32.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.323+0000 7fe9cfa00640 1 Processor -- start 2024-05-23T00:00:32.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.323+0000 7fe9cfa00640 1 -- start start 2024-05-23T00:00:32.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9cfa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9c8103ad0 0x7fe9c8199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9cfa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 0x7fe9c819a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9cfa00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c8105680 0x7fe9c81a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9cfa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c810f780 con 0x7fe9c8104cc0 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9cfa00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c810f600 con 0x7fe9c8103ad0 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9cfa00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9c810f900 con 0x7fe9c8105680 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9c7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 0x7fe9c819a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9c7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 0x7fe9c819a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40236/0 (socket says 172.21.15.103:40236) 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9c7e00640 1 -- 172.21.15.103:0/2109121627 learned_addr learned my addr 172.21.15.103:0/2109121627 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9cda00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c8105680 0x7fe9c81a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9cd000640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9c8103ad0 0x7fe9c8199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9c7e00640 1 -- 172.21.15.103:0/2109121627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c8105680 msgr2=0x7fe9c81a13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9c7e00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c8105680 0x7fe9c81a13a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.329 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9c7e00640 1 -- 172.21.15.103:0/2109121627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9c8103ad0 msgr2=0x7fe9c8199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9c7e00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9c8103ad0 0x7fe9c8199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.324+0000 7fe9c7e00640 1 -- 172.21.15.103:0/2109121627 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9c81a1a10 con 0x7fe9c8104cc0 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.325+0000 7fe9c7e00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 0x7fe9c819a330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4031440 tx=0x7fe9b4032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.325+0000 7fe9c5600640 1 -- 172.21.15.103:0/2109121627 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b403f030 con 0x7fe9c8104cc0 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.325+0000 7fe9c5600640 1 -- 172.21.15.103:0/2109121627 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe9b4003ce0 con 0x7fe9c8104cc0 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.325+0000 7fe9c5600640 1 -- 172.21.15.103:0/2109121627 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4035070 con 0x7fe9c8104cc0 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.325+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9c81a1ca0 con 0x7fe9c8104cc0 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.325+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9c81a2190 con 0x7fe9c8104cc0 2024-05-23T00:00:32.330 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.326+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9c810adf0 con 0x7fe9c8104cc0 2024-05-23T00:00:32.331 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.328+0000 7fe9c5600640 1 -- 172.21.15.103:0/2109121627 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 21) ==== 137851+0+0 (secure 0 0 0) 0x7fe9b4042030 con 0x7fe9c8104cc0 2024-05-23T00:00:32.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.330+0000 7fe9c5600640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9b00a2be0 0x7fe9b00a50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:32.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.331+0000 7fe9cd000640 1 -- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9b00a2be0 msgr2=0x7fe9b00a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:32.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.331+0000 7fe9c5600640 1 -- 172.21.15.103:0/2109121627 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9092+0+0 (secure 0 0 0) 0x7fe9b40ef880 con 0x7fe9c8104cc0 2024-05-23T00:00:32.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.331+0000 7fe9cd000640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9b00a2be0 0x7fe9b00a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-23T00:00:32.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.335+0000 7fe9c5600640 1 -- 172.21.15.103:0/2109121627 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe9b40bc480 con 0x7fe9c8104cc0 2024-05-23T00:00:32.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.532+0000 7fe9cd000640 1 -- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9b00a2be0 msgr2=0x7fe9b00a50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.103:6800/1744216601 2024-05-23T00:00:32.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.532+0000 7fe9cd000640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9b00a2be0 0x7fe9b00a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-23T00:00:32.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.537+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe9c8062470 con 0x7fe9c8104cc0 2024-05-23T00:00:32.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.538+0000 7fe9c5600640 1 -- 172.21.15.103:0/2109121627 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fe9b40bbd80 con 0x7fe9c8104cc0 2024-05-23T00:00:32.542 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:32.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.540+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9b00a2be0 msgr2=0x7fe9b00a50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-23T00:00:32.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.540+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9b00a2be0 0x7fe9b00a50a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.540+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 msgr2=0x7fe9c819a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:32.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.540+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 0x7fe9c819a330 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4031440 tx=0x7fe9b4032ce0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.541+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 shutdown_connections 2024-05-23T00:00:32.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.541+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:6800/1744216601,v1:172.21.15.103:6801/1744216601] conn(0x7fe9b00a2be0 0x7fe9b00a50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.541+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9c8105680 0x7fe9c81a13a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.541+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe9c8104cc0 0x7fe9c819a330 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.541+0000 7fe9cfa00640 1 --2- 172.21.15.103:0/2109121627 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9c8103ad0 0x7fe9c8199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.541+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 >> 172.21.15.103:0/2109121627 conn(0x7fe9c80ff060 msgr2=0x7fe9c8107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:32.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.541+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 shutdown_connections 2024-05-23T00:00:32.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.541+0000 7fe9cfa00640 1 -- 172.21.15.103:0/2109121627 wait complete. 2024-05-23T00:00:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:32 smithi105 ceph-mon[36426]: mgrmap e21: x(active, since 11m), standbys: z, y 2024-05-23T00:00:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:32 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-23T00:00:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:32 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:00:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:32 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1343844300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:32 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/892881402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:32.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:32 smithi103 ceph-mon[31370]: mgrmap e21: x(active, since 11m), standbys: z, y 2024-05-23T00:00:32.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:32 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-23T00:00:32.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:32 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:00:32.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:32 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1343844300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:32.560 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:32 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/892881402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:32.560 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:32.354+0000 7fd8df906200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-23T00:00:32.564 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-23T00:00:32.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:32 smithi174 ceph-mon[36179]: mgrmap e21: x(active, since 11m), standbys: z, y 2024-05-23T00:00:32.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:32 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-23T00:00:32.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:32 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:00:32.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:32 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1343844300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:32.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:32 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/892881402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.755+0000 7fecf8c00640 1 -- 172.21.15.103:0/4247929832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4102990 msgr2=0x7fecf4102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.755+0000 7fecf8c00640 1 --2- 172.21.15.103:0/4247929832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4102990 0x7fecf4102e10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fece0009860 tx=0x7fece002a340 comp rx=0 tx=0).stop 2024-05-23T00:00:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.756+0000 7fecf8c00640 1 -- 172.21.15.103:0/4247929832 shutdown_connections 2024-05-23T00:00:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.756+0000 7fecf8c00640 1 --2- 172.21.15.103:0/4247929832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecf4103350 0x7fecf4109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.756+0000 7fecf8c00640 1 --2- 172.21.15.103:0/4247929832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4102990 0x7fecf4102e10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.756+0000 7fecf8c00640 1 --2- 172.21.15.103:0/4247929832 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecf41017a0 0x7fecf4101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.756+0000 7fecf8c00640 1 -- 172.21.15.103:0/4247929832 >> 172.21.15.103:0/4247929832 conn(0x7fecf40fcd10 msgr2=0x7fecf40ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.756+0000 7fecf8c00640 1 -- 172.21.15.103:0/4247929832 shutdown_connections 2024-05-23T00:00:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.756+0000 7fecf8c00640 1 -- 172.21.15.103:0/4247929832 wait complete. 2024-05-23T00:00:32.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.757+0000 7fecf8c00640 1 Processor -- start 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.757+0000 7fecf8c00640 1 -- start start 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf8c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecf41017a0 0x7fecf4197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf8c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecf4102990 0x7fecf4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf8c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4103350 0x7fecf419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf8c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecf410d400 con 0x7fecf4103350 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf8c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fecf410d280 con 0x7fecf41017a0 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf8c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fecf410d580 con 0x7fecf4102990 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4103350 0x7fecf419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4103350 0x7fecf419f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40290/0 (socket says 172.21.15.103:40290) 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf3e00640 1 -- 172.21.15.103:0/2321464426 learned_addr learned my addr 172.21.15.103:0/2321464426 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf2a00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecf4102990 0x7fecf4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf3400640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecf41017a0 0x7fecf4197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf2a00640 1 -- 172.21.15.103:0/2321464426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecf41017a0 msgr2=0x7fecf4197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf2a00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecf41017a0 0x7fecf4197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf2a00640 1 -- 172.21.15.103:0/2321464426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4103350 msgr2=0x7fecf419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf2a00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4103350 0x7fecf419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf2a00640 1 -- 172.21.15.103:0/2321464426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf419f6c0 con 0x7fecf4102990 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf3e00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4103350 0x7fecf419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.758+0000 7fecf3400640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecf41017a0 0x7fecf4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.759+0000 7fecf2a00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecf4102990 0x7fecf4197fe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fece0035040 tx=0x7fece0033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.759+0000 7fecebe00640 1 -- 172.21.15.103:0/2321464426 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0037040 con 0x7fecf4102990 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.759+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf419f950 con 0x7fecf4102990 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.759+0000 7fecebe00640 1 -- 172.21.15.103:0/2321464426 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fece0004040 con 0x7fecf4102990 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.759+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf419fe90 con 0x7fecf4102990 2024-05-23T00:00:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.759+0000 7fecebe00640 1 -- 172.21.15.103:0/2321464426 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece00055f0 con 0x7fecf4102990 2024-05-23T00:00:32.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.761+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecf4108ac0 con 0x7fecf4102990 2024-05-23T00:00:32.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.763+0000 7fecebe00640 1 -- 172.21.15.103:0/2321464426 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 22) ==== 137518+0+0 (secure 0 0 0) 0x7fece0042020 con 0x7fecf4102990 2024-05-23T00:00:32.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.765+0000 7fecebe00640 1 -- 172.21.15.103:0/2321464426 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fece00ee8e0 con 0x7fecf4102990 2024-05-23T00:00:32.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:32.769+0000 7fecebe00640 1 -- 172.21.15.103:0/2321464426 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fece004a020 con 0x7fecf4102990 2024-05-23T00:00:32.829 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:32.656+0000 7fd8df906200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-23T00:00:33.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.047+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fecf4062470 con 0x7fecf4102990 2024-05-23T00:00:33.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.048+0000 7fecebe00640 1 -- 172.21.15.103:0/2321464426 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7fece00bb810 con 0x7fecf4102990 2024-05-23T00:00:33.051 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecf4102990 msgr2=0x7fecf4197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecf4102990 0x7fecf4197fe0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fece0035040 tx=0x7fece0033d90 comp rx=0 tx=0).stop 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 shutdown_connections 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecf4103350 0x7fecf419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fecf4102990 0x7fecf4197fe0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 --2- 172.21.15.103:0/2321464426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fecf41017a0 0x7fecf4197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 >> 172.21.15.103:0/2321464426 conn(0x7fecf40fcd10 msgr2=0x7fecf4105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 shutdown_connections 2024-05-23T00:00:33.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.050+0000 7fecf8c00640 1 -- 172.21.15.103:0/2321464426 wait complete. 2024-05-23T00:00:33.071 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-23T00:00:33.182 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.010+0000 7fd835200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.182 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.010+0000 7fd835200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.182 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.010+0000 7fd835200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.182 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.010+0000 7fd835200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.182 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.010+0000 7fd835200700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.182 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.011+0000 7fd832000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.183 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.011+0000 7fd832000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.183 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.011+0000 7fd832000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.183 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.011+0000 7fd832000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.183 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:00:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:00:33.011+0000 7fd832000700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:00:33.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.240+0000 7fd321c00640 1 -- 172.21.15.103:0/1722007187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd31c100ae0 msgr2=0x7fd31c100f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.240+0000 7fd321c00640 1 --2- 172.21.15.103:0/1722007187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd31c100ae0 0x7fd31c100f40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd304009900 tx=0x7fd30402a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.241+0000 7fd321c00640 1 -- 172.21.15.103:0/1722007187 shutdown_connections 2024-05-23T00:00:33.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.241+0000 7fd321c00640 1 --2- 172.21.15.103:0/1722007187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd31c101480 0x7fd31c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.241+0000 7fd321c00640 1 --2- 172.21.15.103:0/1722007187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd31c100ae0 0x7fd31c100f40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.241+0000 7fd321c00640 1 --2- 172.21.15.103:0/1722007187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd31c100180 0x7fd31c1005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.241+0000 7fd321c00640 1 -- 172.21.15.103:0/1722007187 >> 172.21.15.103:0/1722007187 conn(0x7fd31c0fbd20 msgr2=0x7fd31c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:33.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.241+0000 7fd321c00640 1 -- 172.21.15.103:0/1722007187 shutdown_connections 2024-05-23T00:00:33.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.241+0000 7fd321c00640 1 -- 172.21.15.103:0/1722007187 wait complete. 2024-05-23T00:00:33.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.242+0000 7fd321c00640 1 Processor -- start 2024-05-23T00:00:33.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.242+0000 7fd321c00640 1 -- start start 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd321c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd31c100180 0x7fd31c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd321c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd31c100ae0 0x7fd31c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd321c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd31c101480 0x7fd31c1a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd321c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c10f6f0 con 0x7fd31c100180 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd321c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c10f570 con 0x7fd31c101480 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd321c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c10f870 con 0x7fd31c100ae0 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd31c100180 0x7fd31c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd313e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd31c100ae0 0x7fd31c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31aa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd31c100180 0x7fd31c199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:40408/0 (socket says 172.21.15.103:40408) 2024-05-23T00:00:33.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31aa00640 1 -- 172.21.15.103:0/620887877 learned_addr learned my addr 172.21.15.103:0/620887877 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:33.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31b400640 1 --2- 172.21.15.103:0/620887877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd31c101480 0x7fd31c1a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:33.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31aa00640 1 -- 172.21.15.103:0/620887877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd31c100ae0 msgr2=0x7fd31c19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31aa00640 1 --2- 172.21.15.103:0/620887877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd31c100ae0 0x7fd31c19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31aa00640 1 -- 172.21.15.103:0/620887877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd31c101480 msgr2=0x7fd31c1a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31aa00640 1 --2- 172.21.15.103:0/620887877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd31c101480 0x7fd31c1a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.243+0000 7fd31aa00640 1 -- 172.21.15.103:0/620887877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c1a19d0 con 0x7fd31c100180 2024-05-23T00:00:33.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.244+0000 7fd31aa00640 1 --2- 172.21.15.103:0/620887877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd31c100180 0x7fd31c199dc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00e9f0 tx=0x7fd30c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:33.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.244+0000 7fd313400640 1 -- 172.21.15.103:0/620887877 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c00ccc0 con 0x7fd31c100180 2024-05-23T00:00:33.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.244+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c1a1cc0 con 0x7fd31c100180 2024-05-23T00:00:33.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.244+0000 7fd313400640 1 -- 172.21.15.103:0/620887877 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd30c010910 con 0x7fd31c100180 2024-05-23T00:00:33.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.244+0000 7fd313400640 1 -- 172.21.15.103:0/620887877 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd30c018840 con 0x7fd31c100180 2024-05-23T00:00:33.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.244+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c1a21b0 con 0x7fd31c100180 2024-05-23T00:00:33.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.245+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd31c10b050 con 0x7fd31c100180 2024-05-23T00:00:33.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.247+0000 7fd313400640 1 -- 172.21.15.103:0/620887877 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 22) ==== 137518+0+0 (secure 0 0 0) 0x7fd30c010ac0 con 0x7fd31c100180 2024-05-23T00:00:33.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.250+0000 7fd313400640 1 -- 172.21.15.103:0/620887877 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd30c0cf3d0 con 0x7fd31c100180 2024-05-23T00:00:33.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.253+0000 7fd313400640 1 -- 172.21.15.103:0/620887877 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd30c09c270 con 0x7fd31c100180 2024-05-23T00:00:33.460 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.456+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd31c062510 con 0x7fd31c100180 2024-05-23T00:00:33.461 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.457+0000 7fd313400640 1 -- 172.21.15.103:0/620887877 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7fd30c09bb70 con 0x7fd31c100180 2024-05-23T00:00:33.461 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:33.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd31c100180 msgr2=0x7fd31c199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 --2- 172.21.15.103:0/620887877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd31c100180 0x7fd31c199dc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd30c00e9f0 tx=0x7fd30c00edf0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 shutdown_connections 2024-05-23T00:00:33.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 --2- 172.21.15.103:0/620887877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd31c101480 0x7fd31c1a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.463 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 --2- 172.21.15.103:0/620887877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd31c100ae0 0x7fd31c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 --2- 172.21.15.103:0/620887877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd31c100180 0x7fd31c199dc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 >> 172.21.15.103:0/620887877 conn(0x7fd31c0fbd20 msgr2=0x7fd31c0fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:33.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 shutdown_connections 2024-05-23T00:00:33.464 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.460+0000 7fd321c00640 1 -- 172.21.15.103:0/620887877 wait complete. 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2109121627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: Active manager daemon x restarted 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: Activating manager daemon x 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: osdmap e94: 12 total, 12 up, 12 in 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: mgrmap e22: x(active, starting, since 0.0218187s), standbys: z, y 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-23T00:00:33.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-23T00:00:33.481 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-23T00:00:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-23T00:00:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-23T00:00:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-23T00:00:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-23T00:00:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-23T00:00:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-23T00:00:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-23T00:00:33.482 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: Manager daemon x is now available 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-23T00:00:33.483 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:33 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2321464426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:33.484 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:00:33.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2109121627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: Active manager daemon x restarted 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: Activating manager daemon x 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: osdmap e94: 12 total, 12 up, 12 in 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: mgrmap e22: x(active, starting, since 0.0218187s), standbys: z, y 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-23T00:00:33.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-23T00:00:33.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: Manager daemon x is now available 2024-05-23T00:00:33.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:33.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-23T00:00:33.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-23T00:00:33.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:33 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2321464426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:33.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2109121627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:33.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: Active manager daemon x restarted 2024-05-23T00:00:33.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: Activating manager daemon x 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: osdmap e94: 12 total, 12 up, 12 in 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: mgrmap e22: x(active, starting, since 0.0218187s), standbys: z, y 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-23T00:00:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-23T00:00:33.619 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-23T00:00:33.620 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: Manager daemon x is now available 2024-05-23T00:00:33.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:33.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-23T00:00:33.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-23T00:00:33.621 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:33 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2321464426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.659+0000 7ff455400640 1 -- 172.21.15.103:0/1894256661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 msgr2=0x7ff450102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.659+0000 7ff455400640 1 --2- 172.21.15.103:0/1894256661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 0x7ff450102d80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff438009900 tx=0x7ff43802a410 comp rx=0 tx=0).stop 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.659+0000 7ff455400640 1 -- 172.21.15.103:0/1894256661 shutdown_connections 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.659+0000 7ff455400640 1 --2- 172.21.15.103:0/1894256661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4501032c0 0x7ff450109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.659+0000 7ff455400640 1 --2- 172.21.15.103:0/1894256661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 0x7ff450102d80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.659+0000 7ff455400640 1 --2- 172.21.15.103:0/1894256661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff450101710 0x7ff450101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.659+0000 7ff455400640 1 -- 172.21.15.103:0/1894256661 >> 172.21.15.103:0/1894256661 conn(0x7ff4500fcc80 msgr2=0x7ff4500ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.659+0000 7ff455400640 1 -- 172.21.15.103:0/1894256661 shutdown_connections 2024-05-23T00:00:33.663 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.660+0000 7ff455400640 1 -- 172.21.15.103:0/1894256661 wait complete. 2024-05-23T00:00:33.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.660+0000 7ff455400640 1 Processor -- start 2024-05-23T00:00:33.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff455400640 1 -- start start 2024-05-23T00:00:33.665 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff455400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff450101710 0x7ff450199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff455400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 0x7ff45019a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff455400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4501032c0 0x7ff4501a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff455400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff45010d3c0 con 0x7ff450101710 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff455400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff45010d240 con 0x7ff450102900 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff455400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff45010d540 con 0x7ff4501032c0 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff454a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4501032c0 0x7ff4501a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff447e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 0x7ff45019a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff454a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4501032c0 0x7ff4501a11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:48070/0 (socket says 172.21.15.103:48070) 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff447e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 0x7ff45019a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48912/0 (socket says 172.21.15.103:48912) 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff454a00640 1 -- 172.21.15.103:0/833603481 learned_addr learned my addr 172.21.15.103:0/833603481 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.661+0000 7ff44fe00640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff450101710 0x7ff450199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:33.666 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff454a00640 1 -- 172.21.15.103:0/833603481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 msgr2=0x7ff45019a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff454a00640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 0x7ff45019a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff454a00640 1 -- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff450101710 msgr2=0x7ff450199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff454a00640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff450101710 0x7ff450199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff454a00640 1 -- 172.21.15.103:0/833603481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4501a1880 con 0x7ff4501032c0 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff454a00640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4501032c0 0x7ff4501a11e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff44800efd0 tx=0x7ff44800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff44d600640 1 -- 172.21.15.103:0/833603481 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff448003b70 con 0x7ff4501032c0 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4501a1b70 con 0x7ff4501032c0 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff44d600640 1 -- 172.21.15.103:0/833603481 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff448003d10 con 0x7ff4501032c0 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff44d600640 1 -- 172.21.15.103:0/833603481 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4480109f0 con 0x7ff4501032c0 2024-05-23T00:00:33.667 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.662+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4501a20c0 con 0x7ff4501032c0 2024-05-23T00:00:33.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.663+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4501a2290 con 0x7ff4501032c0 2024-05-23T00:00:33.668 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.665+0000 7ff44d600640 1 -- 172.21.15.103:0/833603481 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 22) ==== 137518+0+0 (secure 0 0 0) 0x7ff448041d10 con 0x7ff4501032c0 2024-05-23T00:00:33.670 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.667+0000 7ff44d600640 1 -- 172.21.15.103:0/833603481 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ff4480cf3b0 con 0x7ff4501032c0 2024-05-23T00:00:33.672 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.669+0000 7ff44d600640 1 -- 172.21.15.103:0/833603481 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff44809c250 con 0x7ff4501032c0 2024-05-23T00:00:33.686 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.683+0000 7ff44d600640 1 -- 172.21.15.103:0/833603481 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 23) ==== 137645+0+0 (secure 0 0 0) 0x7ff448097810 con 0x7ff4501032c0 2024-05-23T00:00:33.687 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.683+0000 7ff44d600640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff4340aaef0 0x7ff4340ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:33.691 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.687+0000 7ff44fe00640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff4340aaef0 0x7ff4340ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:33.695 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.692+0000 7ff44fe00640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff4340aaef0 0x7ff4340ad2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff45019b140 tx=0x7ff440005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:33.921 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.918+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4501a2460 con 0x7ff4501032c0 2024-05-23T00:00:33.923 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.920+0000 7ff44d600640 1 -- 172.21.15.103:0/833603481 <== mon.1 v2:172.21.15.174:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25041 (secure 0 0 0) 0x7ff44808aee0 con 0x7ff4501032c0 2024-05-23T00:00:33.924 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:33.924 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":94,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-23T00:00:32.661164+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/284924627":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3929139004":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3827978298":"2024-05-24T00:00:32.661117+0000","172.21.15.103:6801/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/2531882208":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:00:33.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.922+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff4340aaef0 msgr2=0x7ff4340ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.922+0000 7ff455400640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff4340aaef0 0x7ff4340ad2e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff45019b140 tx=0x7ff440005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:33.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.922+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4501032c0 msgr2=0x7ff4501a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:33.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.922+0000 7ff455400640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4501032c0 0x7ff4501a11e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff44800efd0 tx=0x7ff44800c6a0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.926 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.923+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 shutdown_connections 2024-05-23T00:00:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.923+0000 7ff455400640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff4340aaef0 0x7ff4340ad2e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.923+0000 7ff455400640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4501032c0 0x7ff4501a11e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.923+0000 7ff455400640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff450102900 0x7ff45019a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.923+0000 7ff455400640 1 --2- 172.21.15.103:0/833603481 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff450101710 0x7ff450199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.923+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 >> 172.21.15.103:0/833603481 conn(0x7ff4500fcc80 msgr2=0x7ff450073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.923+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 shutdown_connections 2024-05-23T00:00:33.927 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:33.923+0000 7ff455400640 1 -- 172.21.15.103:0/833603481 wait complete. 2024-05-23T00:00:33.945 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-23T00:00:33.946 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-23T00:00:34.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.125+0000 7f3e2be00640 1 -- 172.21.15.103:0/3205122292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 msgr2=0x7f3e2c101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.134 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.125+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3205122292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 0x7f3e2c101bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009900 tx=0x7f3e1c02a410 comp rx=0 tx=0).stop 2024-05-23T00:00:34.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.125+0000 7f3e2be00640 1 -- 172.21.15.103:0/3205122292 shutdown_connections 2024-05-23T00:00:34.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.125+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3205122292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2c103350 0x7f3e2c109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.125+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3205122292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e2c102990 0x7f3e2c102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.135 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.125+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3205122292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 0x7f3e2c101bc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.125+0000 7f3e2be00640 1 -- 172.21.15.103:0/3205122292 >> 172.21.15.103:0/3205122292 conn(0x7f3e2c0fcd10 msgr2=0x7f3e2c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:34.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.125+0000 7f3e2be00640 1 -- 172.21.15.103:0/3205122292 shutdown_connections 2024-05-23T00:00:34.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.126+0000 7f3e2be00640 1 -- 172.21.15.103:0/3205122292 wait complete. 2024-05-23T00:00:34.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2be00640 1 Processor -- start 2024-05-23T00:00:34.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2be00640 1 -- start start 2024-05-23T00:00:34.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 0x7f3e2c199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:34.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e2c102990 0x7f3e2c19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:34.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2c103350 0x7f3e2c1a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:34.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2c10d420 con 0x7f3e2c102990 2024-05-23T00:00:34.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2c10d2a0 con 0x7f3e2c1017a0 2024-05-23T00:00:34.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2be00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3e2c10d5a0 con 0x7f3e2c103350 2024-05-23T00:00:34.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 0x7f3e2c199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:34.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 0x7f3e2c199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49314/0 (socket says 172.21.15.103:49314) 2024-05-23T00:00:34.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e2aa00640 1 -- 172.21.15.103:0/3885839073 learned_addr learned my addr 172.21.15.103:0/3885839073 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:34.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.127+0000 7f3e23e00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e2c102990 0x7f3e2c19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:34.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2b400640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2c103350 0x7f3e2c1a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:34.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2aa00640 1 -- 172.21.15.103:0/3885839073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2c103350 msgr2=0x7f3e2c1a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2aa00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2c103350 0x7f3e2c1a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2aa00640 1 -- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e2c102990 msgr2=0x7f3e2c19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2aa00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e2c102990 0x7f3e2c19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2aa00640 1 -- 172.21.15.103:0/3885839073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e2c1a1930 con 0x7f3e2c1017a0 2024-05-23T00:00:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2b400640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2c103350 0x7f3e2c1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2aa00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 0x7f3e2c199d20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c031610 tx=0x7f3e1c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e23400640 1 -- 172.21.15.103:0/3885839073 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c03f030 con 0x7f3e2c1017a0 2024-05-23T00:00:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e23400640 1 -- 172.21.15.103:0/3885839073 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e1c003ad0 con 0x7f3e2c1017a0 2024-05-23T00:00:34.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e2c1a1bc0 con 0x7f3e2c1017a0 2024-05-23T00:00:34.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.128+0000 7f3e23400640 1 -- 172.21.15.103:0/3885839073 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c035070 con 0x7f3e2c1017a0 2024-05-23T00:00:34.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.129+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e2c1a20b0 con 0x7f3e2c1017a0 2024-05-23T00:00:34.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.130+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e2c1a2280 con 0x7f3e2c1017a0 2024-05-23T00:00:34.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.132+0000 7f3e23400640 1 -- 172.21.15.103:0/3885839073 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 23) ==== 137645+0+0 (secure 0 0 0) 0x7f3e1c042030 con 0x7f3e2c1017a0 2024-05-23T00:00:34.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.133+0000 7f3e23400640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3e100a2990 0x7f3e100a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:34.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.133+0000 7f3e23400640 1 -- 172.21.15.103:0/3885839073 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f3e1c0ef7b0 con 0x7f3e2c1017a0 2024-05-23T00:00:34.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.133+0000 7f3e23e00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3e100a2990 0x7f3e100a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:34.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.134+0000 7f3e23e00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3e100a2990 0x7f3e100a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009bd0 tx=0x7f3e14005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:34.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.137+0000 7f3e23400640 1 -- 172.21.15.103:0/3885839073 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3e1c0bc560 con 0x7f3e2c1017a0 2024-05-23T00:00:34.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: [23/May/2024:00:00:33] ENGINE Bus STARTING 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: [23/May/2024:00:00:33] ENGINE Serving on http://172.21.15.103:8765 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/620887877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: mgrmap e23: x(active, since 1.02144s), standbys: z, y 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/833603481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.333+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e2c1a2450 con 0x7f3e2c1017a0 2024-05-23T00:00:34.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.334+0000 7f3e23400640 1 -- 172.21.15.103:0/3885839073 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f3e1c0bbe60 con 0x7f3e2c1017a0 2024-05-23T00:00:34.338 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-23T00:00:34.340 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3e100a2990 msgr2=0x7f3e100a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3e100a2990 0x7f3e100a4e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e14009bd0 tx=0x7f3e14005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 msgr2=0x7f3e2c199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 0x7f3e2c199d20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c031610 tx=0x7f3e1c004270 comp rx=0 tx=0).stop 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 shutdown_connections 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3e100a2990 0x7f3e100a4e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e2c103350 0x7f3e2c1a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e2c102990 0x7f3e2c19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 --2- 172.21.15.103:0/3885839073 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e2c1017a0 0x7f3e2c199d20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 >> 172.21.15.103:0/3885839073 conn(0x7f3e2c0fcd10 msgr2=0x7f3e2c073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 shutdown_connections 2024-05-23T00:00:34.341 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.337+0000 7f3e2be00640 1 -- 172.21.15.103:0/3885839073 wait complete. 2024-05-23T00:00:34.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-23T00:00:34.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.550+0000 7fc477e00640 1 -- 172.21.15.103:0/1269544341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 msgr2=0x7fc4781089d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.550+0000 7fc477e00640 1 --2- 172.21.15.103:0/1269544341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 0x7fc4781089d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc470009ca0 tx=0x7fc47002a8c0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.550+0000 7fc477e00640 1 -- 172.21.15.103:0/1269544341 shutdown_connections 2024-05-23T00:00:34.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.550+0000 7fc477e00640 1 --2- 172.21.15.103:0/1269544341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 0x7fc4781089d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.550+0000 7fc477e00640 1 --2- 172.21.15.103:0/1269544341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc478101780 0x7fc478101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.550+0000 7fc477e00640 1 --2- 172.21.15.103:0/1269544341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc478100590 0x7fc4781009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.550+0000 7fc477e00640 1 -- 172.21.15.103:0/1269544341 >> 172.21.15.103:0/1269544341 conn(0x7fc4780fbb40 msgr2=0x7fc4780fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:34.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.551+0000 7fc477e00640 1 -- 172.21.15.103:0/1269544341 shutdown_connections 2024-05-23T00:00:34.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.551+0000 7fc477e00640 1 -- 172.21.15.103:0/1269544341 wait complete. 2024-05-23T00:00:34.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.552+0000 7fc477e00640 1 Processor -- start 2024-05-23T00:00:34.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.552+0000 7fc477e00640 1 -- start start 2024-05-23T00:00:34.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.552+0000 7fc477e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc478100590 0x7fc478199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:34.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.552+0000 7fc477e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc478101780 0x7fc47819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:34.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.552+0000 7fc477e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 0x7fc4781a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:34.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.552+0000 7fc477e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc47810c1f0 con 0x7fc478100590 2024-05-23T00:00:34.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.552+0000 7fc477e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc47810c070 con 0x7fc478102140 2024-05-23T00:00:34.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.552+0000 7fc477e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc47810c370 con 0x7fc478101780 2024-05-23T00:00:34.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc477400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 0x7fc4781a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:34.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc46fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc478101780 0x7fc47819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:34.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc476a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc478100590 0x7fc478199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:34.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc477400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 0x7fc4781a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49338/0 (socket says 172.21.15.103:49338) 2024-05-23T00:00:34.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc46fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc478101780 0x7fc47819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41458/0 (socket says 172.21.15.103:41458) 2024-05-23T00:00:34.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc477400640 1 -- 172.21.15.103:0/571516081 learned_addr learned my addr 172.21.15.103:0/571516081 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:34.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc46fe00640 1 -- 172.21.15.103:0/571516081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 msgr2=0x7fc4781a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc46fe00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 0x7fc4781a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc46fe00640 1 -- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc478100590 msgr2=0x7fc478199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc46fe00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc478100590 0x7fc478199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc46fe00640 1 -- 172.21.15.103:0/571516081 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4781a17a0 con 0x7fc478101780 2024-05-23T00:00:34.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.553+0000 7fc46fe00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc478101780 0x7fc47819a0c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc46000e9f0 tx=0x7fc46000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:34.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.554+0000 7fc46f400640 1 -- 172.21.15.103:0/571516081 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46000ccc0 con 0x7fc478101780 2024-05-23T00:00:34.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.554+0000 7fc46f400640 1 -- 172.21.15.103:0/571516081 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc460010910 con 0x7fc478101780 2024-05-23T00:00:34.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.554+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4781a1a90 con 0x7fc478101780 2024-05-23T00:00:34.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.554+0000 7fc46f400640 1 -- 172.21.15.103:0/571516081 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc460018960 con 0x7fc478101780 2024-05-23T00:00:34.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.554+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4781a1fe0 con 0x7fc478101780 2024-05-23T00:00:34.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.555+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4781a2270 con 0x7fc478101780 2024-05-23T00:00:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: [23/May/2024:00:00:33] ENGINE Bus STARTING 2024-05-23T00:00:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: [23/May/2024:00:00:33] ENGINE Serving on http://172.21.15.103:8765 2024-05-23T00:00:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/620887877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: mgrmap e23: x(active, since 1.02144s), standbys: z, y 2024-05-23T00:00:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/833603481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.557+0000 7fc46f400640 1 -- 172.21.15.103:0/571516081 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 23) ==== 137645+0+0 (secure 0 0 0) 0x7fc460010ac0 con 0x7fc478101780 2024-05-23T00:00:34.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.558+0000 7fc46f400640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc45c0a2990 0x7fc45c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:34.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.558+0000 7fc46f400640 1 -- 172.21.15.103:0/571516081 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc4600cf430 con 0x7fc478101780 2024-05-23T00:00:34.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.558+0000 7fc476a00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc45c0a2990 0x7fc45c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:34.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.559+0000 7fc476a00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc45c0a2990 0x7fc45c0a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc4781015e0 tx=0x7fc468005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:34.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: [23/May/2024:00:00:33] ENGINE Bus STARTING 2024-05-23T00:00:34.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: [23/May/2024:00:00:33] ENGINE Serving on http://172.21.15.103:8765 2024-05-23T00:00:34.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/620887877' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:34.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: mgrmap e23: x(active, since 1.02144s), standbys: z, y 2024-05-23T00:00:34.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/833603481' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:34.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:34.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.563+0000 7fc46f400640 1 -- 172.21.15.103:0/571516081 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc46009bbd0 con 0x7fc478101780 2024-05-23T00:00:34.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.784+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc4781a2600 con 0x7fc478101780 2024-05-23T00:00:34.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.784+0000 7fc46f400640 1 -- 172.21.15.103:0/571516081 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7fc460086020 con 0x7fc478101780 2024-05-23T00:00:34.787 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:34.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc45c0a2990 msgr2=0x7fc45c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc45c0a2990 0x7fc45c0a4e50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc4781015e0 tx=0x7fc468005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:34.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc478101780 msgr2=0x7fc47819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc478101780 0x7fc47819a0c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc46000e9f0 tx=0x7fc46000eeb0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 shutdown_connections 2024-05-23T00:00:34.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc45c0a2990 0x7fc45c0a4e50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc478102140 0x7fc4781a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc478101780 0x7fc47819a0c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 --2- 172.21.15.103:0/571516081 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc478100590 0x7fc478199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 >> 172.21.15.103:0/571516081 conn(0x7fc4780fbb40 msgr2=0x7fc4780fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:34.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 shutdown_connections 2024-05-23T00:00:34.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.787+0000 7fc477e00640 1 -- 172.21.15.103:0/571516081 wait complete. 2024-05-23T00:00:34.810 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-23T00:00:34.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 -- 172.21.15.103:0/3864205101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c010ddd0 msgr2=0x7f98c01101c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:34.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 --2- 172.21.15.103:0/3864205101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c010ddd0 0x7f98c01101c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98b8009be0 tx=0x7f98b802a6c0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 -- 172.21.15.103:0/3864205101 shutdown_connections 2024-05-23T00:00:34.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 --2- 172.21.15.103:0/3864205101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c010ddd0 0x7f98c01101c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 --2- 172.21.15.103:0/3864205101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c00742d0 0x7f98c0074750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 --2- 172.21.15.103:0/3864205101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c00758b0 0x7f98c0073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:34.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 -- 172.21.15.103:0/3864205101 >> 172.21.15.103:0/3864205101 conn(0x7f98c00fcd70 msgr2=0x7f98c00ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:34.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 -- 172.21.15.103:0/3864205101 shutdown_connections 2024-05-23T00:00:34.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.995+0000 7f98c7000640 1 -- 172.21.15.103:0/3864205101 wait complete. 2024-05-23T00:00:35.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.996+0000 7f98c7000640 1 Processor -- start 2024-05-23T00:00:35.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.996+0000 7f98c7000640 1 -- start start 2024-05-23T00:00:35.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c7000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c00742d0 0x7f98c019e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c7000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c00758b0 0x7f98c019e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c7000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c010ddd0 0x7f98c01a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c7000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98c0113990 con 0x7f98c010ddd0 2024-05-23T00:00:35.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c7000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f98c0113810 con 0x7f98c00742d0 2024-05-23T00:00:35.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c7000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f98c0113b10 con 0x7f98c00758b0 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c00758b0 0x7f98c019e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c00758b0 0x7f98c019e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41486/0 (socket says 172.21.15.103:41486) 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 -- 172.21.15.103:0/122701571 learned_addr learned my addr 172.21.15.103:0/122701571 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c6600640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c010ddd0 0x7f98c01a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5c00640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c00742d0 0x7f98c019e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 -- 172.21.15.103:0/122701571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c00742d0 msgr2=0x7f98c019e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c00742d0 0x7f98c019e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 -- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c010ddd0 msgr2=0x7f98c01a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c010ddd0 0x7f98c01a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 -- 172.21.15.103:0/122701571 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98c01a5c20 con 0x7f98c00758b0 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.997+0000 7f98c5200640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c00758b0 0x7f98c019e540 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00e9c0 tx=0x7f98ac00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.998+0000 7f98be000640 1 -- 172.21.15.103:0/122701571 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac00cde0 con 0x7f98c00758b0 2024-05-23T00:00:35.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.998+0000 7f98be000640 1 -- 172.21.15.103:0/122701571 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98ac0035f0 con 0x7f98c00758b0 2024-05-23T00:00:35.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.998+0000 7f98be000640 1 -- 172.21.15.103:0/122701571 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ac010690 con 0x7f98c00758b0 2024-05-23T00:00:35.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.998+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98c01a5f10 con 0x7f98c00758b0 2024-05-23T00:00:35.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.998+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98c01a6380 con 0x7f98c00758b0 2024-05-23T00:00:35.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:34.999+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98c0103fa0 con 0x7f98c00758b0 2024-05-23T00:00:35.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.002+0000 7f98be000640 1 -- 172.21.15.103:0/122701571 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 23) ==== 137645+0+0 (secure 0 0 0) 0x7f98ac0037a0 con 0x7f98c00758b0 2024-05-23T00:00:35.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.004+0000 7f98be000640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f98a80a2a10 0x7f98a80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.004+0000 7f98be000640 1 -- 172.21.15.103:0/122701571 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f98ac0cf290 con 0x7f98c00758b0 2024-05-23T00:00:35.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.004+0000 7f98c5c00640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f98a80a2a10 0x7f98a80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.004+0000 7f98be000640 1 -- 172.21.15.103:0/122701571 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f98ac09c040 con 0x7f98c00758b0 2024-05-23T00:00:35.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.005+0000 7f98c5c00640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f98a80a2a10 0x7f98a80a4ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009bd0 tx=0x7f98b4005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:35.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.194+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f98c002d470 con 0x7f98c00758b0 2024-05-23T00:00:35.213 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.210+0000 7f98be000640 1 -- 172.21.15.103:0/122701571 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mgrmap(e 24) ==== 137954+0+0 (secure 0 0 0) 0x7f98ac097fb0 con 0x7f98c00758b0 2024-05-23T00:00:35.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.211+0000 7f98be000640 1 -- 172.21.15.103:0/122701571 <== mon.1 v2:172.21.15.174:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f98ac0d1020 con 0x7f98c00758b0 2024-05-23T00:00:35.215 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-23T00:00:35.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f98a80a2a10 msgr2=0x7f98a80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.217 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f98a80a2a10 0x7f98a80a4ed0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009bd0 tx=0x7f98b4005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c00758b0 msgr2=0x7f98c019e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c00758b0 0x7f98c019e540 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00e9c0 tx=0x7f98ac00ee80 comp rx=0 tx=0).stop 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 shutdown_connections 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f98a80a2a10 0x7f98a80a4ed0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98c010ddd0 0x7f98c01a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f98c00758b0 0x7f98c019e540 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 --2- 172.21.15.103:0/122701571 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f98c00742d0 0x7f98c019e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 >> 172.21.15.103:0/122701571 conn(0x7f98c00fcd70 msgr2=0x7f98c0101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 shutdown_connections 2024-05-23T00:00:35.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.214+0000 7f98c7000640 1 -- 172.21.15.103:0/122701571 wait complete. 2024-05-23T00:00:35.234 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-23T00:00:35.425 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 -- 172.21.15.103:0/2534164412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 msgr2=0x7f2e7c10bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 --2- 172.21.15.103:0/2534164412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 0x7f2e7c10bcf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2e74009ca0 tx=0x7f2e7402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 -- 172.21.15.103:0/2534164412 shutdown_connections 2024-05-23T00:00:35.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 --2- 172.21.15.103:0/2534164412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 0x7f2e7c10bcf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 --2- 172.21.15.103:0/2534164412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e7c104aa0 0x7f2e7c104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 --2- 172.21.15.103:0/2534164412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e7c1038b0 0x7f2e7c103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 -- 172.21.15.103:0/2534164412 >> 172.21.15.103:0/2534164412 conn(0x7f2e7c0fee40 msgr2=0x7f2e7c101260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:35.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 -- 172.21.15.103:0/2534164412 shutdown_connections 2024-05-23T00:00:35.426 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.422+0000 7f2e82000640 1 -- 172.21.15.103:0/2534164412 wait complete. 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.423+0000 7f2e82000640 1 Processor -- start 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.423+0000 7f2e82000640 1 -- start start 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.423+0000 7f2e82000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e7c1038b0 0x7f2e7c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.423+0000 7f2e82000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e7c104aa0 0x7f2e7c19a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.423+0000 7f2e82000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 0x7f2e7c1a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.423+0000 7f2e82000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10f510 con 0x7f2e7c1038b0 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.423+0000 7f2e82000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10f390 con 0x7f2e7c104aa0 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.423+0000 7f2e82000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c10f690 con 0x7f2e7c105460 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e80c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e7c1038b0 0x7f2e7c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e80c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e7c1038b0 0x7f2e7c199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38620/0 (socket says 172.21.15.103:38620) 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e80c00640 1 -- 172.21.15.103:0/957718475 learned_addr learned my addr 172.21.15.103:0/957718475 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:35.429 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e81600640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 0x7f2e7c1a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e7be00640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e7c104aa0 0x7f2e7c19a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e80c00640 1 -- 172.21.15.103:0/957718475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 msgr2=0x7f2e7c1a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e80c00640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 0x7f2e7c1a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e80c00640 1 -- 172.21.15.103:0/957718475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e7c104aa0 msgr2=0x7f2e7c19a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e80c00640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e7c104aa0 0x7f2e7c19a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e80c00640 1 -- 172.21.15.103:0/957718475 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7c1a1720 con 0x7f2e7c1038b0 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e81600640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 0x7f2e7c1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.424+0000 7f2e7be00640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e7c104aa0 0x7f2e7c19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.425+0000 7f2e80c00640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e7c1038b0 0x7f2e7c199b00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2e7000e9c0 tx=0x7f2e7000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.425+0000 7f2e79600640 1 -- 172.21.15.103:0/957718475 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7000cd80 con 0x7f2e7c1038b0 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.425+0000 7f2e79600640 1 -- 172.21.15.103:0/957718475 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e700035f0 con 0x7f2e7c1038b0 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.425+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7c1a1a10 con 0x7f2e7c1038b0 2024-05-23T00:00:35.430 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.425+0000 7f2e79600640 1 -- 172.21.15.103:0/957718475 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e70010690 con 0x7f2e7c1038b0 2024-05-23T00:00:35.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.425+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7c1a1f60 con 0x7f2e7c1038b0 2024-05-23T00:00:35.431 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.426+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e7c068a00 con 0x7f2e7c1038b0 2024-05-23T00:00:35.432 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.429+0000 7f2e79600640 1 -- 172.21.15.103:0/957718475 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 24) ==== 137954+0+0 (secure 0 0 0) 0x7f2e700037a0 con 0x7f2e7c1038b0 2024-05-23T00:00:35.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.431+0000 7f2e79600640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2e640a2d00 0x7f2e640a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.431+0000 7f2e79600640 1 -- 172.21.15.103:0/957718475 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2e700cfc10 con 0x7f2e7c1038b0 2024-05-23T00:00:35.434 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.431+0000 7f2e7be00640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2e640a2d00 0x7f2e640a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.435 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.432+0000 7f2e7be00640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2e640a2d00 0x7f2e640a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e680095f0 tx=0x7f2e68005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:35.437 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.434+0000 7f2e79600640 1 -- 172.21.15.103:0/957718475 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2e7009c520 con 0x7f2e7c1038b0 2024-05-23T00:00:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:35 smithi105 ceph-mon[36426]: [23/May/2024:00:00:33] ENGINE Serving on https://172.21.15.103:7150 2024-05-23T00:00:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:35 smithi105 ceph-mon[36426]: [23/May/2024:00:00:33] ENGINE Bus STARTED 2024-05-23T00:00:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:35 smithi105 ceph-mon[36426]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:35 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3885839073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:35.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:35 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/571516081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:35.579 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:00:35 smithi103 sudo[86369]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:35.580 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:00:35 smithi103 sudo[86369]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:35.580 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:00:35 smithi103 sudo[86369]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:35.580 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:00:35 smithi103 sudo[86369]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:35.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:35 smithi103 ceph-mon[31370]: [23/May/2024:00:00:33] ENGINE Serving on https://172.21.15.103:7150 2024-05-23T00:00:35.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:35 smithi103 ceph-mon[31370]: [23/May/2024:00:00:33] ENGINE Bus STARTED 2024-05-23T00:00:35.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:35 smithi103 ceph-mon[31370]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:35.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:35 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3885839073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:35.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:35 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/571516081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:35.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:35 smithi174 ceph-mon[36179]: [23/May/2024:00:00:33] ENGINE Serving on https://172.21.15.103:7150 2024-05-23T00:00:35.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:35 smithi174 ceph-mon[36179]: [23/May/2024:00:00:33] ENGINE Bus STARTED 2024-05-23T00:00:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:35 smithi174 ceph-mon[36179]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:35 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3885839073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:35.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:35 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/571516081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:35.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.639+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2e7c02d280 con 0x7f2e7c1038b0 2024-05-23T00:00:35.643 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.640+0000 7f2e79600640 1 -- 172.21.15.103:0/957718475 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f2e7009be20 con 0x7f2e7c1038b0 2024-05-23T00:00:35.644 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:35.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.642+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2e640a2d00 msgr2=0x7f2e640a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.647 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.642+0000 7f2e82000640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2e640a2d00 0x7f2e640a51c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2e680095f0 tx=0x7f2e68005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:35.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e7c1038b0 msgr2=0x7f2e7c199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e7c1038b0 0x7f2e7c199b00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2e7000e9c0 tx=0x7f2e7000ee80 comp rx=0 tx=0).stop 2024-05-23T00:00:35.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 shutdown_connections 2024-05-23T00:00:35.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2e640a2d00 0x7f2e640a51c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e7c105460 0x7f2e7c1a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.648 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e7c104aa0 0x7f2e7c19a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 --2- 172.21.15.103:0/957718475 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e7c1038b0 0x7f2e7c199b00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 >> 172.21.15.103:0/957718475 conn(0x7f2e7c0fee40 msgr2=0x7f2e7c107da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:35.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 shutdown_connections 2024-05-23T00:00:35.649 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.643+0000 7f2e82000640 1 -- 172.21.15.103:0/957718475 wait complete. 2024-05-23T00:00:35.665 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-23T00:00:35.665 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:00:35.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 -- 172.21.15.103:0/2763154186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18105680 msgr2=0x7f7b1810bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/2763154186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18105680 0x7f7b1810bf10 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7b10009ca0 tx=0x7f7b1002a8c0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 -- 172.21.15.103:0/2763154186 shutdown_connections 2024-05-23T00:00:35.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/2763154186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18105680 0x7f7b1810bf10 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/2763154186 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18104cc0 0x7f7b18105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/2763154186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b18103ad0 0x7f7b18103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 -- 172.21.15.103:0/2763154186 >> 172.21.15.103:0/2763154186 conn(0x7f7b180ff060 msgr2=0x7f7b18101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:35.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 -- 172.21.15.103:0/2763154186 shutdown_connections 2024-05-23T00:00:35.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.846+0000 7f7b1fc00640 1 -- 172.21.15.103:0/2763154186 wait complete. 2024-05-23T00:00:35.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.848+0000 7f7b1fc00640 1 Processor -- start 2024-05-23T00:00:35.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.848+0000 7f7b1fc00640 1 -- start start 2024-05-23T00:00:35.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.848+0000 7f7b1fc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18103ad0 0x7f7b18199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.848+0000 7f7b1fc00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b18104cc0 0x7f7b1819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.848+0000 7f7b1fc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18105680 0x7f7b181a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.848+0000 7f7b1fc00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1810f780 con 0x7f7b18105680 2024-05-23T00:00:35.856 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.848+0000 7f7b1fc00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1810f600 con 0x7f7b18103ad0 2024-05-23T00:00:35.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.848+0000 7f7b1fc00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b1810f900 con 0x7f7b18104cc0 2024-05-23T00:00:35.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18103ad0 0x7f7b18199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18103ad0 0x7f7b18199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49404/0 (socket says 172.21.15.103:49404) 2024-05-23T00:00:35.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 -- 172.21.15.103:0/3694249131 learned_addr learned my addr 172.21.15.103:0/3694249131 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:35.857 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b17e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b18104cc0 0x7f7b1819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1dc00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18105680 0x7f7b181a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 -- 172.21.15.103:0/3694249131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b18104cc0 msgr2=0x7f7b1819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b18104cc0 0x7f7b1819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 -- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18105680 msgr2=0x7f7b181a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:35.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18105680 0x7f7b181a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:35.858 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 -- 172.21.15.103:0/3694249131 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b181a1980 con 0x7f7b18103ad0 2024-05-23T00:00:35.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b17e00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b18104cc0 0x7f7b1819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:35.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1dc00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18105680 0x7f7b181a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:35.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b1d200640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18103ad0 0x7f7b18199d60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00e9c0 tx=0x7f7b0c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:35.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b15600640 1 -- 172.21.15.103:0/3694249131 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c00cd80 con 0x7f7b18103ad0 2024-05-23T00:00:35.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.849+0000 7f7b15600640 1 -- 172.21.15.103:0/3694249131 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b0c0035f0 con 0x7f7b18103ad0 2024-05-23T00:00:35.859 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.850+0000 7f7b15600640 1 -- 172.21.15.103:0/3694249131 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c010690 con 0x7f7b18103ad0 2024-05-23T00:00:35.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.850+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b181a1c70 con 0x7f7b18103ad0 2024-05-23T00:00:35.860 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.850+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b181a2170 con 0x7f7b18103ad0 2024-05-23T00:00:35.861 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.851+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b1810ad90 con 0x7f7b18103ad0 2024-05-23T00:00:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.854+0000 7f7b15600640 1 -- 172.21.15.103:0/3694249131 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 137954+0+0 (secure 0 0 0) 0x7f7b0c0037a0 con 0x7f7b18103ad0 2024-05-23T00:00:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.855+0000 7f7b15600640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7b000a2d80 0x7f7b000a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:35.862 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.855+0000 7f7b15600640 1 -- 172.21.15.103:0/3694249131 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f7b0c0cfba0 con 0x7f7b18103ad0 2024-05-23T00:00:35.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.855+0000 7f7b17e00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7b000a2d80 0x7f7b000a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:35.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.856+0000 7f7b17e00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7b000a2d80 0x7f7b000a5240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b040095f0 tx=0x7f7b04005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:35.863 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:35.859+0000 7f7b15600640 1 -- 172.21.15.103:0/3694249131 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7b0c09c480 con 0x7f7b18103ad0 2024-05-23T00:00:36.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.135+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b1802d200 con 0x7f7b18103ad0 2024-05-23T00:00:36.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.136+0000 7f7b15600640 1 -- 172.21.15.103:0/3694249131 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11896 (secure 0 0 0) 0x7f7b0c09bd80 con 0x7f7b18103ad0 2024-05-23T00:00:36.140 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:36.140 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-23T00:00:31.011025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24559,24563]},"id":1}]} 2024-05-23T00:00:36.140 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 25 2024-05-23T00:00:36.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7b000a2d80 msgr2=0x7f7b000a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7b000a2d80 0x7f7b000a5240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b040095f0 tx=0x7f7b04005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18103ad0 msgr2=0x7f7b18199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18103ad0 0x7f7b18199d60 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00e9c0 tx=0x7f7b0c00ee80 comp rx=0 tx=0).stop 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 shutdown_connections 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7b000a2d80 0x7f7b000a5240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b18105680 0x7f7b181a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b18104cc0 0x7f7b1819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 --2- 172.21.15.103:0/3694249131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b18103ad0 0x7f7b18199d60 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 >> 172.21.15.103:0/3694249131 conn(0x7f7b180ff060 msgr2=0x7f7b18107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 shutdown_connections 2024-05-23T00:00:36.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.139+0000 7f7b1fc00640 1 -- 172.21.15.103:0/3694249131 wait complete. 2024-05-23T00:00:36.161 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-23T00:00:36.161 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-23T00:00:36.161 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-23T00:00:36.329 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:00:36 smithi103 sudo[86488]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:36.330 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:00:36 smithi103 sudo[86488]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:36.330 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:00:36 smithi103 sudo[86488]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:36.330 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:00:36 smithi103 sudo[86488]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:36.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:36.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-23T00:00:36.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/122701571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:36.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/957718475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:36.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:36 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3694249131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:36.350 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 -- 172.21.15.103:0/3968412329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c101710 msgr2=0x7f9f2c101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 --2- 172.21.15.103:0/3968412329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c101710 0x7f9f2c101b30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009860 tx=0x7f9f2002a340 comp rx=0 tx=0).stop 2024-05-23T00:00:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 -- 172.21.15.103:0/3968412329 shutdown_connections 2024-05-23T00:00:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 --2- 172.21.15.103:0/3968412329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f2c1032c0 0x7f9f2c109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 --2- 172.21.15.103:0/3968412329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f2c102900 0x7f9f2c102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 --2- 172.21.15.103:0/3968412329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c101710 0x7f9f2c101b30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 -- 172.21.15.103:0/3968412329 >> 172.21.15.103:0/3968412329 conn(0x7f9f2c0fcc80 msgr2=0x7f9f2c0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 -- 172.21.15.103:0/3968412329 shutdown_connections 2024-05-23T00:00:36.351 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.347+0000 7f9f32400640 1 -- 172.21.15.103:0/3968412329 wait complete. 2024-05-23T00:00:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.348+0000 7f9f32400640 1 Processor -- start 2024-05-23T00:00:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.348+0000 7f9f32400640 1 -- start start 2024-05-23T00:00:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.348+0000 7f9f32400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f2c101710 0x7f9f2c075620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.348+0000 7f9f32400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f2c102900 0x7f9f2c073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.348+0000 7f9f32400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c1032c0 0x7f9f2c0741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.348+0000 7f9f32400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2c10d390 con 0x7f9f2c102900 2024-05-23T00:00:36.353 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.348+0000 7f9f32400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2c10d210 con 0x7f9f2c101710 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.348+0000 7f9f32400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f2c10d510 con 0x7f9f2c1032c0 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f2c101710 0x7f9f2c075620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f2be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f2c102900 0x7f9f2c073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c1032c0 0x7f9f2c0741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f2c101710 0x7f9f2c075620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49420/0 (socket says 172.21.15.103:49420) 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 -- 172.21.15.103:0/649943328 learned_addr learned my addr 172.21.15.103:0/649943328 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 -- 172.21.15.103:0/649943328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c1032c0 msgr2=0x7f9f2c0741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c1032c0 0x7f9f2c0741a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 -- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f2c102900 msgr2=0x7f9f2c073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f2c102900 0x7f9f2c073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 -- 172.21.15.103:0/649943328 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f2c074a30 con 0x7f9f2c101710 2024-05-23T00:00:36.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31a00640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c1032c0 0x7f9f2c0741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f2be00640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f2c102900 0x7f9f2c073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.349+0000 7f9f31000640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f2c101710 0x7f9f2c075620 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9f20035040 tx=0x7f9f20033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.350+0000 7f9f29600640 1 -- 172.21.15.103:0/649943328 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20037040 con 0x7f9f2c101710 2024-05-23T00:00:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.350+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f2c1a1ac0 con 0x7f9f2c101710 2024-05-23T00:00:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.350+0000 7f9f29600640 1 -- 172.21.15.103:0/649943328 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9f20030850 con 0x7f9f2c101710 2024-05-23T00:00:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.350+0000 7f9f29600640 1 -- 172.21.15.103:0/649943328 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f20004920 con 0x7f9f2c101710 2024-05-23T00:00:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.350+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f2c1a2010 con 0x7f9f2c101710 2024-05-23T00:00:36.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.351+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f2c108a30 con 0x7f9f2c101710 2024-05-23T00:00:36.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.354+0000 7f9f29600640 1 -- 172.21.15.103:0/649943328 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 137954+0+0 (secure 0 0 0) 0x7f9f20004050 con 0x7f9f2c101710 2024-05-23T00:00:36.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.356+0000 7f9f29600640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9f140a2d80 0x7f9f140a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:36.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.356+0000 7f9f29600640 1 -- 172.21.15.103:0/649943328 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9f200ef860 con 0x7f9f2c101710 2024-05-23T00:00:36.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.357+0000 7f9f2be00640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9f140a2d80 0x7f9f140a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:36.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.357+0000 7f9f2be00640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9f140a2d80 0x7f9f140a5240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9f180096c0 tx=0x7f9f18009260 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:36.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.360+0000 7f9f29600640 1 -- 172.21.15.103:0/649943328 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f9f200bc1f0 con 0x7f9f2c101710 2024-05-23T00:00:36.558 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:00:36 smithi105 sudo[69015]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:36.558 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:00:36 smithi105 sudo[69015]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:36.558 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:00:36 smithi105 sudo[69015]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:36.558 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:00:36 smithi105 sudo[69015]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/122701571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/957718475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:36 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3694249131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:36.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.575+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f2c061cb0 con 0x7f9f140a2d80 2024-05-23T00:00:36.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.588+0000 7f9f29600640 1 -- 172.21.15.103:0/649943328 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339719 (secure 0 0 0) 0x7f9f2c061cb0 con 0x7f9f140a2d80 2024-05-23T00:00:36.596 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:36.598 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-05-23T00:00:36.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.593+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9f140a2d80 msgr2=0x7f9f140a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.593+0000 7f9f32400640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9f140a2d80 0x7f9f140a5240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9f180096c0 tx=0x7f9f18009260 comp rx=0 tx=0).stop 2024-05-23T00:00:36.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.593+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f2c101710 msgr2=0x7f9f2c075620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.593+0000 7f9f32400640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f2c101710 0x7f9f2c075620 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9f20035040 tx=0x7f9f20033d90 comp rx=0 tx=0).stop 2024-05-23T00:00:36.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.594+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 shutdown_connections 2024-05-23T00:00:36.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.594+0000 7f9f32400640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9f140a2d80 0x7f9f140a5240 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.594+0000 7f9f32400640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f2c1032c0 0x7f9f2c0741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.594+0000 7f9f32400640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9f2c102900 0x7f9f2c073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.594+0000 7f9f32400640 1 --2- 172.21.15.103:0/649943328 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f2c101710 0x7f9f2c075620 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.594+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 >> 172.21.15.103:0/649943328 conn(0x7f9f2c0fcc80 msgr2=0x7f9f2c105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:36.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.594+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 shutdown_connections 2024-05-23T00:00:36.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.594+0000 7f9f32400640 1 -- 172.21.15.103:0/649943328 wait complete. 2024-05-23T00:00:36.617 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":4,"stamp":"2024-05-23T00:00:34.692618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190223,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1154,"num_read_kb":741,"num_write":478,"num_write_kb":4388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1181748,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":873,"ondisk_log_size":873,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357572,"kb_used_data":12556,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584252,"statfs":{"total":1151940427776,"available":1151574274048,"internally_reserved":0,"allocated":12857344,"data_stored":9055355,"data_compressed":424676,"data_compressed_allocated":577536,"data_compressed_original":3448832,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.990804"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810114+0000","last_change":"2024-05-22T23:57:33.328075+0000","last_active":"2024-05-23T00:00:32.810114+0000","last_peered":"2024-05-23T00:00:32.810114+0000","last_clean":"2024-05-23T00:00:32.810114+0000","last_became_active":"2024-05-22T23:57:33.327704+0000","last_became_peered":"2024-05-22T23:57:33.327704+0000","last_unstale":"2024-05-23T00:00:32.810114+0000","last_undegraded":"2024-05-23T00:00:32.810114+0000","last_fullsized":"2024-05-23T00:00:32.810114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:42:32.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670793+0000","last_change":"2024-05-22T23:57:33.332043+0000","last_active":"2024-05-23T00:00:32.670793+0000","last_peered":"2024-05-23T00:00:32.670793+0000","last_clean":"2024-05-23T00:00:32.670793+0000","last_became_active":"2024-05-22T23:57:33.322269+0000","last_became_peered":"2024-05-22T23:57:33.322269+0000","last_unstale":"2024-05-23T00:00:32.670793+0000","last_undegraded":"2024-05-23T00:00:32.670793+0000","last_fullsized":"2024-05-23T00:00:32.670793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:40:23.542938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670609+0000","last_change":"2024-05-22T23:57:33.334993+0000","last_active":"2024-05-23T00:00:32.670609+0000","last_peered":"2024-05-23T00:00:32.670609+0000","last_clean":"2024-05-23T00:00:32.670609+0000","last_became_active":"2024-05-22T23:57:33.326446+0000","last_became_peered":"2024-05-22T23:57:33.326446+0000","last_unstale":"2024-05-23T00:00:32.670609+0000","last_undegraded":"2024-05-23T00:00:32.670609+0000","last_fullsized":"2024-05-23T00:00:32.670609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:44:45.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023391+0000","last_change":"2024-05-22T23:57:33.320181+0000","last_active":"2024-05-23T00:00:33.023391+0000","last_peered":"2024-05-23T00:00:33.023391+0000","last_clean":"2024-05-23T00:00:33.023391+0000","last_became_active":"2024-05-22T23:57:33.319630+0000","last_became_peered":"2024-05-22T23:57:33.319630+0000","last_unstale":"2024-05-23T00:00:33.023391+0000","last_undegraded":"2024-05-23T00:00:33.023391+0000","last_fullsized":"2024-05-23T00:00:33.023391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:38:51.836241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022138+0000","last_change":"2024-05-22T23:57:33.332168+0000","last_active":"2024-05-23T00:00:33.022138+0000","last_peered":"2024-05-23T00:00:33.022138+0000","last_clean":"2024-05-23T00:00:33.022138+0000","last_became_active":"2024-05-22T23:57:33.331755+0000","last_became_peered":"2024-05-22T23:57:33.331755+0000","last_unstale":"2024-05-23T00:00:33.022138+0000","last_undegraded":"2024-05-23T00:00:33.022138+0000","last_fullsized":"2024-05-23T00:00:33.022138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:28.013365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822749+0000","last_change":"2024-05-22T23:57:33.322449+0000","last_active":"2024-05-23T00:00:32.822749+0000","last_peered":"2024-05-23T00:00:32.822749+0000","last_clean":"2024-05-23T00:00:32.822749+0000","last_became_active":"2024-05-22T23:57:33.322099+0000","last_became_peered":"2024-05-22T23:57:33.322099+0000","last_unstale":"2024-05-23T00:00:32.822749+0000","last_undegraded":"2024-05-23T00:00:32.822749+0000","last_fullsized":"2024-05-23T00:00:32.822749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:38.427434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023339+0000","last_change":"2024-05-22T23:57:33.322391+0000","last_active":"2024-05-23T00:00:33.023339+0000","last_peered":"2024-05-23T00:00:33.023339+0000","last_clean":"2024-05-23T00:00:33.023339+0000","last_became_active":"2024-05-22T23:57:33.322076+0000","last_became_peered":"2024-05-22T23:57:33.322076+0000","last_unstale":"2024-05-23T00:00:33.023339+0000","last_undegraded":"2024-05-23T00:00:33.023339+0000","last_fullsized":"2024-05-23T00:00:33.023339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:45:44.857586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810020+0000","last_change":"2024-05-22T23:57:33.333005+0000","last_active":"2024-05-23T00:00:32.810020+0000","last_peered":"2024-05-23T00:00:32.810020+0000","last_clean":"2024-05-23T00:00:32.810020+0000","last_became_active":"2024-05-22T23:57:33.332669+0000","last_became_peered":"2024-05-22T23:57:33.332669+0000","last_unstale":"2024-05-23T00:00:32.810020+0000","last_undegraded":"2024-05-23T00:00:32.810020+0000","last_fullsized":"2024-05-23T00:00:32.810020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:47:25.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021594+0000","last_change":"2024-05-22T23:58:31.104194+0000","last_active":"2024-05-23T00:00:33.021594+0000","last_peered":"2024-05-23T00:00:33.021594+0000","last_clean":"2024-05-23T00:00:33.021594+0000","last_became_active":"2024-05-22T23:58:31.103788+0000","last_became_peered":"2024-05-22T23:58:31.103788+0000","last_unstale":"2024-05-23T00:00:33.021594+0000","last_undegraded":"2024-05-23T00:00:33.021594+0000","last_fullsized":"2024-05-23T00:00:33.021594+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:41:09.215669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822962+0000","last_change":"2024-05-22T23:57:33.324445+0000","last_active":"2024-05-23T00:00:32.822962+0000","last_peered":"2024-05-23T00:00:32.822962+0000","last_clean":"2024-05-23T00:00:32.822962+0000","last_became_active":"2024-05-22T23:57:33.324085+0000","last_became_peered":"2024-05-22T23:57:33.324085+0000","last_unstale":"2024-05-23T00:00:32.822962+0000","last_undegraded":"2024-05-23T00:00:32.822962+0000","last_fullsized":"2024-05-23T00:00:32.822962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:30:51.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024137+0000","last_change":"2024-05-22T23:57:33.321481+0000","last_active":"2024-05-23T00:00:33.024137+0000","last_peered":"2024-05-23T00:00:33.024137+0000","last_clean":"2024-05-23T00:00:33.024137+0000","last_became_active":"2024-05-22T23:57:33.320911+0000","last_became_peered":"2024-05-22T23:57:33.320911+0000","last_unstale":"2024-05-23T00:00:33.024137+0000","last_undegraded":"2024-05-23T00:00:33.024137+0000","last_fullsized":"2024-05-23T00:00:33.024137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:24:24.041980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023362+0000","last_change":"2024-05-22T23:57:33.318819+0000","last_active":"2024-05-23T00:00:33.023362+0000","last_peered":"2024-05-23T00:00:33.023362+0000","last_clean":"2024-05-23T00:00:33.023362+0000","last_became_active":"2024-05-22T23:57:33.318284+0000","last_became_peered":"2024-05-22T23:57:33.318284+0000","last_unstale":"2024-05-23T00:00:33.023362+0000","last_undegraded":"2024-05-23T00:00:33.023362+0000","last_fullsized":"2024-05-23T00:00:33.023362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:56.043446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024014+0000","last_change":"2024-05-22T23:57:33.321268+0000","last_active":"2024-05-23T00:00:33.024014+0000","last_peered":"2024-05-23T00:00:33.024014+0000","last_clean":"2024-05-23T00:00:33.024014+0000","last_became_active":"2024-05-22T23:57:33.320902+0000","last_became_peered":"2024-05-22T23:57:33.320902+0000","last_unstale":"2024-05-23T00:00:33.024014+0000","last_undegraded":"2024-05-23T00:00:33.024014+0000","last_fullsized":"2024-05-23T00:00:33.024014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:51:39.716541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669962+0000","last_change":"2024-05-22T23:57:33.322638+0000","last_active":"2024-05-23T00:00:32.669962+0000","last_peered":"2024-05-23T00:00:32.669962+0000","last_clean":"2024-05-23T00:00:32.669962+0000","last_became_active":"2024-05-22T23:57:33.322318+0000","last_became_peered":"2024-05-22T23:57:33.322318+0000","last_unstale":"2024-05-23T00:00:32.669962+0000","last_undegraded":"2024-05-23T00:00:32.669962+0000","last_fullsized":"2024-05-23T00:00:32.669962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:44:25.487618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669867+0000","last_change":"2024-05-22T23:57:33.325832+0000","last_active":"2024-05-23T00:00:32.669867+0000","last_peered":"2024-05-23T00:00:32.669867+0000","last_clean":"2024-05-23T00:00:32.669867+0000","last_became_active":"2024-05-22T23:57:33.325062+0000","last_became_peered":"2024-05-22T23:57:33.325062+0000","last_unstale":"2024-05-23T00:00:32.669867+0000","last_undegraded":"2024-05-23T00:00:32.669867+0000","last_fullsized":"2024-05-23T00:00:32.669867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:33:04.161351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020743+0000","last_change":"2024-05-22T23:57:33.324864+0000","last_active":"2024-05-23T00:00:33.020743+0000","last_peered":"2024-05-23T00:00:33.020743+0000","last_clean":"2024-05-23T00:00:33.020743+0000","last_became_active":"2024-05-22T23:57:33.324558+0000","last_became_peered":"2024-05-22T23:57:33.324558+0000","last_unstale":"2024-05-23T00:00:33.020743+0000","last_undegraded":"2024-05-23T00:00:33.020743+0000","last_fullsized":"2024-05-23T00:00:33.020743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:48:49.722371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809904+0000","last_change":"2024-05-22T23:57:33.337985+0000","last_active":"2024-05-23T00:00:32.809904+0000","last_peered":"2024-05-23T00:00:32.809904+0000","last_clean":"2024-05-23T00:00:32.809904+0000","last_became_active":"2024-05-22T23:57:33.337719+0000","last_became_peered":"2024-05-22T23:57:33.337719+0000","last_unstale":"2024-05-23T00:00:32.809904+0000","last_undegraded":"2024-05-23T00:00:32.809904+0000","last_fullsized":"2024-05-23T00:00:32.809904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:50:44.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.673585+0000","last_change":"2024-05-22T23:57:33.325709+0000","last_active":"2024-05-23T00:00:32.673585+0000","last_peered":"2024-05-23T00:00:32.673585+0000","last_clean":"2024-05-23T00:00:32.673585+0000","last_became_active":"2024-05-22T23:57:33.325063+0000","last_became_peered":"2024-05-22T23:57:33.325063+0000","last_unstale":"2024-05-23T00:00:32.673585+0000","last_undegraded":"2024-05-23T00:00:32.673585+0000","last_fullsized":"2024-05-23T00:00:32.673585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:39.145217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670219+0000","last_change":"2024-05-22T23:57:33.333251+0000","last_active":"2024-05-23T00:00:32.670219+0000","last_peered":"2024-05-23T00:00:32.670219+0000","last_clean":"2024-05-23T00:00:32.670219+0000","last_became_active":"2024-05-22T23:57:33.332894+0000","last_became_peered":"2024-05-22T23:57:33.332894+0000","last_unstale":"2024-05-23T00:00:32.670219+0000","last_undegraded":"2024-05-23T00:00:32.670219+0000","last_fullsized":"2024-05-23T00:00:32.670219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:13:12.082781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669594+0000","last_change":"2024-05-22T23:57:37.788060+0000","last_active":"2024-05-23T00:00:32.669594+0000","last_peered":"2024-05-23T00:00:32.669594+0000","last_clean":"2024-05-23T00:00:32.669594+0000","last_became_active":"2024-05-22T23:57:37.787656+0000","last_became_peered":"2024-05-22T23:57:37.787656+0000","last_unstale":"2024-05-23T00:00:32.669594+0000","last_undegraded":"2024-05-23T00:00:32.669594+0000","last_fullsized":"2024-05-23T00:00:32.669594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:39:13.718817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672586+0000","last_change":"2024-05-22T23:57:32.323434+0000","last_active":"2024-05-23T00:00:32.672586+0000","last_peered":"2024-05-23T00:00:32.672586+0000","last_clean":"2024-05-23T00:00:32.672586+0000","last_became_active":"2024-05-22T23:57:32.323033+0000","last_became_peered":"2024-05-22T23:57:32.323033+0000","last_unstale":"2024-05-23T00:00:32.672586+0000","last_undegraded":"2024-05-23T00:00:32.672586+0000","last_fullsized":"2024-05-23T00:00:32.672586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:15:27.676027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812037+0000","last_change":"2024-05-22T23:57:33.323336+0000","last_active":"2024-05-23T00:00:32.812037+0000","last_peered":"2024-05-23T00:00:32.812037+0000","last_clean":"2024-05-23T00:00:32.812037+0000","last_became_active":"2024-05-22T23:57:33.323026+0000","last_became_peered":"2024-05-22T23:57:33.323026+0000","last_unstale":"2024-05-23T00:00:32.812037+0000","last_undegraded":"2024-05-23T00:00:32.812037+0000","last_fullsized":"2024-05-23T00:00:32.812037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:02:20.142760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022462+0000","last_change":"2024-05-22T23:57:37.785137+0000","last_active":"2024-05-23T00:00:33.022462+0000","last_peered":"2024-05-23T00:00:33.022462+0000","last_clean":"2024-05-23T00:00:33.022462+0000","last_became_active":"2024-05-22T23:57:37.784791+0000","last_became_peered":"2024-05-22T23:57:37.784791+0000","last_unstale":"2024-05-23T00:00:33.022462+0000","last_undegraded":"2024-05-23T00:00:33.022462+0000","last_fullsized":"2024-05-23T00:00:33.022462+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:54:51.036334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812106+0000","last_change":"2024-05-22T23:57:32.332969+0000","last_active":"2024-05-23T00:00:32.812106+0000","last_peered":"2024-05-23T00:00:32.812106+0000","last_clean":"2024-05-23T00:00:32.812106+0000","last_became_active":"2024-05-22T23:57:32.332611+0000","last_became_peered":"2024-05-22T23:57:32.332611+0000","last_unstale":"2024-05-23T00:00:32.812106+0000","last_undegraded":"2024-05-23T00:00:32.812106+0000","last_fullsized":"2024-05-23T00:00:32.812106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:00:22.006355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670839+0000","last_change":"2024-05-22T23:57:33.332151+0000","last_active":"2024-05-23T00:00:32.670839+0000","last_peered":"2024-05-23T00:00:32.670839+0000","last_clean":"2024-05-23T00:00:32.670839+0000","last_became_active":"2024-05-22T23:57:33.329518+0000","last_became_peered":"2024-05-22T23:57:33.329518+0000","last_unstale":"2024-05-23T00:00:32.670839+0000","last_undegraded":"2024-05-23T00:00:32.670839+0000","last_fullsized":"2024-05-23T00:00:32.670839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:10:12.083766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022602+0000","last_change":"2024-05-22T23:57:37.787524+0000","last_active":"2024-05-23T00:00:33.022602+0000","last_peered":"2024-05-23T00:00:33.022602+0000","last_clean":"2024-05-23T00:00:33.022602+0000","last_became_active":"2024-05-22T23:57:37.787103+0000","last_became_peered":"2024-05-22T23:57:37.787103+0000","last_unstale":"2024-05-23T00:00:33.022602+0000","last_undegraded":"2024-05-23T00:00:33.022602+0000","last_fullsized":"2024-05-23T00:00:33.022602+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:19:16.963159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022671+0000","last_change":"2024-05-22T23:57:32.318469+0000","last_active":"2024-05-23T00:00:33.022671+0000","last_peered":"2024-05-23T00:00:33.022671+0000","last_clean":"2024-05-23T00:00:33.022671+0000","last_became_active":"2024-05-22T23:57:32.317640+0000","last_became_peered":"2024-05-22T23:57:32.317640+0000","last_unstale":"2024-05-23T00:00:33.022671+0000","last_undegraded":"2024-05-23T00:00:33.022671+0000","last_fullsized":"2024-05-23T00:00:33.022671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:01:55.710019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021134+0000","last_change":"2024-05-22T23:57:33.318604+0000","last_active":"2024-05-23T00:00:33.021134+0000","last_peered":"2024-05-23T00:00:33.021134+0000","last_clean":"2024-05-23T00:00:33.021134+0000","last_became_active":"2024-05-22T23:57:33.318184+0000","last_became_peered":"2024-05-22T23:57:33.318184+0000","last_unstale":"2024-05-23T00:00:33.021134+0000","last_undegraded":"2024-05-23T00:00:33.021134+0000","last_fullsized":"2024-05-23T00:00:33.021134+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:16.140799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022571+0000","last_change":"2024-05-22T23:57:37.796085+0000","last_active":"2024-05-23T00:00:33.022571+0000","last_peered":"2024-05-23T00:00:33.022571+0000","last_clean":"2024-05-23T00:00:33.022571+0000","last_became_active":"2024-05-22T23:57:37.795265+0000","last_became_peered":"2024-05-22T23:57:37.795265+0000","last_unstale":"2024-05-23T00:00:33.022571+0000","last_undegraded":"2024-05-23T00:00:33.022571+0000","last_fullsized":"2024-05-23T00:00:33.022571+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:43:55.577072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022541+0000","last_change":"2024-05-22T23:57:32.316346+0000","last_active":"2024-05-23T00:00:33.022541+0000","last_peered":"2024-05-23T00:00:33.022541+0000","last_clean":"2024-05-23T00:00:33.022541+0000","last_became_active":"2024-05-22T23:57:32.316038+0000","last_became_peered":"2024-05-22T23:57:32.316038+0000","last_unstale":"2024-05-23T00:00:33.022541+0000","last_undegraded":"2024-05-23T00:00:33.022541+0000","last_fullsized":"2024-05-23T00:00:33.022541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:57:41.987250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819205+0000","last_change":"2024-05-22T23:57:33.329341+0000","last_active":"2024-05-23T00:00:32.819205+0000","last_peered":"2024-05-23T00:00:32.819205+0000","last_clean":"2024-05-23T00:00:32.819205+0000","last_became_active":"2024-05-22T23:57:33.329009+0000","last_became_peered":"2024-05-22T23:57:33.329009+0000","last_unstale":"2024-05-23T00:00:32.819205+0000","last_undegraded":"2024-05-23T00:00:32.819205+0000","last_fullsized":"2024-05-23T00:00:32.819205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:53:07.591497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670094+0000","last_change":"2024-05-22T23:57:37.779097+0000","last_active":"2024-05-23T00:00:32.670094+0000","last_peered":"2024-05-23T00:00:32.670094+0000","last_clean":"2024-05-23T00:00:32.670094+0000","last_became_active":"2024-05-22T23:57:37.778564+0000","last_became_peered":"2024-05-22T23:57:37.778564+0000","last_unstale":"2024-05-23T00:00:32.670094+0000","last_undegraded":"2024-05-23T00:00:32.670094+0000","last_fullsized":"2024-05-23T00:00:32.670094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:09:16.351947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":205,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.298705+0000","last_change":"2024-05-22T23:58:31.109013+0000","last_active":"2024-05-23T00:00:33.298705+0000","last_peered":"2024-05-23T00:00:33.298705+0000","last_clean":"2024-05-23T00:00:33.298705+0000","last_became_active":"2024-05-22T23:58:31.103858+0000","last_became_peered":"2024-05-22T23:58:31.103858+0000","last_unstale":"2024-05-23T00:00:33.298705+0000","last_undegraded":"2024-05-23T00:00:33.298705+0000","last_fullsized":"2024-05-23T00:00:33.298705+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:20:08.783189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":153,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022157+0000","last_change":"2024-05-22T23:57:33.324587+0000","last_active":"2024-05-23T00:00:33.022157+0000","last_peered":"2024-05-23T00:00:33.022157+0000","last_clean":"2024-05-23T00:00:33.022157+0000","last_became_active":"2024-05-22T23:57:33.323972+0000","last_became_peered":"2024-05-22T23:57:33.323972+0000","last_unstale":"2024-05-23T00:00:33.022157+0000","last_undegraded":"2024-05-23T00:00:33.022157+0000","last_fullsized":"2024-05-23T00:00:33.022157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:23:52.551699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670431+0000","last_change":"2024-05-22T23:57:37.791726+0000","last_active":"2024-05-23T00:00:32.670431+0000","last_peered":"2024-05-23T00:00:32.670431+0000","last_clean":"2024-05-23T00:00:32.670431+0000","last_became_active":"2024-05-22T23:57:37.791185+0000","last_became_peered":"2024-05-22T23:57:37.791185+0000","last_unstale":"2024-05-23T00:00:32.670431+0000","last_undegraded":"2024-05-23T00:00:32.670431+0000","last_fullsized":"2024-05-23T00:00:32.670431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:55:30.994272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822489+0000","last_change":"2024-05-22T23:58:31.116594+0000","last_active":"2024-05-23T00:00:32.822489+0000","last_peered":"2024-05-23T00:00:32.822489+0000","last_clean":"2024-05-23T00:00:32.822489+0000","last_became_active":"2024-05-22T23:58:31.113117+0000","last_became_peered":"2024-05-22T23:58:31.113117+0000","last_unstale":"2024-05-23T00:00:32.822489+0000","last_undegraded":"2024-05-23T00:00:32.822489+0000","last_fullsized":"2024-05-23T00:00:32.822489+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:05:30.758638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819100+0000","last_change":"2024-05-22T23:58:31.116203+0000","last_active":"2024-05-23T00:00:32.819100+0000","last_peered":"2024-05-23T00:00:32.819100+0000","last_clean":"2024-05-23T00:00:32.819100+0000","last_became_active":"2024-05-22T23:58:31.115881+0000","last_became_peered":"2024-05-22T23:58:31.115881+0000","last_unstale":"2024-05-23T00:00:32.819100+0000","last_undegraded":"2024-05-23T00:00:32.819100+0000","last_fullsized":"2024-05-23T00:00:32.819100+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:39:31.247426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809889+0000","last_change":"2024-05-22T23:57:37.787910+0000","last_active":"2024-05-23T00:00:32.809889+0000","last_peered":"2024-05-23T00:00:32.809889+0000","last_clean":"2024-05-23T00:00:32.809889+0000","last_became_active":"2024-05-22T23:57:37.787465+0000","last_became_peered":"2024-05-22T23:57:37.787465+0000","last_unstale":"2024-05-23T00:00:32.809889+0000","last_undegraded":"2024-05-23T00:00:32.809889+0000","last_fullsized":"2024-05-23T00:00:32.809889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:54:59.202855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023402+0000","last_change":"2024-05-22T23:57:32.329365+0000","last_active":"2024-05-23T00:00:33.023402+0000","last_peered":"2024-05-23T00:00:33.023402+0000","last_clean":"2024-05-23T00:00:33.023402+0000","last_became_active":"2024-05-22T23:57:32.327864+0000","last_became_peered":"2024-05-22T23:57:32.327864+0000","last_unstale":"2024-05-23T00:00:33.023402+0000","last_undegraded":"2024-05-23T00:00:33.023402+0000","last_fullsized":"2024-05-23T00:00:33.023402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:26.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809992+0000","last_change":"2024-05-22T23:57:33.313930+0000","last_active":"2024-05-23T00:00:32.809992+0000","last_peered":"2024-05-23T00:00:32.809992+0000","last_clean":"2024-05-23T00:00:32.809992+0000","last_became_active":"2024-05-22T23:57:33.313612+0000","last_became_peered":"2024-05-22T23:57:33.313612+0000","last_unstale":"2024-05-23T00:00:32.809992+0000","last_undegraded":"2024-05-23T00:00:32.809992+0000","last_fullsized":"2024-05-23T00:00:32.809992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:45:25.800338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670644+0000","last_change":"2024-05-22T23:57:37.791537+0000","last_active":"2024-05-23T00:00:32.670644+0000","last_peered":"2024-05-23T00:00:32.670644+0000","last_clean":"2024-05-23T00:00:32.670644+0000","last_became_active":"2024-05-22T23:57:37.791052+0000","last_became_peered":"2024-05-22T23:57:37.791052+0000","last_unstale":"2024-05-23T00:00:32.670644+0000","last_undegraded":"2024-05-23T00:00:32.670644+0000","last_fullsized":"2024-05-23T00:00:32.670644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:18:11.759840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818596+0000","last_change":"2024-05-22T23:57:32.315907+0000","last_active":"2024-05-23T00:00:32.818596+0000","last_peered":"2024-05-23T00:00:32.818596+0000","last_clean":"2024-05-23T00:00:32.818596+0000","last_became_active":"2024-05-22T23:57:32.315576+0000","last_became_peered":"2024-05-22T23:57:32.315576+0000","last_unstale":"2024-05-23T00:00:32.818596+0000","last_undegraded":"2024-05-23T00:00:32.818596+0000","last_fullsized":"2024-05-23T00:00:32.818596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:45:48.998377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671704+0000","last_change":"2024-05-22T23:57:33.338681+0000","last_active":"2024-05-23T00:00:32.671704+0000","last_peered":"2024-05-23T00:00:32.671704+0000","last_clean":"2024-05-23T00:00:32.671704+0000","last_became_active":"2024-05-22T23:57:33.334705+0000","last_became_peered":"2024-05-22T23:57:33.334705+0000","last_unstale":"2024-05-23T00:00:32.671704+0000","last_undegraded":"2024-05-23T00:00:32.671704+0000","last_fullsized":"2024-05-23T00:00:32.671704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:22.135137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810677+0000","last_change":"2024-05-22T23:57:37.780470+0000","last_active":"2024-05-23T00:00:32.810677+0000","last_peered":"2024-05-23T00:00:32.810677+0000","last_clean":"2024-05-23T00:00:32.810677+0000","last_became_active":"2024-05-22T23:57:37.779110+0000","last_became_peered":"2024-05-22T23:57:37.779110+0000","last_unstale":"2024-05-23T00:00:32.810677+0000","last_undegraded":"2024-05-23T00:00:32.810677+0000","last_fullsized":"2024-05-23T00:00:32.810677+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:55:45.486862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":183,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822985+0000","last_change":"2024-05-22T23:57:32.322489+0000","last_active":"2024-05-23T00:00:32.822985+0000","last_peered":"2024-05-23T00:00:32.822985+0000","last_clean":"2024-05-23T00:00:32.822985+0000","last_became_active":"2024-05-22T23:57:32.322125+0000","last_became_peered":"2024-05-22T23:57:32.322125+0000","last_unstale":"2024-05-23T00:00:32.822985+0000","last_undegraded":"2024-05-23T00:00:32.822985+0000","last_fullsized":"2024-05-23T00:00:32.822985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:11:47.707506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021521+0000","last_change":"2024-05-22T23:57:33.322417+0000","last_active":"2024-05-23T00:00:33.021521+0000","last_peered":"2024-05-23T00:00:33.021521+0000","last_clean":"2024-05-23T00:00:33.021521+0000","last_became_active":"2024-05-22T23:57:33.322102+0000","last_became_peered":"2024-05-22T23:57:33.322102+0000","last_unstale":"2024-05-23T00:00:33.021521+0000","last_undegraded":"2024-05-23T00:00:33.021521+0000","last_fullsized":"2024-05-23T00:00:33.021521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:26:43.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671629+0000","last_change":"2024-05-22T23:57:37.784424+0000","last_active":"2024-05-23T00:00:32.671629+0000","last_peered":"2024-05-23T00:00:32.671629+0000","last_clean":"2024-05-23T00:00:32.671629+0000","last_became_active":"2024-05-22T23:57:37.783949+0000","last_became_peered":"2024-05-22T23:57:37.783949+0000","last_unstale":"2024-05-23T00:00:32.671629+0000","last_undegraded":"2024-05-23T00:00:32.671629+0000","last_fullsized":"2024-05-23T00:00:32.671629+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:52:23.763273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023938+0000","last_change":"2024-05-22T23:58:31.110764+0000","last_active":"2024-05-23T00:00:33.023938+0000","last_peered":"2024-05-23T00:00:33.023938+0000","last_clean":"2024-05-23T00:00:33.023938+0000","last_became_active":"2024-05-22T23:58:31.103962+0000","last_became_peered":"2024-05-22T23:58:31.103962+0000","last_unstale":"2024-05-23T00:00:33.023938+0000","last_undegraded":"2024-05-23T00:00:33.023938+0000","last_fullsized":"2024-05-23T00:00:33.023938+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:14:24.248179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670954+0000","last_change":"2024-05-22T23:57:33.324237+0000","last_active":"2024-05-23T00:00:32.670954+0000","last_peered":"2024-05-23T00:00:32.670954+0000","last_clean":"2024-05-23T00:00:32.670954+0000","last_became_active":"2024-05-22T23:57:33.323870+0000","last_became_peered":"2024-05-22T23:57:33.323870+0000","last_unstale":"2024-05-23T00:00:32.670954+0000","last_undegraded":"2024-05-23T00:00:32.670954+0000","last_fullsized":"2024-05-23T00:00:32.670954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:41:29.328104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022755+0000","last_change":"2024-05-22T23:57:37.787795+0000","last_active":"2024-05-23T00:00:33.022755+0000","last_peered":"2024-05-23T00:00:33.022755+0000","last_clean":"2024-05-23T00:00:33.022755+0000","last_became_active":"2024-05-22T23:57:37.787210+0000","last_became_peered":"2024-05-22T23:57:37.787210+0000","last_unstale":"2024-05-23T00:00:33.022755+0000","last_undegraded":"2024-05-23T00:00:33.022755+0000","last_fullsized":"2024-05-23T00:00:33.022755+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:19:04.616758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819178+0000","last_change":"2024-05-22T23:57:32.321415+0000","last_active":"2024-05-23T00:00:32.819178+0000","last_peered":"2024-05-23T00:00:32.819178+0000","last_clean":"2024-05-23T00:00:32.819178+0000","last_became_active":"2024-05-22T23:57:32.321055+0000","last_became_peered":"2024-05-22T23:57:32.321055+0000","last_unstale":"2024-05-23T00:00:32.819178+0000","last_undegraded":"2024-05-23T00:00:32.819178+0000","last_fullsized":"2024-05-23T00:00:32.819178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:28:58.577548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021090+0000","last_change":"2024-05-22T23:58:31.116991+0000","last_active":"2024-05-23T00:00:33.021090+0000","last_peered":"2024-05-23T00:00:33.021090+0000","last_clean":"2024-05-23T00:00:33.021090+0000","last_became_active":"2024-05-22T23:58:31.116526+0000","last_became_peered":"2024-05-22T23:58:31.116526+0000","last_unstale":"2024-05-23T00:00:33.021090+0000","last_undegraded":"2024-05-23T00:00:33.021090+0000","last_fullsized":"2024-05-23T00:00:33.021090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:51:23.350871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671835+0000","last_change":"2024-05-22T23:57:37.781367+0000","last_active":"2024-05-23T00:00:32.671835+0000","last_peered":"2024-05-23T00:00:32.671835+0000","last_clean":"2024-05-23T00:00:32.671835+0000","last_became_active":"2024-05-22T23:57:37.780955+0000","last_became_peered":"2024-05-22T23:57:37.780955+0000","last_unstale":"2024-05-23T00:00:32.671835+0000","last_undegraded":"2024-05-23T00:00:32.671835+0000","last_fullsized":"2024-05-23T00:00:32.671835+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:36:18.962940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":380,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.298872+0000","last_change":"2024-05-22T23:57:32.313873+0000","last_active":"2024-05-23T00:00:33.298872+0000","last_peered":"2024-05-23T00:00:33.298872+0000","last_clean":"2024-05-23T00:00:33.298872+0000","last_became_active":"2024-05-22T23:57:32.313388+0000","last_became_peered":"2024-05-22T23:57:32.313388+0000","last_unstale":"2024-05-23T00:00:33.298872+0000","last_undegraded":"2024-05-23T00:00:33.298872+0000","last_fullsized":"2024-05-23T00:00:33.298872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:16.383384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69043,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":308,"num_read_kb":221,"num_write":43,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022567+0000","last_change":"2024-05-22T23:57:33.318751+0000","last_active":"2024-05-23T00:00:33.022567+0000","last_peered":"2024-05-23T00:00:33.022567+0000","last_clean":"2024-05-23T00:00:33.022567+0000","last_became_active":"2024-05-22T23:57:33.317710+0000","last_became_peered":"2024-05-22T23:57:33.317710+0000","last_unstale":"2024-05-23T00:00:33.022567+0000","last_undegraded":"2024-05-23T00:00:33.022567+0000","last_fullsized":"2024-05-23T00:00:33.022567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:57:50.852615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819509+0000","last_change":"2024-05-22T23:57:37.790073+0000","last_active":"2024-05-23T00:00:32.819509+0000","last_peered":"2024-05-23T00:00:32.819509+0000","last_clean":"2024-05-23T00:00:32.819509+0000","last_became_active":"2024-05-22T23:57:37.789477+0000","last_became_peered":"2024-05-22T23:57:37.789477+0000","last_unstale":"2024-05-23T00:00:32.819509+0000","last_undegraded":"2024-05-23T00:00:32.819509+0000","last_fullsized":"2024-05-23T00:00:32.819509+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:11.003201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670178+0000","last_change":"2024-05-22T23:57:32.329846+0000","last_active":"2024-05-23T00:00:32.670178+0000","last_peered":"2024-05-23T00:00:32.670178+0000","last_clean":"2024-05-23T00:00:32.670178+0000","last_became_active":"2024-05-22T23:57:32.329414+0000","last_became_peered":"2024-05-22T23:57:32.329414+0000","last_unstale":"2024-05-23T00:00:32.670178+0000","last_undegraded":"2024-05-23T00:00:32.670178+0000","last_fullsized":"2024-05-23T00:00:32.670178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:15:38.176790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023754+0000","last_change":"2024-05-22T23:57:33.333298+0000","last_active":"2024-05-23T00:00:33.023754+0000","last_peered":"2024-05-23T00:00:33.023754+0000","last_clean":"2024-05-23T00:00:33.023754+0000","last_became_active":"2024-05-22T23:57:33.332934+0000","last_became_peered":"2024-05-22T23:57:33.332934+0000","last_unstale":"2024-05-23T00:00:33.023754+0000","last_undegraded":"2024-05-23T00:00:33.023754+0000","last_fullsized":"2024-05-23T00:00:33.023754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:06:00.427627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809610+0000","last_change":"2024-05-22T23:57:37.795628+0000","last_active":"2024-05-23T00:00:32.809610+0000","last_peered":"2024-05-23T00:00:32.809610+0000","last_clean":"2024-05-23T00:00:32.809610+0000","last_became_active":"2024-05-22T23:57:37.795208+0000","last_became_peered":"2024-05-22T23:57:37.795208+0000","last_unstale":"2024-05-23T00:00:32.809610+0000","last_undegraded":"2024-05-23T00:00:32.809610+0000","last_fullsized":"2024-05-23T00:00:32.809610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:03:12.684474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818880+0000","last_change":"2024-05-22T23:57:32.334039+0000","last_active":"2024-05-23T00:00:32.818880+0000","last_peered":"2024-05-23T00:00:32.818880+0000","last_clean":"2024-05-23T00:00:32.818880+0000","last_became_active":"2024-05-22T23:57:32.333709+0000","last_became_peered":"2024-05-22T23:57:32.333709+0000","last_unstale":"2024-05-23T00:00:32.818880+0000","last_undegraded":"2024-05-23T00:00:32.818880+0000","last_fullsized":"2024-05-23T00:00:32.818880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:41:43.319094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023784+0000","last_change":"2024-05-22T23:57:33.328056+0000","last_active":"2024-05-23T00:00:33.023784+0000","last_peered":"2024-05-23T00:00:33.023784+0000","last_clean":"2024-05-23T00:00:33.023784+0000","last_became_active":"2024-05-22T23:57:33.326559+0000","last_became_peered":"2024-05-22T23:57:33.326559+0000","last_unstale":"2024-05-23T00:00:33.023784+0000","last_undegraded":"2024-05-23T00:00:33.023784+0000","last_fullsized":"2024-05-23T00:00:33.023784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:06.463837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672875+0000","last_change":"2024-05-22T23:57:37.795892+0000","last_active":"2024-05-23T00:00:32.672875+0000","last_peered":"2024-05-23T00:00:32.672875+0000","last_clean":"2024-05-23T00:00:32.672875+0000","last_became_active":"2024-05-22T23:57:37.792246+0000","last_became_peered":"2024-05-22T23:57:37.792246+0000","last_unstale":"2024-05-23T00:00:32.672875+0000","last_undegraded":"2024-05-23T00:00:32.672875+0000","last_fullsized":"2024-05-23T00:00:32.672875+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:18.462267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":190,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.560988+0000","last_change":"2024-05-22T23:57:32.319379+0000","last_active":"2024-05-23T00:00:33.560988+0000","last_peered":"2024-05-23T00:00:33.560988+0000","last_clean":"2024-05-23T00:00:33.560988+0000","last_became_active":"2024-05-22T23:57:32.318803+0000","last_became_peered":"2024-05-22T23:57:32.318803+0000","last_unstale":"2024-05-23T00:00:33.560988+0000","last_undegraded":"2024-05-23T00:00:33.560988+0000","last_fullsized":"2024-05-23T00:00:33.560988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:59:15.689119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1509,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023586+0000","last_change":"2024-05-22T23:57:33.325294+0000","last_active":"2024-05-23T00:00:33.023586+0000","last_peered":"2024-05-23T00:00:33.023586+0000","last_clean":"2024-05-23T00:00:33.023586+0000","last_became_active":"2024-05-22T23:57:33.324339+0000","last_became_peered":"2024-05-22T23:57:33.324339+0000","last_unstale":"2024-05-23T00:00:33.023586+0000","last_undegraded":"2024-05-23T00:00:33.023586+0000","last_fullsized":"2024-05-23T00:00:33.023586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:29:47.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024152+0000","last_change":"2024-05-22T23:57:37.797961+0000","last_active":"2024-05-23T00:00:33.024152+0000","last_peered":"2024-05-23T00:00:33.024152+0000","last_clean":"2024-05-23T00:00:33.024152+0000","last_became_active":"2024-05-22T23:57:37.797532+0000","last_became_peered":"2024-05-22T23:57:37.797532+0000","last_unstale":"2024-05-23T00:00:33.024152+0000","last_undegraded":"2024-05-23T00:00:33.024152+0000","last_fullsized":"2024-05-23T00:00:33.024152+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:51:12.997674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022065+0000","last_change":"2024-05-22T23:57:32.317434+0000","last_active":"2024-05-23T00:00:33.022065+0000","last_peered":"2024-05-23T00:00:33.022065+0000","last_clean":"2024-05-23T00:00:33.022065+0000","last_became_active":"2024-05-22T23:57:32.316987+0000","last_became_peered":"2024-05-22T23:57:32.316987+0000","last_unstale":"2024-05-23T00:00:33.022065+0000","last_undegraded":"2024-05-23T00:00:33.022065+0000","last_fullsized":"2024-05-23T00:00:33.022065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:46:50.306756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818865+0000","last_change":"2024-05-22T23:57:33.328670+0000","last_active":"2024-05-23T00:00:32.818865+0000","last_peered":"2024-05-23T00:00:32.818865+0000","last_clean":"2024-05-23T00:00:32.818865+0000","last_became_active":"2024-05-22T23:57:33.328294+0000","last_became_peered":"2024-05-22T23:57:33.328294+0000","last_unstale":"2024-05-23T00:00:32.818865+0000","last_undegraded":"2024-05-23T00:00:32.818865+0000","last_fullsized":"2024-05-23T00:00:32.818865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:09:10.807844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818693+0000","last_change":"2024-05-22T23:57:37.788885+0000","last_active":"2024-05-23T00:00:32.818693+0000","last_peered":"2024-05-23T00:00:32.818693+0000","last_clean":"2024-05-23T00:00:32.818693+0000","last_became_active":"2024-05-22T23:57:37.788145+0000","last_became_peered":"2024-05-22T23:57:37.788145+0000","last_unstale":"2024-05-23T00:00:32.818693+0000","last_undegraded":"2024-05-23T00:00:32.818693+0000","last_fullsized":"2024-05-23T00:00:32.818693+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:14:50.027785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022537+0000","last_change":"2024-05-22T23:57:32.317855+0000","last_active":"2024-05-23T00:00:33.022537+0000","last_peered":"2024-05-23T00:00:33.022537+0000","last_clean":"2024-05-23T00:00:33.022537+0000","last_became_active":"2024-05-22T23:57:32.317087+0000","last_became_peered":"2024-05-22T23:57:32.317087+0000","last_unstale":"2024-05-23T00:00:33.022537+0000","last_undegraded":"2024-05-23T00:00:33.022537+0000","last_fullsized":"2024-05-23T00:00:33.022537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:49:40.486222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809847+0000","last_change":"2024-05-22T23:57:33.323897+0000","last_active":"2024-05-23T00:00:32.809847+0000","last_peered":"2024-05-23T00:00:32.809847+0000","last_clean":"2024-05-23T00:00:32.809847+0000","last_became_active":"2024-05-22T23:57:33.323567+0000","last_became_peered":"2024-05-22T23:57:33.323567+0000","last_unstale":"2024-05-23T00:00:32.809847+0000","last_undegraded":"2024-05-23T00:00:32.809847+0000","last_fullsized":"2024-05-23T00:00:32.809847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:17:48.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022150+0000","last_change":"2024-05-22T23:57:38.218313+0000","last_active":"2024-05-23T00:00:33.022150+0000","last_peered":"2024-05-23T00:00:33.022150+0000","last_clean":"2024-05-23T00:00:33.022150+0000","last_became_active":"2024-05-22T23:57:38.217756+0000","last_became_peered":"2024-05-22T23:57:38.217756+0000","last_unstale":"2024-05-23T00:00:33.022150+0000","last_undegraded":"2024-05-23T00:00:33.022150+0000","last_fullsized":"2024-05-23T00:00:33.022150+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:07:55.122260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672176+0000","last_change":"2024-05-22T23:57:32.325729+0000","last_active":"2024-05-23T00:00:32.672176+0000","last_peered":"2024-05-23T00:00:32.672176+0000","last_clean":"2024-05-23T00:00:32.672176+0000","last_became_active":"2024-05-22T23:57:32.325383+0000","last_became_peered":"2024-05-22T23:57:32.325383+0000","last_unstale":"2024-05-23T00:00:32.672176+0000","last_undegraded":"2024-05-23T00:00:32.672176+0000","last_fullsized":"2024-05-23T00:00:32.672176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:57.978806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822720+0000","last_change":"2024-05-22T23:57:33.337499+0000","last_active":"2024-05-23T00:00:32.822720+0000","last_peered":"2024-05-23T00:00:32.822720+0000","last_clean":"2024-05-23T00:00:32.822720+0000","last_became_active":"2024-05-22T23:57:33.337271+0000","last_became_peered":"2024-05-22T23:57:33.337271+0000","last_unstale":"2024-05-23T00:00:32.822720+0000","last_undegraded":"2024-05-23T00:00:32.822720+0000","last_fullsized":"2024-05-23T00:00:32.822720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:32:39.133984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024205+0000","last_change":"2024-05-22T23:57:37.792707+0000","last_active":"2024-05-23T00:00:33.024205+0000","last_peered":"2024-05-23T00:00:33.024205+0000","last_clean":"2024-05-23T00:00:33.024205+0000","last_became_active":"2024-05-22T23:57:37.792209+0000","last_became_peered":"2024-05-22T23:57:37.792209+0000","last_unstale":"2024-05-23T00:00:33.024205+0000","last_undegraded":"2024-05-23T00:00:33.024205+0000","last_fullsized":"2024-05-23T00:00:33.024205+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:36:34.089424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809543+0000","last_change":"2024-05-22T23:58:31.112612+0000","last_active":"2024-05-23T00:00:32.809543+0000","last_peered":"2024-05-23T00:00:32.809543+0000","last_clean":"2024-05-23T00:00:32.809543+0000","last_became_active":"2024-05-22T23:58:31.112181+0000","last_became_peered":"2024-05-22T23:58:31.112181+0000","last_unstale":"2024-05-23T00:00:32.809543+0000","last_undegraded":"2024-05-23T00:00:32.809543+0000","last_fullsized":"2024-05-23T00:00:32.809543+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:25:18.717294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822488+0000","last_change":"2024-05-22T23:57:33.328129+0000","last_active":"2024-05-23T00:00:32.822488+0000","last_peered":"2024-05-23T00:00:32.822488+0000","last_clean":"2024-05-23T00:00:32.822488+0000","last_became_active":"2024-05-22T23:57:33.327732+0000","last_became_peered":"2024-05-22T23:57:33.327732+0000","last_unstale":"2024-05-23T00:00:32.822488+0000","last_undegraded":"2024-05-23T00:00:32.822488+0000","last_fullsized":"2024-05-23T00:00:32.822488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:25:14.105235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024320+0000","last_change":"2024-05-22T23:57:38.220884+0000","last_active":"2024-05-23T00:00:33.024320+0000","last_peered":"2024-05-23T00:00:33.024320+0000","last_clean":"2024-05-23T00:00:33.024320+0000","last_became_active":"2024-05-22T23:57:38.219365+0000","last_became_peered":"2024-05-22T23:57:38.219365+0000","last_unstale":"2024-05-23T00:00:33.024320+0000","last_undegraded":"2024-05-23T00:00:33.024320+0000","last_fullsized":"2024-05-23T00:00:33.024320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:54:56.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818771+0000","last_change":"2024-05-22T23:57:33.328272+0000","last_active":"2024-05-23T00:00:32.818771+0000","last_peered":"2024-05-23T00:00:32.818771+0000","last_clean":"2024-05-23T00:00:32.818771+0000","last_became_active":"2024-05-22T23:57:33.327915+0000","last_became_peered":"2024-05-22T23:57:33.327915+0000","last_unstale":"2024-05-23T00:00:32.818771+0000","last_undegraded":"2024-05-23T00:00:32.818771+0000","last_fullsized":"2024-05-23T00:00:32.818771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:59:09.085649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":185,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.031020+0000","last_change":"2024-05-22T23:57:32.331979+0000","last_active":"2024-05-23T00:00:33.031020+0000","last_peered":"2024-05-23T00:00:33.031020+0000","last_clean":"2024-05-23T00:00:33.031020+0000","last_became_active":"2024-05-22T23:57:32.330983+0000","last_became_peered":"2024-05-22T23:57:32.330983+0000","last_unstale":"2024-05-23T00:00:33.031020+0000","last_undegraded":"2024-05-23T00:00:33.031020+0000","last_fullsized":"2024-05-23T00:00:33.031020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:07:19.137652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023940+0000","last_change":"2024-05-22T23:57:37.787935+0000","last_active":"2024-05-23T00:00:33.023940+0000","last_peered":"2024-05-23T00:00:33.023940+0000","last_clean":"2024-05-23T00:00:33.023940+0000","last_became_active":"2024-05-22T23:57:37.787495+0000","last_became_peered":"2024-05-22T23:57:37.787495+0000","last_unstale":"2024-05-23T00:00:33.023940+0000","last_undegraded":"2024-05-23T00:00:33.023940+0000","last_fullsized":"2024-05-23T00:00:33.023940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:37.682711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810969+0000","last_change":"2024-05-22T23:57:33.329451+0000","last_active":"2024-05-23T00:00:32.810969+0000","last_peered":"2024-05-23T00:00:32.810969+0000","last_clean":"2024-05-23T00:00:32.810969+0000","last_became_active":"2024-05-22T23:57:33.329107+0000","last_became_peered":"2024-05-22T23:57:33.329107+0000","last_unstale":"2024-05-23T00:00:32.810969+0000","last_undegraded":"2024-05-23T00:00:32.810969+0000","last_fullsized":"2024-05-23T00:00:32.810969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:44:50.198040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022720+0000","last_change":"2024-05-22T23:57:32.321873+0000","last_active":"2024-05-23T00:00:33.022720+0000","last_peered":"2024-05-23T00:00:33.022720+0000","last_clean":"2024-05-23T00:00:33.022720+0000","last_became_active":"2024-05-22T23:57:32.321550+0000","last_became_peered":"2024-05-22T23:57:32.321550+0000","last_unstale":"2024-05-23T00:00:33.022720+0000","last_undegraded":"2024-05-23T00:00:33.022720+0000","last_fullsized":"2024-05-23T00:00:33.022720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:30:17.479238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":363,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023381+0000","last_change":"2024-05-22T23:56:11.327788+0000","last_active":"2024-05-23T00:00:33.023381+0000","last_peered":"2024-05-23T00:00:33.023381+0000","last_clean":"2024-05-23T00:00:33.023381+0000","last_became_active":"2024-05-22T23:56:11.327232+0000","last_became_peered":"2024-05-22T23:56:11.327232+0000","last_unstale":"2024-05-23T00:00:33.023381+0000","last_undegraded":"2024-05-23T00:00:33.023381+0000","last_fullsized":"2024-05-23T00:00:33.023381+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_clean_scrub_stamp":"2024-05-22T23:52:16.719643+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:26:13.077470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819294+0000","last_change":"2024-05-22T23:57:38.219326+0000","last_active":"2024-05-23T00:00:32.819294+0000","last_peered":"2024-05-23T00:00:32.819294+0000","last_clean":"2024-05-23T00:00:32.819294+0000","last_became_active":"2024-05-22T23:57:38.218835+0000","last_became_peered":"2024-05-22T23:57:38.218835+0000","last_unstale":"2024-05-23T00:00:32.819294+0000","last_undegraded":"2024-05-23T00:00:32.819294+0000","last_fullsized":"2024-05-23T00:00:32.819294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:29.722862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023039+0000","last_change":"2024-05-22T23:57:33.316760+0000","last_active":"2024-05-23T00:00:33.023039+0000","last_peered":"2024-05-23T00:00:33.023039+0000","last_clean":"2024-05-23T00:00:33.023039+0000","last_became_active":"2024-05-22T23:57:33.316328+0000","last_became_peered":"2024-05-22T23:57:33.316328+0000","last_unstale":"2024-05-23T00:00:33.023039+0000","last_undegraded":"2024-05-23T00:00:33.023039+0000","last_fullsized":"2024-05-23T00:00:33.023039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:53:11.592134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023557+0000","last_change":"2024-05-22T23:57:32.319154+0000","last_active":"2024-05-23T00:00:33.023557+0000","last_peered":"2024-05-23T00:00:33.023557+0000","last_clean":"2024-05-23T00:00:33.023557+0000","last_became_active":"2024-05-22T23:57:32.318808+0000","last_became_peered":"2024-05-22T23:57:32.318808+0000","last_unstale":"2024-05-23T00:00:33.023557+0000","last_undegraded":"2024-05-23T00:00:33.023557+0000","last_fullsized":"2024-05-23T00:00:33.023557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:37:47.271767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809550+0000","last_change":"2024-05-22T23:57:37.791464+0000","last_active":"2024-05-23T00:00:32.809550+0000","last_peered":"2024-05-23T00:00:32.809550+0000","last_clean":"2024-05-23T00:00:32.809550+0000","last_became_active":"2024-05-22T23:57:37.791044+0000","last_became_peered":"2024-05-22T23:57:37.791044+0000","last_unstale":"2024-05-23T00:00:32.809550+0000","last_undegraded":"2024-05-23T00:00:32.809550+0000","last_fullsized":"2024-05-23T00:00:32.809550+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:47:00.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022492+0000","last_change":"2024-05-22T23:57:32.322740+0000","last_active":"2024-05-23T00:00:33.022492+0000","last_peered":"2024-05-23T00:00:33.022492+0000","last_clean":"2024-05-23T00:00:33.022492+0000","last_became_active":"2024-05-22T23:57:32.322425+0000","last_became_peered":"2024-05-22T23:57:32.322425+0000","last_unstale":"2024-05-23T00:00:33.022492+0000","last_undegraded":"2024-05-23T00:00:33.022492+0000","last_fullsized":"2024-05-23T00:00:33.022492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:05:10.522462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023588+0000","last_change":"2024-05-22T23:57:33.324667+0000","last_active":"2024-05-23T00:00:33.023588+0000","last_peered":"2024-05-23T00:00:33.023588+0000","last_clean":"2024-05-23T00:00:33.023588+0000","last_became_active":"2024-05-22T23:57:33.324251+0000","last_became_peered":"2024-05-22T23:57:33.324251+0000","last_unstale":"2024-05-23T00:00:33.023588+0000","last_undegraded":"2024-05-23T00:00:33.023588+0000","last_fullsized":"2024-05-23T00:00:33.023588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:49:57.203996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809807+0000","last_change":"2024-05-22T23:57:38.222346+0000","last_active":"2024-05-23T00:00:32.809807+0000","last_peered":"2024-05-23T00:00:32.809807+0000","last_clean":"2024-05-23T00:00:32.809807+0000","last_became_active":"2024-05-22T23:57:38.220248+0000","last_became_peered":"2024-05-22T23:57:38.220248+0000","last_unstale":"2024-05-23T00:00:32.809807+0000","last_undegraded":"2024-05-23T00:00:32.809807+0000","last_fullsized":"2024-05-23T00:00:32.809807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:26:44.026641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.963348+0000","last_change":"2024-05-22T23:58:31.126056+0000","last_active":"2024-05-23T00:00:32.963348+0000","last_peered":"2024-05-23T00:00:32.963348+0000","last_clean":"2024-05-23T00:00:32.963348+0000","last_became_active":"2024-05-22T23:58:31.117186+0000","last_became_peered":"2024-05-22T23:58:31.117186+0000","last_unstale":"2024-05-23T00:00:32.963348+0000","last_undegraded":"2024-05-23T00:00:32.963348+0000","last_fullsized":"2024-05-23T00:00:32.963348+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:21:34.156655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":79,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671990+0000","last_change":"2024-05-22T23:58:31.116649+0000","last_active":"2024-05-23T00:00:32.671990+0000","last_peered":"2024-05-23T00:00:32.671990+0000","last_clean":"2024-05-23T00:00:32.671990+0000","last_became_active":"2024-05-22T23:58:31.116009+0000","last_became_peered":"2024-05-22T23:58:31.116009+0000","last_unstale":"2024-05-23T00:00:32.671990+0000","last_undegraded":"2024-05-23T00:00:32.671990+0000","last_fullsized":"2024-05-23T00:00:32.671990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:00:29.964565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669481+0000","last_change":"2024-05-22T23:57:37.778524+0000","last_active":"2024-05-23T00:00:32.669481+0000","last_peered":"2024-05-23T00:00:32.669481+0000","last_clean":"2024-05-23T00:00:32.669481+0000","last_became_active":"2024-05-22T23:57:37.777827+0000","last_became_peered":"2024-05-22T23:57:37.777827+0000","last_unstale":"2024-05-23T00:00:32.669481+0000","last_undegraded":"2024-05-23T00:00:32.669481+0000","last_fullsized":"2024-05-23T00:00:32.669481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:30:59.154921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023206+0000","last_change":"2024-05-22T23:57:32.317649+0000","last_active":"2024-05-23T00:00:33.023206+0000","last_peered":"2024-05-23T00:00:33.023206+0000","last_clean":"2024-05-23T00:00:33.023206+0000","last_became_active":"2024-05-22T23:57:32.317280+0000","last_became_peered":"2024-05-22T23:57:32.317280+0000","last_unstale":"2024-05-23T00:00:33.023206+0000","last_undegraded":"2024-05-23T00:00:33.023206+0000","last_fullsized":"2024-05-23T00:00:33.023206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:01:16.917079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670710+0000","last_change":"2024-05-22T23:57:33.339296+0000","last_active":"2024-05-23T00:00:32.670710+0000","last_peered":"2024-05-23T00:00:32.670710+0000","last_clean":"2024-05-23T00:00:32.670710+0000","last_became_active":"2024-05-22T23:57:33.338457+0000","last_became_peered":"2024-05-22T23:57:33.338457+0000","last_unstale":"2024-05-23T00:00:32.670710+0000","last_undegraded":"2024-05-23T00:00:32.670710+0000","last_fullsized":"2024-05-23T00:00:32.670710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:13:05.927893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023012+0000","last_change":"2024-05-22T23:57:38.218649+0000","last_active":"2024-05-23T00:00:33.023012+0000","last_peered":"2024-05-23T00:00:33.023012+0000","last_clean":"2024-05-23T00:00:33.023012+0000","last_became_active":"2024-05-22T23:57:38.218189+0000","last_became_peered":"2024-05-22T23:57:38.218189+0000","last_unstale":"2024-05-23T00:00:33.023012+0000","last_undegraded":"2024-05-23T00:00:33.023012+0000","last_fullsized":"2024-05-23T00:00:33.023012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:30.747293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020655+0000","last_change":"2024-05-22T23:57:32.309379+0000","last_active":"2024-05-23T00:00:33.020655+0000","last_peered":"2024-05-23T00:00:33.020655+0000","last_clean":"2024-05-23T00:00:33.020655+0000","last_became_active":"2024-05-22T23:57:32.309068+0000","last_became_peered":"2024-05-22T23:57:32.309068+0000","last_unstale":"2024-05-23T00:00:33.020655+0000","last_undegraded":"2024-05-23T00:00:33.020655+0000","last_fullsized":"2024-05-23T00:00:33.020655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:43:54.787996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022150+0000","last_change":"2024-05-22T23:57:33.320253+0000","last_active":"2024-05-23T00:00:33.022150+0000","last_peered":"2024-05-23T00:00:33.022150+0000","last_clean":"2024-05-23T00:00:33.022150+0000","last_became_active":"2024-05-22T23:57:33.319604+0000","last_became_peered":"2024-05-22T23:57:33.319604+0000","last_unstale":"2024-05-23T00:00:33.022150+0000","last_undegraded":"2024-05-23T00:00:33.022150+0000","last_fullsized":"2024-05-23T00:00:33.022150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:23:51.745040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672719+0000","last_change":"2024-05-22T23:57:37.795368+0000","last_active":"2024-05-23T00:00:32.672719+0000","last_peered":"2024-05-23T00:00:32.672719+0000","last_clean":"2024-05-23T00:00:32.672719+0000","last_became_active":"2024-05-22T23:57:37.792848+0000","last_became_peered":"2024-05-22T23:57:37.792848+0000","last_unstale":"2024-05-23T00:00:32.672719+0000","last_undegraded":"2024-05-23T00:00:32.672719+0000","last_fullsized":"2024-05-23T00:00:32.672719+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:45.859867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810299+0000","last_change":"2024-05-22T23:57:32.329147+0000","last_active":"2024-05-23T00:00:32.810299+0000","last_peered":"2024-05-23T00:00:32.810299+0000","last_clean":"2024-05-23T00:00:32.810299+0000","last_became_active":"2024-05-22T23:57:32.327254+0000","last_became_peered":"2024-05-22T23:57:32.327254+0000","last_unstale":"2024-05-23T00:00:32.810299+0000","last_undegraded":"2024-05-23T00:00:32.810299+0000","last_fullsized":"2024-05-23T00:00:32.810299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:55:24.915817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809447+0000","last_change":"2024-05-22T23:57:33.332843+0000","last_active":"2024-05-23T00:00:32.809447+0000","last_peered":"2024-05-23T00:00:32.809447+0000","last_clean":"2024-05-23T00:00:32.809447+0000","last_became_active":"2024-05-22T23:57:33.332497+0000","last_became_peered":"2024-05-22T23:57:33.332497+0000","last_unstale":"2024-05-23T00:00:32.809447+0000","last_undegraded":"2024-05-23T00:00:32.809447+0000","last_fullsized":"2024-05-23T00:00:32.809447+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:55:16.109676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023022+0000","last_change":"2024-05-22T23:57:37.787609+0000","last_active":"2024-05-23T00:00:33.023022+0000","last_peered":"2024-05-23T00:00:33.023022+0000","last_clean":"2024-05-23T00:00:33.023022+0000","last_became_active":"2024-05-22T23:57:37.787153+0000","last_became_peered":"2024-05-22T23:57:37.787153+0000","last_unstale":"2024-05-23T00:00:33.023022+0000","last_undegraded":"2024-05-23T00:00:33.023022+0000","last_fullsized":"2024-05-23T00:00:33.023022+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:57:51.640037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669838+0000","last_change":"2024-05-22T23:57:32.329784+0000","last_active":"2024-05-23T00:00:32.669838+0000","last_peered":"2024-05-23T00:00:32.669838+0000","last_clean":"2024-05-23T00:00:32.669838+0000","last_became_active":"2024-05-22T23:57:32.329393+0000","last_became_peered":"2024-05-22T23:57:32.329393+0000","last_unstale":"2024-05-23T00:00:32.669838+0000","last_undegraded":"2024-05-23T00:00:32.669838+0000","last_fullsized":"2024-05-23T00:00:32.669838+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:17:54.811962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.811747+0000","last_change":"2024-05-22T23:57:33.318803+0000","last_active":"2024-05-23T00:00:32.811747+0000","last_peered":"2024-05-23T00:00:32.811747+0000","last_clean":"2024-05-23T00:00:32.811747+0000","last_became_active":"2024-05-22T23:57:33.318170+0000","last_became_peered":"2024-05-22T23:57:33.318170+0000","last_unstale":"2024-05-23T00:00:32.811747+0000","last_undegraded":"2024-05-23T00:00:32.811747+0000","last_fullsized":"2024-05-23T00:00:32.811747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:56:56.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023028+0000","last_change":"2024-05-22T23:57:38.218583+0000","last_active":"2024-05-23T00:00:33.023028+0000","last_peered":"2024-05-23T00:00:33.023028+0000","last_clean":"2024-05-23T00:00:33.023028+0000","last_became_active":"2024-05-22T23:57:38.218048+0000","last_became_peered":"2024-05-22T23:57:38.218048+0000","last_unstale":"2024-05-23T00:00:33.023028+0000","last_undegraded":"2024-05-23T00:00:33.023028+0000","last_fullsized":"2024-05-23T00:00:33.023028+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:10:57.579656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022209+0000","last_change":"2024-05-22T23:57:32.317908+0000","last_active":"2024-05-23T00:00:33.022209+0000","last_peered":"2024-05-23T00:00:33.022209+0000","last_clean":"2024-05-23T00:00:33.022209+0000","last_became_active":"2024-05-22T23:57:32.317580+0000","last_became_peered":"2024-05-22T23:57:32.317580+0000","last_unstale":"2024-05-23T00:00:33.022209+0000","last_undegraded":"2024-05-23T00:00:33.022209+0000","last_fullsized":"2024-05-23T00:00:33.022209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:50:02.044817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023705+0000","last_change":"2024-05-22T23:57:33.321184+0000","last_active":"2024-05-23T00:00:33.023705+0000","last_peered":"2024-05-23T00:00:33.023705+0000","last_clean":"2024-05-23T00:00:33.023705+0000","last_became_active":"2024-05-22T23:57:33.320418+0000","last_became_peered":"2024-05-22T23:57:33.320418+0000","last_unstale":"2024-05-23T00:00:33.023705+0000","last_undegraded":"2024-05-23T00:00:33.023705+0000","last_fullsized":"2024-05-23T00:00:33.023705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:22:43.019809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020743+0000","last_change":"2024-05-22T23:57:37.795718+0000","last_active":"2024-05-23T00:00:33.020743+0000","last_peered":"2024-05-23T00:00:33.020743+0000","last_clean":"2024-05-23T00:00:33.020743+0000","last_became_active":"2024-05-22T23:57:37.795297+0000","last_became_peered":"2024-05-22T23:57:37.795297+0000","last_unstale":"2024-05-23T00:00:33.020743+0000","last_undegraded":"2024-05-23T00:00:33.020743+0000","last_fullsized":"2024-05-23T00:00:33.020743+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:56:47.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670584+0000","last_change":"2024-05-22T23:57:32.311921+0000","last_active":"2024-05-23T00:00:32.670584+0000","last_peered":"2024-05-23T00:00:32.670584+0000","last_clean":"2024-05-23T00:00:32.670584+0000","last_became_active":"2024-05-22T23:57:32.311600+0000","last_became_peered":"2024-05-22T23:57:32.311600+0000","last_unstale":"2024-05-23T00:00:32.670584+0000","last_undegraded":"2024-05-23T00:00:32.670584+0000","last_fullsized":"2024-05-23T00:00:32.670584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:35:21.882773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024166+0000","last_change":"2024-05-22T23:57:33.321289+0000","last_active":"2024-05-23T00:00:33.024166+0000","last_peered":"2024-05-23T00:00:33.024166+0000","last_clean":"2024-05-23T00:00:33.024166+0000","last_became_active":"2024-05-22T23:57:33.320458+0000","last_became_peered":"2024-05-22T23:57:33.320458+0000","last_unstale":"2024-05-23T00:00:33.024166+0000","last_undegraded":"2024-05-23T00:00:33.024166+0000","last_fullsized":"2024-05-23T00:00:33.024166+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:02:05.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670012+0000","last_change":"2024-05-22T23:57:37.786062+0000","last_active":"2024-05-23T00:00:32.670012+0000","last_peered":"2024-05-23T00:00:32.670012+0000","last_clean":"2024-05-23T00:00:32.670012+0000","last_became_active":"2024-05-22T23:57:37.785624+0000","last_became_peered":"2024-05-22T23:57:37.785624+0000","last_unstale":"2024-05-23T00:00:32.670012+0000","last_undegraded":"2024-05-23T00:00:32.670012+0000","last_fullsized":"2024-05-23T00:00:32.670012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:38:56.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.960516+0000","last_change":"2024-05-22T23:57:32.333909+0000","last_active":"2024-05-23T00:00:32.960516+0000","last_peered":"2024-05-23T00:00:32.960516+0000","last_clean":"2024-05-23T00:00:32.960516+0000","last_became_active":"2024-05-22T23:57:32.333531+0000","last_became_peered":"2024-05-22T23:57:32.333531+0000","last_unstale":"2024-05-23T00:00:32.960516+0000","last_undegraded":"2024-05-23T00:00:32.960516+0000","last_fullsized":"2024-05-23T00:00:32.960516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:53:21.843150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023876+0000","last_change":"2024-05-22T23:57:33.334366+0000","last_active":"2024-05-23T00:00:33.023876+0000","last_peered":"2024-05-23T00:00:33.023876+0000","last_clean":"2024-05-23T00:00:33.023876+0000","last_became_active":"2024-05-22T23:57:33.334037+0000","last_became_peered":"2024-05-22T23:57:33.334037+0000","last_unstale":"2024-05-23T00:00:33.023876+0000","last_undegraded":"2024-05-23T00:00:33.023876+0000","last_fullsized":"2024-05-23T00:00:33.023876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:15:10.631087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818480+0000","last_change":"2024-05-22T23:57:37.780241+0000","last_active":"2024-05-23T00:00:32.818480+0000","last_peered":"2024-05-23T00:00:32.818480+0000","last_clean":"2024-05-23T00:00:32.818480+0000","last_became_active":"2024-05-22T23:57:37.779816+0000","last_became_peered":"2024-05-22T23:57:37.779816+0000","last_unstale":"2024-05-23T00:00:32.818480+0000","last_undegraded":"2024-05-23T00:00:32.818480+0000","last_fullsized":"2024-05-23T00:00:32.818480+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:11.397681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024119+0000","last_change":"2024-05-22T23:57:32.334829+0000","last_active":"2024-05-23T00:00:33.024119+0000","last_peered":"2024-05-23T00:00:33.024119+0000","last_clean":"2024-05-23T00:00:33.024119+0000","last_became_active":"2024-05-22T23:57:32.334368+0000","last_became_peered":"2024-05-22T23:57:32.334368+0000","last_unstale":"2024-05-23T00:00:33.024119+0000","last_undegraded":"2024-05-23T00:00:33.024119+0000","last_fullsized":"2024-05-23T00:00:33.024119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:17:42.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024133+0000","last_change":"2024-05-22T23:57:33.318740+0000","last_active":"2024-05-23T00:00:33.024133+0000","last_peered":"2024-05-23T00:00:33.024133+0000","last_clean":"2024-05-23T00:00:33.024133+0000","last_became_active":"2024-05-22T23:57:33.318363+0000","last_became_peered":"2024-05-22T23:57:33.318363+0000","last_unstale":"2024-05-23T00:00:33.024133+0000","last_undegraded":"2024-05-23T00:00:33.024133+0000","last_fullsized":"2024-05-23T00:00:33.024133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:51:42.319667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809555+0000","last_change":"2024-05-22T23:57:33.325582+0000","last_active":"2024-05-23T00:00:32.809555+0000","last_peered":"2024-05-23T00:00:32.809555+0000","last_clean":"2024-05-23T00:00:32.809555+0000","last_became_active":"2024-05-22T23:57:33.325268+0000","last_became_peered":"2024-05-22T23:57:33.325268+0000","last_unstale":"2024-05-23T00:00:32.809555+0000","last_undegraded":"2024-05-23T00:00:32.809555+0000","last_fullsized":"2024-05-23T00:00:32.809555+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:02:18.968634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669768+0000","last_change":"2024-05-22T23:58:31.109626+0000","last_active":"2024-05-23T00:00:32.669768+0000","last_peered":"2024-05-23T00:00:32.669768+0000","last_clean":"2024-05-23T00:00:32.669768+0000","last_became_active":"2024-05-22T23:58:31.109179+0000","last_became_peered":"2024-05-22T23:58:31.109179+0000","last_unstale":"2024-05-23T00:00:32.669768+0000","last_undegraded":"2024-05-23T00:00:32.669768+0000","last_fullsized":"2024-05-23T00:00:32.669768+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:36.653833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669573+0000","last_change":"2024-05-22T23:57:33.332935+0000","last_active":"2024-05-23T00:00:32.669573+0000","last_peered":"2024-05-23T00:00:32.669573+0000","last_clean":"2024-05-23T00:00:32.669573+0000","last_became_active":"2024-05-22T23:57:33.332607+0000","last_became_peered":"2024-05-22T23:57:33.332607+0000","last_unstale":"2024-05-23T00:00:32.669573+0000","last_undegraded":"2024-05-23T00:00:32.669573+0000","last_fullsized":"2024-05-23T00:00:32.669573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:09.589896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670318+0000","last_change":"2024-05-22T23:57:33.333269+0000","last_active":"2024-05-23T00:00:32.670318+0000","last_peered":"2024-05-23T00:00:32.670318+0000","last_clean":"2024-05-23T00:00:32.670318+0000","last_became_active":"2024-05-22T23:57:33.329017+0000","last_became_peered":"2024-05-22T23:57:33.329017+0000","last_unstale":"2024-05-23T00:00:32.670318+0000","last_undegraded":"2024-05-23T00:00:32.670318+0000","last_fullsized":"2024-05-23T00:00:32.670318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:35:42.156434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023462+0000","last_change":"2024-05-22T23:57:33.328721+0000","last_active":"2024-05-23T00:00:33.023462+0000","last_peered":"2024-05-23T00:00:33.023462+0000","last_clean":"2024-05-23T00:00:33.023462+0000","last_became_active":"2024-05-22T23:57:33.328318+0000","last_became_peered":"2024-05-22T23:57:33.328318+0000","last_unstale":"2024-05-23T00:00:33.023462+0000","last_undegraded":"2024-05-23T00:00:33.023462+0000","last_fullsized":"2024-05-23T00:00:33.023462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:14.772688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809824+0000","last_change":"2024-05-22T23:57:33.324777+0000","last_active":"2024-05-23T00:00:32.809824+0000","last_peered":"2024-05-23T00:00:32.809824+0000","last_clean":"2024-05-23T00:00:32.809824+0000","last_became_active":"2024-05-22T23:57:33.324446+0000","last_became_peered":"2024-05-22T23:57:33.324446+0000","last_unstale":"2024-05-23T00:00:32.809824+0000","last_undegraded":"2024-05-23T00:00:32.809824+0000","last_fullsized":"2024-05-23T00:00:32.809824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:20:42.327094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670314+0000","last_change":"2024-05-22T23:57:33.318783+0000","last_active":"2024-05-23T00:00:32.670314+0000","last_peered":"2024-05-23T00:00:32.670314+0000","last_clean":"2024-05-23T00:00:32.670314+0000","last_became_active":"2024-05-22T23:57:33.318481+0000","last_became_peered":"2024-05-22T23:57:33.318481+0000","last_unstale":"2024-05-23T00:00:32.670314+0000","last_undegraded":"2024-05-23T00:00:32.670314+0000","last_fullsized":"2024-05-23T00:00:32.670314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:14:35.745683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812167+0000","last_change":"2024-05-22T23:57:33.319939+0000","last_active":"2024-05-23T00:00:32.812167+0000","last_peered":"2024-05-23T00:00:32.812167+0000","last_clean":"2024-05-23T00:00:32.812167+0000","last_became_active":"2024-05-22T23:57:33.316029+0000","last_became_peered":"2024-05-22T23:57:33.316029+0000","last_unstale":"2024-05-23T00:00:32.812167+0000","last_undegraded":"2024-05-23T00:00:32.812167+0000","last_fullsized":"2024-05-23T00:00:32.812167+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:46:34.383953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818610+0000","last_change":"2024-05-22T23:57:33.334625+0000","last_active":"2024-05-23T00:00:32.818610+0000","last_peered":"2024-05-23T00:00:32.818610+0000","last_clean":"2024-05-23T00:00:32.818610+0000","last_became_active":"2024-05-22T23:57:33.334341+0000","last_became_peered":"2024-05-22T23:57:33.334341+0000","last_unstale":"2024-05-23T00:00:32.818610+0000","last_undegraded":"2024-05-23T00:00:32.818610+0000","last_fullsized":"2024-05-23T00:00:32.818610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:38.319721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810428+0000","last_change":"2024-05-22T23:57:33.330958+0000","last_active":"2024-05-23T00:00:32.810428+0000","last_peered":"2024-05-23T00:00:32.810428+0000","last_clean":"2024-05-23T00:00:32.810428+0000","last_became_active":"2024-05-22T23:57:33.317355+0000","last_became_peered":"2024-05-22T23:57:33.317355+0000","last_unstale":"2024-05-23T00:00:32.810428+0000","last_undegraded":"2024-05-23T00:00:32.810428+0000","last_fullsized":"2024-05-23T00:00:32.810428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:49:43.784780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021047+0000","last_change":"2024-05-22T23:57:33.319530+0000","last_active":"2024-05-23T00:00:33.021047+0000","last_peered":"2024-05-23T00:00:33.021047+0000","last_clean":"2024-05-23T00:00:33.021047+0000","last_became_active":"2024-05-22T23:57:33.319122+0000","last_became_peered":"2024-05-22T23:57:33.319122+0000","last_unstale":"2024-05-23T00:00:33.021047+0000","last_undegraded":"2024-05-23T00:00:33.021047+0000","last_fullsized":"2024-05-23T00:00:33.021047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:31:26.937804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024256+0000","last_change":"2024-05-22T23:57:33.323722+0000","last_active":"2024-05-23T00:00:33.024256+0000","last_peered":"2024-05-23T00:00:33.024256+0000","last_clean":"2024-05-23T00:00:33.024256+0000","last_became_active":"2024-05-22T23:57:33.323343+0000","last_became_peered":"2024-05-22T23:57:33.323343+0000","last_unstale":"2024-05-23T00:00:33.024256+0000","last_undegraded":"2024-05-23T00:00:33.024256+0000","last_fullsized":"2024-05-23T00:00:33.024256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:24:59.575536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809773+0000","last_change":"2024-05-22T23:57:33.320925+0000","last_active":"2024-05-23T00:00:32.809773+0000","last_peered":"2024-05-23T00:00:32.809773+0000","last_clean":"2024-05-23T00:00:32.809773+0000","last_became_active":"2024-05-22T23:57:33.320570+0000","last_became_peered":"2024-05-22T23:57:33.320570+0000","last_unstale":"2024-05-23T00:00:32.809773+0000","last_undegraded":"2024-05-23T00:00:32.809773+0000","last_fullsized":"2024-05-23T00:00:32.809773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:38:27.659025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107088,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599621,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1018,"num_read_kb":622,"num_write":295,"num_write_kb":1448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1012,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1798863,"data_compressed":80985,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318321,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":911392,"data_compressed":42724,"data_compressed_allocated":57344,"data_compressed_original":458752,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579959,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1164430,"data_compressed":57992,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743484,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":522508,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005122,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":518332,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266761,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1105414,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430284,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":503780,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":582541,"data_compressed":25271,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":514976,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":582489,"data_compressed":25271,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1038598,"data_compressed":48880,"data_compressed_allocated":65536,"data_compressed_original":581632,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888153,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":510424,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691925,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1100471,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69043,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69771,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134307,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203142,"data_compressed":7849,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-23T00:00:36.624 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-23T00:00:36.624 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: mgrmap e24: x(active, since 2s), standbys: z, y 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/122701571' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/957718475' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:36.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:36 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3694249131' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:36.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 -- 172.21.15.103:0/2491519782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa80c103290 msgr2=0x7fa80c109b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 --2- 172.21.15.103:0/2491519782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa80c103290 0x7fa80c109b20 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa804009ca0 tx=0x7fa80402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 -- 172.21.15.103:0/2491519782 shutdown_connections 2024-05-23T00:00:36.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 --2- 172.21.15.103:0/2491519782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa80c103290 0x7fa80c109b20 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 --2- 172.21.15.103:0/2491519782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa80c1028d0 0x7fa80c102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.844 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 --2- 172.21.15.103:0/2491519782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa80c1016e0 0x7fa80c101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 -- 172.21.15.103:0/2491519782 >> 172.21.15.103:0/2491519782 conn(0x7fa80c0fcc50 msgr2=0x7fa80c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:36.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 -- 172.21.15.103:0/2491519782 shutdown_connections 2024-05-23T00:00:36.845 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.840+0000 7fa80be00640 1 -- 172.21.15.103:0/2491519782 wait complete. 2024-05-23T00:00:36.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.841+0000 7fa80be00640 1 Processor -- start 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.841+0000 7fa80be00640 1 -- start start 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.841+0000 7fa80be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa80c1016e0 0x7fa80c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.841+0000 7fa80be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa80c1028d0 0x7fa80c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.841+0000 7fa80be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa80c103290 0x7fa80c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.841+0000 7fa80be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c10d360 con 0x7fa80c1028d0 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.841+0000 7fa80be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c10d1e0 con 0x7fa80c1016e0 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.841+0000 7fa80be00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c10d4e0 con 0x7fa80c103290 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa80c1016e0 0x7fa80c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa80c1016e0 0x7fa80c197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49430/0 (socket says 172.21.15.103:49430) 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 -- 172.21.15.103:0/2838437384 learned_addr learned my addr 172.21.15.103:0/2838437384 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:36.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80b400640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa80c103290 0x7fa80c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80a000640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa80c1028d0 0x7fa80c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 -- 172.21.15.103:0/2838437384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa80c103290 msgr2=0x7fa80c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa80c103290 0x7fa80c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 -- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa80c1028d0 msgr2=0x7fa80c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa80c1028d0 0x7fa80c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 -- 172.21.15.103:0/2838437384 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80c19f5a0 con 0x7fa80c1016e0 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80a000640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa80c1028d0 0x7fa80c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa80aa00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa80c1016e0 0x7fa80c197980 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00e9c0 tx=0x7fa7fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.842+0000 7fa803400640 1 -- 172.21.15.103:0/2838437384 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc00cde0 con 0x7fa80c1016e0 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.843+0000 7fa803400640 1 -- 172.21.15.103:0/2838437384 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7fc010910 con 0x7fa80c1016e0 2024-05-23T00:00:36.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.843+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80c19f890 con 0x7fa80c1016e0 2024-05-23T00:00:36.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.843+0000 7fa803400640 1 -- 172.21.15.103:0/2838437384 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc018a00 con 0x7fa80c1016e0 2024-05-23T00:00:36.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.843+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa80c073e10 con 0x7fa80c1016e0 2024-05-23T00:00:36.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.844+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa80c108a00 con 0x7fa80c1016e0 2024-05-23T00:00:36.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.846+0000 7fa803400640 1 -- 172.21.15.103:0/2838437384 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 137954+0+0 (secure 0 0 0) 0x7fa7fc01f020 con 0x7fa80c1016e0 2024-05-23T00:00:36.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.847+0000 7fa803400640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa7f00a7190 0x7fa7f00a9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:36.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.847+0000 7fa803400640 1 -- 172.21.15.103:0/2838437384 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa7fc0cfa10 con 0x7fa80c1016e0 2024-05-23T00:00:36.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.847+0000 7fa80a000640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa7f00a7190 0x7fa7f00a9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:36.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.848+0000 7fa80a000640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa7f00a7190 0x7fa7f00a9650 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa80c198e90 tx=0x7fa7f8005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:36.854 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:36.851+0000 7fa803400640 1 -- 172.21.15.103:0/2838437384 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa7fc09c290 con 0x7fa80c1016e0 2024-05-23T00:00:37.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.054+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa80c061cb0 con 0x7fa7f00a7190 2024-05-23T00:00:37.058 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:00:36 smithi105 sudo[69037]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:37.058 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:00:36 smithi105 sudo[69037]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:37.058 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:00:36 smithi105 sudo[69037]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:37.058 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:00:36 smithi105 sudo[69037]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:37.072 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.069+0000 7fa803400640 1 -- 172.21.15.103:0/2838437384 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339719 (secure 0 0 0) 0x7fa80c061cb0 con 0x7fa7f00a7190 2024-05-23T00:00:37.077 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:37.079 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-05-23T00:00:37.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.074+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa7f00a7190 msgr2=0x7fa7f00a9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.074+0000 7fa80be00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa7f00a7190 0x7fa7f00a9650 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa80c198e90 tx=0x7fa7f8005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:37.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.074+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa80c1016e0 msgr2=0x7fa80c197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.074+0000 7fa80be00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa80c1016e0 0x7fa80c197980 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00e9c0 tx=0x7fa7fc00ee80 comp rx=0 tx=0).stop 2024-05-23T00:00:37.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.075+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 shutdown_connections 2024-05-23T00:00:37.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.075+0000 7fa80be00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa7f00a7190 0x7fa7f00a9650 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.075+0000 7fa80be00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa80c103290 0x7fa80c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.075+0000 7fa80be00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa80c1028d0 0x7fa80c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.075+0000 7fa80be00640 1 --2- 172.21.15.103:0/2838437384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa80c1016e0 0x7fa80c197980 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.075+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 >> 172.21.15.103:0/2838437384 conn(0x7fa80c0fcc50 msgr2=0x7fa80c105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:37.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.075+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 shutdown_connections 2024-05-23T00:00:37.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.075+0000 7fa80be00640 1 -- 172.21.15.103:0/2838437384 wait complete. 2024-05-23T00:00:37.097 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-23T00:00:36.693138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190223,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1154,"num_read_kb":741,"num_write":478,"num_write_kb":4388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1181748,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":873,"ondisk_log_size":873,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357572,"kb_used_data":12556,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584252,"statfs":{"total":1151940427776,"available":1151574274048,"internally_reserved":0,"allocated":12857344,"data_stored":9055355,"data_compressed":424676,"data_compressed_allocated":577536,"data_compressed_original":3448832,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991324"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810114+0000","last_change":"2024-05-22T23:57:33.328075+0000","last_active":"2024-05-23T00:00:32.810114+0000","last_peered":"2024-05-23T00:00:32.810114+0000","last_clean":"2024-05-23T00:00:32.810114+0000","last_became_active":"2024-05-22T23:57:33.327704+0000","last_became_peered":"2024-05-22T23:57:33.327704+0000","last_unstale":"2024-05-23T00:00:32.810114+0000","last_undegraded":"2024-05-23T00:00:32.810114+0000","last_fullsized":"2024-05-23T00:00:32.810114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:42:32.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670793+0000","last_change":"2024-05-22T23:57:33.332043+0000","last_active":"2024-05-23T00:00:32.670793+0000","last_peered":"2024-05-23T00:00:32.670793+0000","last_clean":"2024-05-23T00:00:32.670793+0000","last_became_active":"2024-05-22T23:57:33.322269+0000","last_became_peered":"2024-05-22T23:57:33.322269+0000","last_unstale":"2024-05-23T00:00:32.670793+0000","last_undegraded":"2024-05-23T00:00:32.670793+0000","last_fullsized":"2024-05-23T00:00:32.670793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:40:23.542938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670609+0000","last_change":"2024-05-22T23:57:33.334993+0000","last_active":"2024-05-23T00:00:32.670609+0000","last_peered":"2024-05-23T00:00:32.670609+0000","last_clean":"2024-05-23T00:00:32.670609+0000","last_became_active":"2024-05-22T23:57:33.326446+0000","last_became_peered":"2024-05-22T23:57:33.326446+0000","last_unstale":"2024-05-23T00:00:32.670609+0000","last_undegraded":"2024-05-23T00:00:32.670609+0000","last_fullsized":"2024-05-23T00:00:32.670609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:44:45.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023391+0000","last_change":"2024-05-22T23:57:33.320181+0000","last_active":"2024-05-23T00:00:33.023391+0000","last_peered":"2024-05-23T00:00:33.023391+0000","last_clean":"2024-05-23T00:00:33.023391+0000","last_became_active":"2024-05-22T23:57:33.319630+0000","last_became_peered":"2024-05-22T23:57:33.319630+0000","last_unstale":"2024-05-23T00:00:33.023391+0000","last_undegraded":"2024-05-23T00:00:33.023391+0000","last_fullsized":"2024-05-23T00:00:33.023391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:38:51.836241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022138+0000","last_change":"2024-05-22T23:57:33.332168+0000","last_active":"2024-05-23T00:00:33.022138+0000","last_peered":"2024-05-23T00:00:33.022138+0000","last_clean":"2024-05-23T00:00:33.022138+0000","last_became_active":"2024-05-22T23:57:33.331755+0000","last_became_peered":"2024-05-22T23:57:33.331755+0000","last_unstale":"2024-05-23T00:00:33.022138+0000","last_undegraded":"2024-05-23T00:00:33.022138+0000","last_fullsized":"2024-05-23T00:00:33.022138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:28.013365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822749+0000","last_change":"2024-05-22T23:57:33.322449+0000","last_active":"2024-05-23T00:00:32.822749+0000","last_peered":"2024-05-23T00:00:32.822749+0000","last_clean":"2024-05-23T00:00:32.822749+0000","last_became_active":"2024-05-22T23:57:33.322099+0000","last_became_peered":"2024-05-22T23:57:33.322099+0000","last_unstale":"2024-05-23T00:00:32.822749+0000","last_undegraded":"2024-05-23T00:00:32.822749+0000","last_fullsized":"2024-05-23T00:00:32.822749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:38.427434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023339+0000","last_change":"2024-05-22T23:57:33.322391+0000","last_active":"2024-05-23T00:00:33.023339+0000","last_peered":"2024-05-23T00:00:33.023339+0000","last_clean":"2024-05-23T00:00:33.023339+0000","last_became_active":"2024-05-22T23:57:33.322076+0000","last_became_peered":"2024-05-22T23:57:33.322076+0000","last_unstale":"2024-05-23T00:00:33.023339+0000","last_undegraded":"2024-05-23T00:00:33.023339+0000","last_fullsized":"2024-05-23T00:00:33.023339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:45:44.857586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810020+0000","last_change":"2024-05-22T23:57:33.333005+0000","last_active":"2024-05-23T00:00:32.810020+0000","last_peered":"2024-05-23T00:00:32.810020+0000","last_clean":"2024-05-23T00:00:32.810020+0000","last_became_active":"2024-05-22T23:57:33.332669+0000","last_became_peered":"2024-05-22T23:57:33.332669+0000","last_unstale":"2024-05-23T00:00:32.810020+0000","last_undegraded":"2024-05-23T00:00:32.810020+0000","last_fullsized":"2024-05-23T00:00:32.810020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:47:25.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021594+0000","last_change":"2024-05-22T23:58:31.104194+0000","last_active":"2024-05-23T00:00:33.021594+0000","last_peered":"2024-05-23T00:00:33.021594+0000","last_clean":"2024-05-23T00:00:33.021594+0000","last_became_active":"2024-05-22T23:58:31.103788+0000","last_became_peered":"2024-05-22T23:58:31.103788+0000","last_unstale":"2024-05-23T00:00:33.021594+0000","last_undegraded":"2024-05-23T00:00:33.021594+0000","last_fullsized":"2024-05-23T00:00:33.021594+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:41:09.215669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822962+0000","last_change":"2024-05-22T23:57:33.324445+0000","last_active":"2024-05-23T00:00:32.822962+0000","last_peered":"2024-05-23T00:00:32.822962+0000","last_clean":"2024-05-23T00:00:32.822962+0000","last_became_active":"2024-05-22T23:57:33.324085+0000","last_became_peered":"2024-05-22T23:57:33.324085+0000","last_unstale":"2024-05-23T00:00:32.822962+0000","last_undegraded":"2024-05-23T00:00:32.822962+0000","last_fullsized":"2024-05-23T00:00:32.822962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:30:51.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024137+0000","last_change":"2024-05-22T23:57:33.321481+0000","last_active":"2024-05-23T00:00:33.024137+0000","last_peered":"2024-05-23T00:00:33.024137+0000","last_clean":"2024-05-23T00:00:33.024137+0000","last_became_active":"2024-05-22T23:57:33.320911+0000","last_became_peered":"2024-05-22T23:57:33.320911+0000","last_unstale":"2024-05-23T00:00:33.024137+0000","last_undegraded":"2024-05-23T00:00:33.024137+0000","last_fullsized":"2024-05-23T00:00:33.024137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:24:24.041980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023362+0000","last_change":"2024-05-22T23:57:33.318819+0000","last_active":"2024-05-23T00:00:33.023362+0000","last_peered":"2024-05-23T00:00:33.023362+0000","last_clean":"2024-05-23T00:00:33.023362+0000","last_became_active":"2024-05-22T23:57:33.318284+0000","last_became_peered":"2024-05-22T23:57:33.318284+0000","last_unstale":"2024-05-23T00:00:33.023362+0000","last_undegraded":"2024-05-23T00:00:33.023362+0000","last_fullsized":"2024-05-23T00:00:33.023362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:56.043446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024014+0000","last_change":"2024-05-22T23:57:33.321268+0000","last_active":"2024-05-23T00:00:33.024014+0000","last_peered":"2024-05-23T00:00:33.024014+0000","last_clean":"2024-05-23T00:00:33.024014+0000","last_became_active":"2024-05-22T23:57:33.320902+0000","last_became_peered":"2024-05-22T23:57:33.320902+0000","last_unstale":"2024-05-23T00:00:33.024014+0000","last_undegraded":"2024-05-23T00:00:33.024014+0000","last_fullsized":"2024-05-23T00:00:33.024014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:51:39.716541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669962+0000","last_change":"2024-05-22T23:57:33.322638+0000","last_active":"2024-05-23T00:00:32.669962+0000","last_peered":"2024-05-23T00:00:32.669962+0000","last_clean":"2024-05-23T00:00:32.669962+0000","last_became_active":"2024-05-22T23:57:33.322318+0000","last_became_peered":"2024-05-22T23:57:33.322318+0000","last_unstale":"2024-05-23T00:00:32.669962+0000","last_undegraded":"2024-05-23T00:00:32.669962+0000","last_fullsized":"2024-05-23T00:00:32.669962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:44:25.487618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669867+0000","last_change":"2024-05-22T23:57:33.325832+0000","last_active":"2024-05-23T00:00:32.669867+0000","last_peered":"2024-05-23T00:00:32.669867+0000","last_clean":"2024-05-23T00:00:32.669867+0000","last_became_active":"2024-05-22T23:57:33.325062+0000","last_became_peered":"2024-05-22T23:57:33.325062+0000","last_unstale":"2024-05-23T00:00:32.669867+0000","last_undegraded":"2024-05-23T00:00:32.669867+0000","last_fullsized":"2024-05-23T00:00:32.669867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:33:04.161351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020743+0000","last_change":"2024-05-22T23:57:33.324864+0000","last_active":"2024-05-23T00:00:33.020743+0000","last_peered":"2024-05-23T00:00:33.020743+0000","last_clean":"2024-05-23T00:00:33.020743+0000","last_became_active":"2024-05-22T23:57:33.324558+0000","last_became_peered":"2024-05-22T23:57:33.324558+0000","last_unstale":"2024-05-23T00:00:33.020743+0000","last_undegraded":"2024-05-23T00:00:33.020743+0000","last_fullsized":"2024-05-23T00:00:33.020743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:48:49.722371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809904+0000","last_change":"2024-05-22T23:57:33.337985+0000","last_active":"2024-05-23T00:00:32.809904+0000","last_peered":"2024-05-23T00:00:32.809904+0000","last_clean":"2024-05-23T00:00:32.809904+0000","last_became_active":"2024-05-22T23:57:33.337719+0000","last_became_peered":"2024-05-22T23:57:33.337719+0000","last_unstale":"2024-05-23T00:00:32.809904+0000","last_undegraded":"2024-05-23T00:00:32.809904+0000","last_fullsized":"2024-05-23T00:00:32.809904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:50:44.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.673585+0000","last_change":"2024-05-22T23:57:33.325709+0000","last_active":"2024-05-23T00:00:32.673585+0000","last_peered":"2024-05-23T00:00:32.673585+0000","last_clean":"2024-05-23T00:00:32.673585+0000","last_became_active":"2024-05-22T23:57:33.325063+0000","last_became_peered":"2024-05-22T23:57:33.325063+0000","last_unstale":"2024-05-23T00:00:32.673585+0000","last_undegraded":"2024-05-23T00:00:32.673585+0000","last_fullsized":"2024-05-23T00:00:32.673585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:39.145217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670219+0000","last_change":"2024-05-22T23:57:33.333251+0000","last_active":"2024-05-23T00:00:32.670219+0000","last_peered":"2024-05-23T00:00:32.670219+0000","last_clean":"2024-05-23T00:00:32.670219+0000","last_became_active":"2024-05-22T23:57:33.332894+0000","last_became_peered":"2024-05-22T23:57:33.332894+0000","last_unstale":"2024-05-23T00:00:32.670219+0000","last_undegraded":"2024-05-23T00:00:32.670219+0000","last_fullsized":"2024-05-23T00:00:32.670219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:13:12.082781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669594+0000","last_change":"2024-05-22T23:57:37.788060+0000","last_active":"2024-05-23T00:00:32.669594+0000","last_peered":"2024-05-23T00:00:32.669594+0000","last_clean":"2024-05-23T00:00:32.669594+0000","last_became_active":"2024-05-22T23:57:37.787656+0000","last_became_peered":"2024-05-22T23:57:37.787656+0000","last_unstale":"2024-05-23T00:00:32.669594+0000","last_undegraded":"2024-05-23T00:00:32.669594+0000","last_fullsized":"2024-05-23T00:00:32.669594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:39:13.718817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672586+0000","last_change":"2024-05-22T23:57:32.323434+0000","last_active":"2024-05-23T00:00:32.672586+0000","last_peered":"2024-05-23T00:00:32.672586+0000","last_clean":"2024-05-23T00:00:32.672586+0000","last_became_active":"2024-05-22T23:57:32.323033+0000","last_became_peered":"2024-05-22T23:57:32.323033+0000","last_unstale":"2024-05-23T00:00:32.672586+0000","last_undegraded":"2024-05-23T00:00:32.672586+0000","last_fullsized":"2024-05-23T00:00:32.672586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:15:27.676027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812037+0000","last_change":"2024-05-22T23:57:33.323336+0000","last_active":"2024-05-23T00:00:32.812037+0000","last_peered":"2024-05-23T00:00:32.812037+0000","last_clean":"2024-05-23T00:00:32.812037+0000","last_became_active":"2024-05-22T23:57:33.323026+0000","last_became_peered":"2024-05-22T23:57:33.323026+0000","last_unstale":"2024-05-23T00:00:32.812037+0000","last_undegraded":"2024-05-23T00:00:32.812037+0000","last_fullsized":"2024-05-23T00:00:32.812037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:02:20.142760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022462+0000","last_change":"2024-05-22T23:57:37.785137+0000","last_active":"2024-05-23T00:00:33.022462+0000","last_peered":"2024-05-23T00:00:33.022462+0000","last_clean":"2024-05-23T00:00:33.022462+0000","last_became_active":"2024-05-22T23:57:37.784791+0000","last_became_peered":"2024-05-22T23:57:37.784791+0000","last_unstale":"2024-05-23T00:00:33.022462+0000","last_undegraded":"2024-05-23T00:00:33.022462+0000","last_fullsized":"2024-05-23T00:00:33.022462+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:54:51.036334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812106+0000","last_change":"2024-05-22T23:57:32.332969+0000","last_active":"2024-05-23T00:00:32.812106+0000","last_peered":"2024-05-23T00:00:32.812106+0000","last_clean":"2024-05-23T00:00:32.812106+0000","last_became_active":"2024-05-22T23:57:32.332611+0000","last_became_peered":"2024-05-22T23:57:32.332611+0000","last_unstale":"2024-05-23T00:00:32.812106+0000","last_undegraded":"2024-05-23T00:00:32.812106+0000","last_fullsized":"2024-05-23T00:00:32.812106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:00:22.006355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670839+0000","last_change":"2024-05-22T23:57:33.332151+0000","last_active":"2024-05-23T00:00:32.670839+0000","last_peered":"2024-05-23T00:00:32.670839+0000","last_clean":"2024-05-23T00:00:32.670839+0000","last_became_active":"2024-05-22T23:57:33.329518+0000","last_became_peered":"2024-05-22T23:57:33.329518+0000","last_unstale":"2024-05-23T00:00:32.670839+0000","last_undegraded":"2024-05-23T00:00:32.670839+0000","last_fullsized":"2024-05-23T00:00:32.670839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:10:12.083766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022602+0000","last_change":"2024-05-22T23:57:37.787524+0000","last_active":"2024-05-23T00:00:33.022602+0000","last_peered":"2024-05-23T00:00:33.022602+0000","last_clean":"2024-05-23T00:00:33.022602+0000","last_became_active":"2024-05-22T23:57:37.787103+0000","last_became_peered":"2024-05-22T23:57:37.787103+0000","last_unstale":"2024-05-23T00:00:33.022602+0000","last_undegraded":"2024-05-23T00:00:33.022602+0000","last_fullsized":"2024-05-23T00:00:33.022602+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:19:16.963159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022671+0000","last_change":"2024-05-22T23:57:32.318469+0000","last_active":"2024-05-23T00:00:33.022671+0000","last_peered":"2024-05-23T00:00:33.022671+0000","last_clean":"2024-05-23T00:00:33.022671+0000","last_became_active":"2024-05-22T23:57:32.317640+0000","last_became_peered":"2024-05-22T23:57:32.317640+0000","last_unstale":"2024-05-23T00:00:33.022671+0000","last_undegraded":"2024-05-23T00:00:33.022671+0000","last_fullsized":"2024-05-23T00:00:33.022671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:01:55.710019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021134+0000","last_change":"2024-05-22T23:57:33.318604+0000","last_active":"2024-05-23T00:00:33.021134+0000","last_peered":"2024-05-23T00:00:33.021134+0000","last_clean":"2024-05-23T00:00:33.021134+0000","last_became_active":"2024-05-22T23:57:33.318184+0000","last_became_peered":"2024-05-22T23:57:33.318184+0000","last_unstale":"2024-05-23T00:00:33.021134+0000","last_undegraded":"2024-05-23T00:00:33.021134+0000","last_fullsized":"2024-05-23T00:00:33.021134+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:16.140799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022571+0000","last_change":"2024-05-22T23:57:37.796085+0000","last_active":"2024-05-23T00:00:33.022571+0000","last_peered":"2024-05-23T00:00:33.022571+0000","last_clean":"2024-05-23T00:00:33.022571+0000","last_became_active":"2024-05-22T23:57:37.795265+0000","last_became_peered":"2024-05-22T23:57:37.795265+0000","last_unstale":"2024-05-23T00:00:33.022571+0000","last_undegraded":"2024-05-23T00:00:33.022571+0000","last_fullsized":"2024-05-23T00:00:33.022571+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:43:55.577072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022541+0000","last_change":"2024-05-22T23:57:32.316346+0000","last_active":"2024-05-23T00:00:33.022541+0000","last_peered":"2024-05-23T00:00:33.022541+0000","last_clean":"2024-05-23T00:00:33.022541+0000","last_became_active":"2024-05-22T23:57:32.316038+0000","last_became_peered":"2024-05-22T23:57:32.316038+0000","last_unstale":"2024-05-23T00:00:33.022541+0000","last_undegraded":"2024-05-23T00:00:33.022541+0000","last_fullsized":"2024-05-23T00:00:33.022541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:57:41.987250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819205+0000","last_change":"2024-05-22T23:57:33.329341+0000","last_active":"2024-05-23T00:00:32.819205+0000","last_peered":"2024-05-23T00:00:32.819205+0000","last_clean":"2024-05-23T00:00:32.819205+0000","last_became_active":"2024-05-22T23:57:33.329009+0000","last_became_peered":"2024-05-22T23:57:33.329009+0000","last_unstale":"2024-05-23T00:00:32.819205+0000","last_undegraded":"2024-05-23T00:00:32.819205+0000","last_fullsized":"2024-05-23T00:00:32.819205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:53:07.591497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670094+0000","last_change":"2024-05-22T23:57:37.779097+0000","last_active":"2024-05-23T00:00:32.670094+0000","last_peered":"2024-05-23T00:00:32.670094+0000","last_clean":"2024-05-23T00:00:32.670094+0000","last_became_active":"2024-05-22T23:57:37.778564+0000","last_became_peered":"2024-05-22T23:57:37.778564+0000","last_unstale":"2024-05-23T00:00:32.670094+0000","last_undegraded":"2024-05-23T00:00:32.670094+0000","last_fullsized":"2024-05-23T00:00:32.670094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:09:16.351947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":205,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.298705+0000","last_change":"2024-05-22T23:58:31.109013+0000","last_active":"2024-05-23T00:00:33.298705+0000","last_peered":"2024-05-23T00:00:33.298705+0000","last_clean":"2024-05-23T00:00:33.298705+0000","last_became_active":"2024-05-22T23:58:31.103858+0000","last_became_peered":"2024-05-22T23:58:31.103858+0000","last_unstale":"2024-05-23T00:00:33.298705+0000","last_undegraded":"2024-05-23T00:00:33.298705+0000","last_fullsized":"2024-05-23T00:00:33.298705+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:20:08.783189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":153,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022157+0000","last_change":"2024-05-22T23:57:33.324587+0000","last_active":"2024-05-23T00:00:33.022157+0000","last_peered":"2024-05-23T00:00:33.022157+0000","last_clean":"2024-05-23T00:00:33.022157+0000","last_became_active":"2024-05-22T23:57:33.323972+0000","last_became_peered":"2024-05-22T23:57:33.323972+0000","last_unstale":"2024-05-23T00:00:33.022157+0000","last_undegraded":"2024-05-23T00:00:33.022157+0000","last_fullsized":"2024-05-23T00:00:33.022157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:23:52.551699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670431+0000","last_change":"2024-05-22T23:57:37.791726+0000","last_active":"2024-05-23T00:00:32.670431+0000","last_peered":"2024-05-23T00:00:32.670431+0000","last_clean":"2024-05-23T00:00:32.670431+0000","last_became_active":"2024-05-22T23:57:37.791185+0000","last_became_peered":"2024-05-22T23:57:37.791185+0000","last_unstale":"2024-05-23T00:00:32.670431+0000","last_undegraded":"2024-05-23T00:00:32.670431+0000","last_fullsized":"2024-05-23T00:00:32.670431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:55:30.994272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822489+0000","last_change":"2024-05-22T23:58:31.116594+0000","last_active":"2024-05-23T00:00:32.822489+0000","last_peered":"2024-05-23T00:00:32.822489+0000","last_clean":"2024-05-23T00:00:32.822489+0000","last_became_active":"2024-05-22T23:58:31.113117+0000","last_became_peered":"2024-05-22T23:58:31.113117+0000","last_unstale":"2024-05-23T00:00:32.822489+0000","last_undegraded":"2024-05-23T00:00:32.822489+0000","last_fullsized":"2024-05-23T00:00:32.822489+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:05:30.758638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819100+0000","last_change":"2024-05-22T23:58:31.116203+0000","last_active":"2024-05-23T00:00:32.819100+0000","last_peered":"2024-05-23T00:00:32.819100+0000","last_clean":"2024-05-23T00:00:32.819100+0000","last_became_active":"2024-05-22T23:58:31.115881+0000","last_became_peered":"2024-05-22T23:58:31.115881+0000","last_unstale":"2024-05-23T00:00:32.819100+0000","last_undegraded":"2024-05-23T00:00:32.819100+0000","last_fullsized":"2024-05-23T00:00:32.819100+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:39:31.247426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809889+0000","last_change":"2024-05-22T23:57:37.787910+0000","last_active":"2024-05-23T00:00:32.809889+0000","last_peered":"2024-05-23T00:00:32.809889+0000","last_clean":"2024-05-23T00:00:32.809889+0000","last_became_active":"2024-05-22T23:57:37.787465+0000","last_became_peered":"2024-05-22T23:57:37.787465+0000","last_unstale":"2024-05-23T00:00:32.809889+0000","last_undegraded":"2024-05-23T00:00:32.809889+0000","last_fullsized":"2024-05-23T00:00:32.809889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:54:59.202855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023402+0000","last_change":"2024-05-22T23:57:32.329365+0000","last_active":"2024-05-23T00:00:33.023402+0000","last_peered":"2024-05-23T00:00:33.023402+0000","last_clean":"2024-05-23T00:00:33.023402+0000","last_became_active":"2024-05-22T23:57:32.327864+0000","last_became_peered":"2024-05-22T23:57:32.327864+0000","last_unstale":"2024-05-23T00:00:33.023402+0000","last_undegraded":"2024-05-23T00:00:33.023402+0000","last_fullsized":"2024-05-23T00:00:33.023402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:26.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809992+0000","last_change":"2024-05-22T23:57:33.313930+0000","last_active":"2024-05-23T00:00:32.809992+0000","last_peered":"2024-05-23T00:00:32.809992+0000","last_clean":"2024-05-23T00:00:32.809992+0000","last_became_active":"2024-05-22T23:57:33.313612+0000","last_became_peered":"2024-05-22T23:57:33.313612+0000","last_unstale":"2024-05-23T00:00:32.809992+0000","last_undegraded":"2024-05-23T00:00:32.809992+0000","last_fullsized":"2024-05-23T00:00:32.809992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:45:25.800338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670644+0000","last_change":"2024-05-22T23:57:37.791537+0000","last_active":"2024-05-23T00:00:32.670644+0000","last_peered":"2024-05-23T00:00:32.670644+0000","last_clean":"2024-05-23T00:00:32.670644+0000","last_became_active":"2024-05-22T23:57:37.791052+0000","last_became_peered":"2024-05-22T23:57:37.791052+0000","last_unstale":"2024-05-23T00:00:32.670644+0000","last_undegraded":"2024-05-23T00:00:32.670644+0000","last_fullsized":"2024-05-23T00:00:32.670644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:18:11.759840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818596+0000","last_change":"2024-05-22T23:57:32.315907+0000","last_active":"2024-05-23T00:00:32.818596+0000","last_peered":"2024-05-23T00:00:32.818596+0000","last_clean":"2024-05-23T00:00:32.818596+0000","last_became_active":"2024-05-22T23:57:32.315576+0000","last_became_peered":"2024-05-22T23:57:32.315576+0000","last_unstale":"2024-05-23T00:00:32.818596+0000","last_undegraded":"2024-05-23T00:00:32.818596+0000","last_fullsized":"2024-05-23T00:00:32.818596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:45:48.998377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671704+0000","last_change":"2024-05-22T23:57:33.338681+0000","last_active":"2024-05-23T00:00:32.671704+0000","last_peered":"2024-05-23T00:00:32.671704+0000","last_clean":"2024-05-23T00:00:32.671704+0000","last_became_active":"2024-05-22T23:57:33.334705+0000","last_became_peered":"2024-05-22T23:57:33.334705+0000","last_unstale":"2024-05-23T00:00:32.671704+0000","last_undegraded":"2024-05-23T00:00:32.671704+0000","last_fullsized":"2024-05-23T00:00:32.671704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:22.135137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810677+0000","last_change":"2024-05-22T23:57:37.780470+0000","last_active":"2024-05-23T00:00:32.810677+0000","last_peered":"2024-05-23T00:00:32.810677+0000","last_clean":"2024-05-23T00:00:32.810677+0000","last_became_active":"2024-05-22T23:57:37.779110+0000","last_became_peered":"2024-05-22T23:57:37.779110+0000","last_unstale":"2024-05-23T00:00:32.810677+0000","last_undegraded":"2024-05-23T00:00:32.810677+0000","last_fullsized":"2024-05-23T00:00:32.810677+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:55:45.486862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":183,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822985+0000","last_change":"2024-05-22T23:57:32.322489+0000","last_active":"2024-05-23T00:00:32.822985+0000","last_peered":"2024-05-23T00:00:32.822985+0000","last_clean":"2024-05-23T00:00:32.822985+0000","last_became_active":"2024-05-22T23:57:32.322125+0000","last_became_peered":"2024-05-22T23:57:32.322125+0000","last_unstale":"2024-05-23T00:00:32.822985+0000","last_undegraded":"2024-05-23T00:00:32.822985+0000","last_fullsized":"2024-05-23T00:00:32.822985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:11:47.707506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021521+0000","last_change":"2024-05-22T23:57:33.322417+0000","last_active":"2024-05-23T00:00:33.021521+0000","last_peered":"2024-05-23T00:00:33.021521+0000","last_clean":"2024-05-23T00:00:33.021521+0000","last_became_active":"2024-05-22T23:57:33.322102+0000","last_became_peered":"2024-05-22T23:57:33.322102+0000","last_unstale":"2024-05-23T00:00:33.021521+0000","last_undegraded":"2024-05-23T00:00:33.021521+0000","last_fullsized":"2024-05-23T00:00:33.021521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:26:43.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671629+0000","last_change":"2024-05-22T23:57:37.784424+0000","last_active":"2024-05-23T00:00:32.671629+0000","last_peered":"2024-05-23T00:00:32.671629+0000","last_clean":"2024-05-23T00:00:32.671629+0000","last_became_active":"2024-05-22T23:57:37.783949+0000","last_became_peered":"2024-05-22T23:57:37.783949+0000","last_unstale":"2024-05-23T00:00:32.671629+0000","last_undegraded":"2024-05-23T00:00:32.671629+0000","last_fullsized":"2024-05-23T00:00:32.671629+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:52:23.763273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023938+0000","last_change":"2024-05-22T23:58:31.110764+0000","last_active":"2024-05-23T00:00:33.023938+0000","last_peered":"2024-05-23T00:00:33.023938+0000","last_clean":"2024-05-23T00:00:33.023938+0000","last_became_active":"2024-05-22T23:58:31.103962+0000","last_became_peered":"2024-05-22T23:58:31.103962+0000","last_unstale":"2024-05-23T00:00:33.023938+0000","last_undegraded":"2024-05-23T00:00:33.023938+0000","last_fullsized":"2024-05-23T00:00:33.023938+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:14:24.248179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670954+0000","last_change":"2024-05-22T23:57:33.324237+0000","last_active":"2024-05-23T00:00:32.670954+0000","last_peered":"2024-05-23T00:00:32.670954+0000","last_clean":"2024-05-23T00:00:32.670954+0000","last_became_active":"2024-05-22T23:57:33.323870+0000","last_became_peered":"2024-05-22T23:57:33.323870+0000","last_unstale":"2024-05-23T00:00:32.670954+0000","last_undegraded":"2024-05-23T00:00:32.670954+0000","last_fullsized":"2024-05-23T00:00:32.670954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:41:29.328104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022755+0000","last_change":"2024-05-22T23:57:37.787795+0000","last_active":"2024-05-23T00:00:33.022755+0000","last_peered":"2024-05-23T00:00:33.022755+0000","last_clean":"2024-05-23T00:00:33.022755+0000","last_became_active":"2024-05-22T23:57:37.787210+0000","last_became_peered":"2024-05-22T23:57:37.787210+0000","last_unstale":"2024-05-23T00:00:33.022755+0000","last_undegraded":"2024-05-23T00:00:33.022755+0000","last_fullsized":"2024-05-23T00:00:33.022755+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:19:04.616758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819178+0000","last_change":"2024-05-22T23:57:32.321415+0000","last_active":"2024-05-23T00:00:32.819178+0000","last_peered":"2024-05-23T00:00:32.819178+0000","last_clean":"2024-05-23T00:00:32.819178+0000","last_became_active":"2024-05-22T23:57:32.321055+0000","last_became_peered":"2024-05-22T23:57:32.321055+0000","last_unstale":"2024-05-23T00:00:32.819178+0000","last_undegraded":"2024-05-23T00:00:32.819178+0000","last_fullsized":"2024-05-23T00:00:32.819178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:28:58.577548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021090+0000","last_change":"2024-05-22T23:58:31.116991+0000","last_active":"2024-05-23T00:00:33.021090+0000","last_peered":"2024-05-23T00:00:33.021090+0000","last_clean":"2024-05-23T00:00:33.021090+0000","last_became_active":"2024-05-22T23:58:31.116526+0000","last_became_peered":"2024-05-22T23:58:31.116526+0000","last_unstale":"2024-05-23T00:00:33.021090+0000","last_undegraded":"2024-05-23T00:00:33.021090+0000","last_fullsized":"2024-05-23T00:00:33.021090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:51:23.350871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671835+0000","last_change":"2024-05-22T23:57:37.781367+0000","last_active":"2024-05-23T00:00:32.671835+0000","last_peered":"2024-05-23T00:00:32.671835+0000","last_clean":"2024-05-23T00:00:32.671835+0000","last_became_active":"2024-05-22T23:57:37.780955+0000","last_became_peered":"2024-05-22T23:57:37.780955+0000","last_unstale":"2024-05-23T00:00:32.671835+0000","last_undegraded":"2024-05-23T00:00:32.671835+0000","last_fullsized":"2024-05-23T00:00:32.671835+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:36:18.962940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":380,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.298872+0000","last_change":"2024-05-22T23:57:32.313873+0000","last_active":"2024-05-23T00:00:33.298872+0000","last_peered":"2024-05-23T00:00:33.298872+0000","last_clean":"2024-05-23T00:00:33.298872+0000","last_became_active":"2024-05-22T23:57:32.313388+0000","last_became_peered":"2024-05-22T23:57:32.313388+0000","last_unstale":"2024-05-23T00:00:33.298872+0000","last_undegraded":"2024-05-23T00:00:33.298872+0000","last_fullsized":"2024-05-23T00:00:33.298872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:16.383384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69043,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":308,"num_read_kb":221,"num_write":43,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022567+0000","last_change":"2024-05-22T23:57:33.318751+0000","last_active":"2024-05-23T00:00:33.022567+0000","last_peered":"2024-05-23T00:00:33.022567+0000","last_clean":"2024-05-23T00:00:33.022567+0000","last_became_active":"2024-05-22T23:57:33.317710+0000","last_became_peered":"2024-05-22T23:57:33.317710+0000","last_unstale":"2024-05-23T00:00:33.022567+0000","last_undegraded":"2024-05-23T00:00:33.022567+0000","last_fullsized":"2024-05-23T00:00:33.022567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:57:50.852615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819509+0000","last_change":"2024-05-22T23:57:37.790073+0000","last_active":"2024-05-23T00:00:32.819509+0000","last_peered":"2024-05-23T00:00:32.819509+0000","last_clean":"2024-05-23T00:00:32.819509+0000","last_became_active":"2024-05-22T23:57:37.789477+0000","last_became_peered":"2024-05-22T23:57:37.789477+0000","last_unstale":"2024-05-23T00:00:32.819509+0000","last_undegraded":"2024-05-23T00:00:32.819509+0000","last_fullsized":"2024-05-23T00:00:32.819509+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:11.003201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670178+0000","last_change":"2024-05-22T23:57:32.329846+0000","last_active":"2024-05-23T00:00:32.670178+0000","last_peered":"2024-05-23T00:00:32.670178+0000","last_clean":"2024-05-23T00:00:32.670178+0000","last_became_active":"2024-05-22T23:57:32.329414+0000","last_became_peered":"2024-05-22T23:57:32.329414+0000","last_unstale":"2024-05-23T00:00:32.670178+0000","last_undegraded":"2024-05-23T00:00:32.670178+0000","last_fullsized":"2024-05-23T00:00:32.670178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:15:38.176790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023754+0000","last_change":"2024-05-22T23:57:33.333298+0000","last_active":"2024-05-23T00:00:33.023754+0000","last_peered":"2024-05-23T00:00:33.023754+0000","last_clean":"2024-05-23T00:00:33.023754+0000","last_became_active":"2024-05-22T23:57:33.332934+0000","last_became_peered":"2024-05-22T23:57:33.332934+0000","last_unstale":"2024-05-23T00:00:33.023754+0000","last_undegraded":"2024-05-23T00:00:33.023754+0000","last_fullsized":"2024-05-23T00:00:33.023754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:06:00.427627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809610+0000","last_change":"2024-05-22T23:57:37.795628+0000","last_active":"2024-05-23T00:00:32.809610+0000","last_peered":"2024-05-23T00:00:32.809610+0000","last_clean":"2024-05-23T00:00:32.809610+0000","last_became_active":"2024-05-22T23:57:37.795208+0000","last_became_peered":"2024-05-22T23:57:37.795208+0000","last_unstale":"2024-05-23T00:00:32.809610+0000","last_undegraded":"2024-05-23T00:00:32.809610+0000","last_fullsized":"2024-05-23T00:00:32.809610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:03:12.684474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818880+0000","last_change":"2024-05-22T23:57:32.334039+0000","last_active":"2024-05-23T00:00:32.818880+0000","last_peered":"2024-05-23T00:00:32.818880+0000","last_clean":"2024-05-23T00:00:32.818880+0000","last_became_active":"2024-05-22T23:57:32.333709+0000","last_became_peered":"2024-05-22T23:57:32.333709+0000","last_unstale":"2024-05-23T00:00:32.818880+0000","last_undegraded":"2024-05-23T00:00:32.818880+0000","last_fullsized":"2024-05-23T00:00:32.818880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:41:43.319094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023784+0000","last_change":"2024-05-22T23:57:33.328056+0000","last_active":"2024-05-23T00:00:33.023784+0000","last_peered":"2024-05-23T00:00:33.023784+0000","last_clean":"2024-05-23T00:00:33.023784+0000","last_became_active":"2024-05-22T23:57:33.326559+0000","last_became_peered":"2024-05-22T23:57:33.326559+0000","last_unstale":"2024-05-23T00:00:33.023784+0000","last_undegraded":"2024-05-23T00:00:33.023784+0000","last_fullsized":"2024-05-23T00:00:33.023784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:06.463837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672875+0000","last_change":"2024-05-22T23:57:37.795892+0000","last_active":"2024-05-23T00:00:32.672875+0000","last_peered":"2024-05-23T00:00:32.672875+0000","last_clean":"2024-05-23T00:00:32.672875+0000","last_became_active":"2024-05-22T23:57:37.792246+0000","last_became_peered":"2024-05-22T23:57:37.792246+0000","last_unstale":"2024-05-23T00:00:32.672875+0000","last_undegraded":"2024-05-23T00:00:32.672875+0000","last_fullsized":"2024-05-23T00:00:32.672875+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:18.462267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":190,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.560988+0000","last_change":"2024-05-22T23:57:32.319379+0000","last_active":"2024-05-23T00:00:33.560988+0000","last_peered":"2024-05-23T00:00:33.560988+0000","last_clean":"2024-05-23T00:00:33.560988+0000","last_became_active":"2024-05-22T23:57:32.318803+0000","last_became_peered":"2024-05-22T23:57:32.318803+0000","last_unstale":"2024-05-23T00:00:33.560988+0000","last_undegraded":"2024-05-23T00:00:33.560988+0000","last_fullsized":"2024-05-23T00:00:33.560988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:59:15.689119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1509,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023586+0000","last_change":"2024-05-22T23:57:33.325294+0000","last_active":"2024-05-23T00:00:33.023586+0000","last_peered":"2024-05-23T00:00:33.023586+0000","last_clean":"2024-05-23T00:00:33.023586+0000","last_became_active":"2024-05-22T23:57:33.324339+0000","last_became_peered":"2024-05-22T23:57:33.324339+0000","last_unstale":"2024-05-23T00:00:33.023586+0000","last_undegraded":"2024-05-23T00:00:33.023586+0000","last_fullsized":"2024-05-23T00:00:33.023586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:29:47.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024152+0000","last_change":"2024-05-22T23:57:37.797961+0000","last_active":"2024-05-23T00:00:33.024152+0000","last_peered":"2024-05-23T00:00:33.024152+0000","last_clean":"2024-05-23T00:00:33.024152+0000","last_became_active":"2024-05-22T23:57:37.797532+0000","last_became_peered":"2024-05-22T23:57:37.797532+0000","last_unstale":"2024-05-23T00:00:33.024152+0000","last_undegraded":"2024-05-23T00:00:33.024152+0000","last_fullsized":"2024-05-23T00:00:33.024152+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:51:12.997674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022065+0000","last_change":"2024-05-22T23:57:32.317434+0000","last_active":"2024-05-23T00:00:33.022065+0000","last_peered":"2024-05-23T00:00:33.022065+0000","last_clean":"2024-05-23T00:00:33.022065+0000","last_became_active":"2024-05-22T23:57:32.316987+0000","last_became_peered":"2024-05-22T23:57:32.316987+0000","last_unstale":"2024-05-23T00:00:33.022065+0000","last_undegraded":"2024-05-23T00:00:33.022065+0000","last_fullsized":"2024-05-23T00:00:33.022065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:46:50.306756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818865+0000","last_change":"2024-05-22T23:57:33.328670+0000","last_active":"2024-05-23T00:00:32.818865+0000","last_peered":"2024-05-23T00:00:32.818865+0000","last_clean":"2024-05-23T00:00:32.818865+0000","last_became_active":"2024-05-22T23:57:33.328294+0000","last_became_peered":"2024-05-22T23:57:33.328294+0000","last_unstale":"2024-05-23T00:00:32.818865+0000","last_undegraded":"2024-05-23T00:00:32.818865+0000","last_fullsized":"2024-05-23T00:00:32.818865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:09:10.807844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818693+0000","last_change":"2024-05-22T23:57:37.788885+0000","last_active":"2024-05-23T00:00:32.818693+0000","last_peered":"2024-05-23T00:00:32.818693+0000","last_clean":"2024-05-23T00:00:32.818693+0000","last_became_active":"2024-05-22T23:57:37.788145+0000","last_became_peered":"2024-05-22T23:57:37.788145+0000","last_unstale":"2024-05-23T00:00:32.818693+0000","last_undegraded":"2024-05-23T00:00:32.818693+0000","last_fullsized":"2024-05-23T00:00:32.818693+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:14:50.027785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022537+0000","last_change":"2024-05-22T23:57:32.317855+0000","last_active":"2024-05-23T00:00:33.022537+0000","last_peered":"2024-05-23T00:00:33.022537+0000","last_clean":"2024-05-23T00:00:33.022537+0000","last_became_active":"2024-05-22T23:57:32.317087+0000","last_became_peered":"2024-05-22T23:57:32.317087+0000","last_unstale":"2024-05-23T00:00:33.022537+0000","last_undegraded":"2024-05-23T00:00:33.022537+0000","last_fullsized":"2024-05-23T00:00:33.022537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:49:40.486222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809847+0000","last_change":"2024-05-22T23:57:33.323897+0000","last_active":"2024-05-23T00:00:32.809847+0000","last_peered":"2024-05-23T00:00:32.809847+0000","last_clean":"2024-05-23T00:00:32.809847+0000","last_became_active":"2024-05-22T23:57:33.323567+0000","last_became_peered":"2024-05-22T23:57:33.323567+0000","last_unstale":"2024-05-23T00:00:32.809847+0000","last_undegraded":"2024-05-23T00:00:32.809847+0000","last_fullsized":"2024-05-23T00:00:32.809847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:17:48.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022150+0000","last_change":"2024-05-22T23:57:38.218313+0000","last_active":"2024-05-23T00:00:33.022150+0000","last_peered":"2024-05-23T00:00:33.022150+0000","last_clean":"2024-05-23T00:00:33.022150+0000","last_became_active":"2024-05-22T23:57:38.217756+0000","last_became_peered":"2024-05-22T23:57:38.217756+0000","last_unstale":"2024-05-23T00:00:33.022150+0000","last_undegraded":"2024-05-23T00:00:33.022150+0000","last_fullsized":"2024-05-23T00:00:33.022150+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:07:55.122260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672176+0000","last_change":"2024-05-22T23:57:32.325729+0000","last_active":"2024-05-23T00:00:32.672176+0000","last_peered":"2024-05-23T00:00:32.672176+0000","last_clean":"2024-05-23T00:00:32.672176+0000","last_became_active":"2024-05-22T23:57:32.325383+0000","last_became_peered":"2024-05-22T23:57:32.325383+0000","last_unstale":"2024-05-23T00:00:32.672176+0000","last_undegraded":"2024-05-23T00:00:32.672176+0000","last_fullsized":"2024-05-23T00:00:32.672176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:57.978806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822720+0000","last_change":"2024-05-22T23:57:33.337499+0000","last_active":"2024-05-23T00:00:32.822720+0000","last_peered":"2024-05-23T00:00:32.822720+0000","last_clean":"2024-05-23T00:00:32.822720+0000","last_became_active":"2024-05-22T23:57:33.337271+0000","last_became_peered":"2024-05-22T23:57:33.337271+0000","last_unstale":"2024-05-23T00:00:32.822720+0000","last_undegraded":"2024-05-23T00:00:32.822720+0000","last_fullsized":"2024-05-23T00:00:32.822720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:32:39.133984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024205+0000","last_change":"2024-05-22T23:57:37.792707+0000","last_active":"2024-05-23T00:00:33.024205+0000","last_peered":"2024-05-23T00:00:33.024205+0000","last_clean":"2024-05-23T00:00:33.024205+0000","last_became_active":"2024-05-22T23:57:37.792209+0000","last_became_peered":"2024-05-22T23:57:37.792209+0000","last_unstale":"2024-05-23T00:00:33.024205+0000","last_undegraded":"2024-05-23T00:00:33.024205+0000","last_fullsized":"2024-05-23T00:00:33.024205+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:36:34.089424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809543+0000","last_change":"2024-05-22T23:58:31.112612+0000","last_active":"2024-05-23T00:00:32.809543+0000","last_peered":"2024-05-23T00:00:32.809543+0000","last_clean":"2024-05-23T00:00:32.809543+0000","last_became_active":"2024-05-22T23:58:31.112181+0000","last_became_peered":"2024-05-22T23:58:31.112181+0000","last_unstale":"2024-05-23T00:00:32.809543+0000","last_undegraded":"2024-05-23T00:00:32.809543+0000","last_fullsized":"2024-05-23T00:00:32.809543+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:25:18.717294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822488+0000","last_change":"2024-05-22T23:57:33.328129+0000","last_active":"2024-05-23T00:00:32.822488+0000","last_peered":"2024-05-23T00:00:32.822488+0000","last_clean":"2024-05-23T00:00:32.822488+0000","last_became_active":"2024-05-22T23:57:33.327732+0000","last_became_peered":"2024-05-22T23:57:33.327732+0000","last_unstale":"2024-05-23T00:00:32.822488+0000","last_undegraded":"2024-05-23T00:00:32.822488+0000","last_fullsized":"2024-05-23T00:00:32.822488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:25:14.105235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024320+0000","last_change":"2024-05-22T23:57:38.220884+0000","last_active":"2024-05-23T00:00:33.024320+0000","last_peered":"2024-05-23T00:00:33.024320+0000","last_clean":"2024-05-23T00:00:33.024320+0000","last_became_active":"2024-05-22T23:57:38.219365+0000","last_became_peered":"2024-05-22T23:57:38.219365+0000","last_unstale":"2024-05-23T00:00:33.024320+0000","last_undegraded":"2024-05-23T00:00:33.024320+0000","last_fullsized":"2024-05-23T00:00:33.024320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:54:56.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818771+0000","last_change":"2024-05-22T23:57:33.328272+0000","last_active":"2024-05-23T00:00:32.818771+0000","last_peered":"2024-05-23T00:00:32.818771+0000","last_clean":"2024-05-23T00:00:32.818771+0000","last_became_active":"2024-05-22T23:57:33.327915+0000","last_became_peered":"2024-05-22T23:57:33.327915+0000","last_unstale":"2024-05-23T00:00:32.818771+0000","last_undegraded":"2024-05-23T00:00:32.818771+0000","last_fullsized":"2024-05-23T00:00:32.818771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:59:09.085649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":185,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.031020+0000","last_change":"2024-05-22T23:57:32.331979+0000","last_active":"2024-05-23T00:00:33.031020+0000","last_peered":"2024-05-23T00:00:33.031020+0000","last_clean":"2024-05-23T00:00:33.031020+0000","last_became_active":"2024-05-22T23:57:32.330983+0000","last_became_peered":"2024-05-22T23:57:32.330983+0000","last_unstale":"2024-05-23T00:00:33.031020+0000","last_undegraded":"2024-05-23T00:00:33.031020+0000","last_fullsized":"2024-05-23T00:00:33.031020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:07:19.137652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023940+0000","last_change":"2024-05-22T23:57:37.787935+0000","last_active":"2024-05-23T00:00:33.023940+0000","last_peered":"2024-05-23T00:00:33.023940+0000","last_clean":"2024-05-23T00:00:33.023940+0000","last_became_active":"2024-05-22T23:57:37.787495+0000","last_became_peered":"2024-05-22T23:57:37.787495+0000","last_unstale":"2024-05-23T00:00:33.023940+0000","last_undegraded":"2024-05-23T00:00:33.023940+0000","last_fullsized":"2024-05-23T00:00:33.023940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:37.682711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810969+0000","last_change":"2024-05-22T23:57:33.329451+0000","last_active":"2024-05-23T00:00:32.810969+0000","last_peered":"2024-05-23T00:00:32.810969+0000","last_clean":"2024-05-23T00:00:32.810969+0000","last_became_active":"2024-05-22T23:57:33.329107+0000","last_became_peered":"2024-05-22T23:57:33.329107+0000","last_unstale":"2024-05-23T00:00:32.810969+0000","last_undegraded":"2024-05-23T00:00:32.810969+0000","last_fullsized":"2024-05-23T00:00:32.810969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:44:50.198040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022720+0000","last_change":"2024-05-22T23:57:32.321873+0000","last_active":"2024-05-23T00:00:33.022720+0000","last_peered":"2024-05-23T00:00:33.022720+0000","last_clean":"2024-05-23T00:00:33.022720+0000","last_became_active":"2024-05-22T23:57:32.321550+0000","last_became_peered":"2024-05-22T23:57:32.321550+0000","last_unstale":"2024-05-23T00:00:33.022720+0000","last_undegraded":"2024-05-23T00:00:33.022720+0000","last_fullsized":"2024-05-23T00:00:33.022720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:30:17.479238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":363,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023381+0000","last_change":"2024-05-22T23:56:11.327788+0000","last_active":"2024-05-23T00:00:33.023381+0000","last_peered":"2024-05-23T00:00:33.023381+0000","last_clean":"2024-05-23T00:00:33.023381+0000","last_became_active":"2024-05-22T23:56:11.327232+0000","last_became_peered":"2024-05-22T23:56:11.327232+0000","last_unstale":"2024-05-23T00:00:33.023381+0000","last_undegraded":"2024-05-23T00:00:33.023381+0000","last_fullsized":"2024-05-23T00:00:33.023381+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_clean_scrub_stamp":"2024-05-22T23:52:16.719643+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:26:13.077470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819294+0000","last_change":"2024-05-22T23:57:38.219326+0000","last_active":"2024-05-23T00:00:32.819294+0000","last_peered":"2024-05-23T00:00:32.819294+0000","last_clean":"2024-05-23T00:00:32.819294+0000","last_became_active":"2024-05-22T23:57:38.218835+0000","last_became_peered":"2024-05-22T23:57:38.218835+0000","last_unstale":"2024-05-23T00:00:32.819294+0000","last_undegraded":"2024-05-23T00:00:32.819294+0000","last_fullsized":"2024-05-23T00:00:32.819294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:29.722862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023039+0000","last_change":"2024-05-22T23:57:33.316760+0000","last_active":"2024-05-23T00:00:33.023039+0000","last_peered":"2024-05-23T00:00:33.023039+0000","last_clean":"2024-05-23T00:00:33.023039+0000","last_became_active":"2024-05-22T23:57:33.316328+0000","last_became_peered":"2024-05-22T23:57:33.316328+0000","last_unstale":"2024-05-23T00:00:33.023039+0000","last_undegraded":"2024-05-23T00:00:33.023039+0000","last_fullsized":"2024-05-23T00:00:33.023039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:53:11.592134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023557+0000","last_change":"2024-05-22T23:57:32.319154+0000","last_active":"2024-05-23T00:00:33.023557+0000","last_peered":"2024-05-23T00:00:33.023557+0000","last_clean":"2024-05-23T00:00:33.023557+0000","last_became_active":"2024-05-22T23:57:32.318808+0000","last_became_peered":"2024-05-22T23:57:32.318808+0000","last_unstale":"2024-05-23T00:00:33.023557+0000","last_undegraded":"2024-05-23T00:00:33.023557+0000","last_fullsized":"2024-05-23T00:00:33.023557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:37:47.271767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809550+0000","last_change":"2024-05-22T23:57:37.791464+0000","last_active":"2024-05-23T00:00:32.809550+0000","last_peered":"2024-05-23T00:00:32.809550+0000","last_clean":"2024-05-23T00:00:32.809550+0000","last_became_active":"2024-05-22T23:57:37.791044+0000","last_became_peered":"2024-05-22T23:57:37.791044+0000","last_unstale":"2024-05-23T00:00:32.809550+0000","last_undegraded":"2024-05-23T00:00:32.809550+0000","last_fullsized":"2024-05-23T00:00:32.809550+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:47:00.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022492+0000","last_change":"2024-05-22T23:57:32.322740+0000","last_active":"2024-05-23T00:00:33.022492+0000","last_peered":"2024-05-23T00:00:33.022492+0000","last_clean":"2024-05-23T00:00:33.022492+0000","last_became_active":"2024-05-22T23:57:32.322425+0000","last_became_peered":"2024-05-22T23:57:32.322425+0000","last_unstale":"2024-05-23T00:00:33.022492+0000","last_undegraded":"2024-05-23T00:00:33.022492+0000","last_fullsized":"2024-05-23T00:00:33.022492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:05:10.522462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023588+0000","last_change":"2024-05-22T23:57:33.324667+0000","last_active":"2024-05-23T00:00:33.023588+0000","last_peered":"2024-05-23T00:00:33.023588+0000","last_clean":"2024-05-23T00:00:33.023588+0000","last_became_active":"2024-05-22T23:57:33.324251+0000","last_became_peered":"2024-05-22T23:57:33.324251+0000","last_unstale":"2024-05-23T00:00:33.023588+0000","last_undegraded":"2024-05-23T00:00:33.023588+0000","last_fullsized":"2024-05-23T00:00:33.023588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:49:57.203996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809807+0000","last_change":"2024-05-22T23:57:38.222346+0000","last_active":"2024-05-23T00:00:32.809807+0000","last_peered":"2024-05-23T00:00:32.809807+0000","last_clean":"2024-05-23T00:00:32.809807+0000","last_became_active":"2024-05-22T23:57:38.220248+0000","last_became_peered":"2024-05-22T23:57:38.220248+0000","last_unstale":"2024-05-23T00:00:32.809807+0000","last_undegraded":"2024-05-23T00:00:32.809807+0000","last_fullsized":"2024-05-23T00:00:32.809807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:26:44.026641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.963348+0000","last_change":"2024-05-22T23:58:31.126056+0000","last_active":"2024-05-23T00:00:32.963348+0000","last_peered":"2024-05-23T00:00:32.963348+0000","last_clean":"2024-05-23T00:00:32.963348+0000","last_became_active":"2024-05-22T23:58:31.117186+0000","last_became_peered":"2024-05-22T23:58:31.117186+0000","last_unstale":"2024-05-23T00:00:32.963348+0000","last_undegraded":"2024-05-23T00:00:32.963348+0000","last_fullsized":"2024-05-23T00:00:32.963348+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:21:34.156655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":79,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671990+0000","last_change":"2024-05-22T23:58:31.116649+0000","last_active":"2024-05-23T00:00:32.671990+0000","last_peered":"2024-05-23T00:00:32.671990+0000","last_clean":"2024-05-23T00:00:32.671990+0000","last_became_active":"2024-05-22T23:58:31.116009+0000","last_became_peered":"2024-05-22T23:58:31.116009+0000","last_unstale":"2024-05-23T00:00:32.671990+0000","last_undegraded":"2024-05-23T00:00:32.671990+0000","last_fullsized":"2024-05-23T00:00:32.671990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:00:29.964565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669481+0000","last_change":"2024-05-22T23:57:37.778524+0000","last_active":"2024-05-23T00:00:32.669481+0000","last_peered":"2024-05-23T00:00:32.669481+0000","last_clean":"2024-05-23T00:00:32.669481+0000","last_became_active":"2024-05-22T23:57:37.777827+0000","last_became_peered":"2024-05-22T23:57:37.777827+0000","last_unstale":"2024-05-23T00:00:32.669481+0000","last_undegraded":"2024-05-23T00:00:32.669481+0000","last_fullsized":"2024-05-23T00:00:32.669481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:30:59.154921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023206+0000","last_change":"2024-05-22T23:57:32.317649+0000","last_active":"2024-05-23T00:00:33.023206+0000","last_peered":"2024-05-23T00:00:33.023206+0000","last_clean":"2024-05-23T00:00:33.023206+0000","last_became_active":"2024-05-22T23:57:32.317280+0000","last_became_peered":"2024-05-22T23:57:32.317280+0000","last_unstale":"2024-05-23T00:00:33.023206+0000","last_undegraded":"2024-05-23T00:00:33.023206+0000","last_fullsized":"2024-05-23T00:00:33.023206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:01:16.917079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670710+0000","last_change":"2024-05-22T23:57:33.339296+0000","last_active":"2024-05-23T00:00:32.670710+0000","last_peered":"2024-05-23T00:00:32.670710+0000","last_clean":"2024-05-23T00:00:32.670710+0000","last_became_active":"2024-05-22T23:57:33.338457+0000","last_became_peered":"2024-05-22T23:57:33.338457+0000","last_unstale":"2024-05-23T00:00:32.670710+0000","last_undegraded":"2024-05-23T00:00:32.670710+0000","last_fullsized":"2024-05-23T00:00:32.670710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:13:05.927893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023012+0000","last_change":"2024-05-22T23:57:38.218649+0000","last_active":"2024-05-23T00:00:33.023012+0000","last_peered":"2024-05-23T00:00:33.023012+0000","last_clean":"2024-05-23T00:00:33.023012+0000","last_became_active":"2024-05-22T23:57:38.218189+0000","last_became_peered":"2024-05-22T23:57:38.218189+0000","last_unstale":"2024-05-23T00:00:33.023012+0000","last_undegraded":"2024-05-23T00:00:33.023012+0000","last_fullsized":"2024-05-23T00:00:33.023012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:30.747293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020655+0000","last_change":"2024-05-22T23:57:32.309379+0000","last_active":"2024-05-23T00:00:33.020655+0000","last_peered":"2024-05-23T00:00:33.020655+0000","last_clean":"2024-05-23T00:00:33.020655+0000","last_became_active":"2024-05-22T23:57:32.309068+0000","last_became_peered":"2024-05-22T23:57:32.309068+0000","last_unstale":"2024-05-23T00:00:33.020655+0000","last_undegraded":"2024-05-23T00:00:33.020655+0000","last_fullsized":"2024-05-23T00:00:33.020655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:43:54.787996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022150+0000","last_change":"2024-05-22T23:57:33.320253+0000","last_active":"2024-05-23T00:00:33.022150+0000","last_peered":"2024-05-23T00:00:33.022150+0000","last_clean":"2024-05-23T00:00:33.022150+0000","last_became_active":"2024-05-22T23:57:33.319604+0000","last_became_peered":"2024-05-22T23:57:33.319604+0000","last_unstale":"2024-05-23T00:00:33.022150+0000","last_undegraded":"2024-05-23T00:00:33.022150+0000","last_fullsized":"2024-05-23T00:00:33.022150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:23:51.745040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672719+0000","last_change":"2024-05-22T23:57:37.795368+0000","last_active":"2024-05-23T00:00:32.672719+0000","last_peered":"2024-05-23T00:00:32.672719+0000","last_clean":"2024-05-23T00:00:32.672719+0000","last_became_active":"2024-05-22T23:57:37.792848+0000","last_became_peered":"2024-05-22T23:57:37.792848+0000","last_unstale":"2024-05-23T00:00:32.672719+0000","last_undegraded":"2024-05-23T00:00:32.672719+0000","last_fullsized":"2024-05-23T00:00:32.672719+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:45.859867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810299+0000","last_change":"2024-05-22T23:57:32.329147+0000","last_active":"2024-05-23T00:00:32.810299+0000","last_peered":"2024-05-23T00:00:32.810299+0000","last_clean":"2024-05-23T00:00:32.810299+0000","last_became_active":"2024-05-22T23:57:32.327254+0000","last_became_peered":"2024-05-22T23:57:32.327254+0000","last_unstale":"2024-05-23T00:00:32.810299+0000","last_undegraded":"2024-05-23T00:00:32.810299+0000","last_fullsized":"2024-05-23T00:00:32.810299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:55:24.915817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809447+0000","last_change":"2024-05-22T23:57:33.332843+0000","last_active":"2024-05-23T00:00:32.809447+0000","last_peered":"2024-05-23T00:00:32.809447+0000","last_clean":"2024-05-23T00:00:32.809447+0000","last_became_active":"2024-05-22T23:57:33.332497+0000","last_became_peered":"2024-05-22T23:57:33.332497+0000","last_unstale":"2024-05-23T00:00:32.809447+0000","last_undegraded":"2024-05-23T00:00:32.809447+0000","last_fullsized":"2024-05-23T00:00:32.809447+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:55:16.109676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023022+0000","last_change":"2024-05-22T23:57:37.787609+0000","last_active":"2024-05-23T00:00:33.023022+0000","last_peered":"2024-05-23T00:00:33.023022+0000","last_clean":"2024-05-23T00:00:33.023022+0000","last_became_active":"2024-05-22T23:57:37.787153+0000","last_became_peered":"2024-05-22T23:57:37.787153+0000","last_unstale":"2024-05-23T00:00:33.023022+0000","last_undegraded":"2024-05-23T00:00:33.023022+0000","last_fullsized":"2024-05-23T00:00:33.023022+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:57:51.640037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669838+0000","last_change":"2024-05-22T23:57:32.329784+0000","last_active":"2024-05-23T00:00:32.669838+0000","last_peered":"2024-05-23T00:00:32.669838+0000","last_clean":"2024-05-23T00:00:32.669838+0000","last_became_active":"2024-05-22T23:57:32.329393+0000","last_became_peered":"2024-05-22T23:57:32.329393+0000","last_unstale":"2024-05-23T00:00:32.669838+0000","last_undegraded":"2024-05-23T00:00:32.669838+0000","last_fullsized":"2024-05-23T00:00:32.669838+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:17:54.811962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.811747+0000","last_change":"2024-05-22T23:57:33.318803+0000","last_active":"2024-05-23T00:00:32.811747+0000","last_peered":"2024-05-23T00:00:32.811747+0000","last_clean":"2024-05-23T00:00:32.811747+0000","last_became_active":"2024-05-22T23:57:33.318170+0000","last_became_peered":"2024-05-22T23:57:33.318170+0000","last_unstale":"2024-05-23T00:00:32.811747+0000","last_undegraded":"2024-05-23T00:00:32.811747+0000","last_fullsized":"2024-05-23T00:00:32.811747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:56:56.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023028+0000","last_change":"2024-05-22T23:57:38.218583+0000","last_active":"2024-05-23T00:00:33.023028+0000","last_peered":"2024-05-23T00:00:33.023028+0000","last_clean":"2024-05-23T00:00:33.023028+0000","last_became_active":"2024-05-22T23:57:38.218048+0000","last_became_peered":"2024-05-22T23:57:38.218048+0000","last_unstale":"2024-05-23T00:00:33.023028+0000","last_undegraded":"2024-05-23T00:00:33.023028+0000","last_fullsized":"2024-05-23T00:00:33.023028+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:10:57.579656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022209+0000","last_change":"2024-05-22T23:57:32.317908+0000","last_active":"2024-05-23T00:00:33.022209+0000","last_peered":"2024-05-23T00:00:33.022209+0000","last_clean":"2024-05-23T00:00:33.022209+0000","last_became_active":"2024-05-22T23:57:32.317580+0000","last_became_peered":"2024-05-22T23:57:32.317580+0000","last_unstale":"2024-05-23T00:00:33.022209+0000","last_undegraded":"2024-05-23T00:00:33.022209+0000","last_fullsized":"2024-05-23T00:00:33.022209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:50:02.044817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023705+0000","last_change":"2024-05-22T23:57:33.321184+0000","last_active":"2024-05-23T00:00:33.023705+0000","last_peered":"2024-05-23T00:00:33.023705+0000","last_clean":"2024-05-23T00:00:33.023705+0000","last_became_active":"2024-05-22T23:57:33.320418+0000","last_became_peered":"2024-05-22T23:57:33.320418+0000","last_unstale":"2024-05-23T00:00:33.023705+0000","last_undegraded":"2024-05-23T00:00:33.023705+0000","last_fullsized":"2024-05-23T00:00:33.023705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:22:43.019809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020743+0000","last_change":"2024-05-22T23:57:37.795718+0000","last_active":"2024-05-23T00:00:33.020743+0000","last_peered":"2024-05-23T00:00:33.020743+0000","last_clean":"2024-05-23T00:00:33.020743+0000","last_became_active":"2024-05-22T23:57:37.795297+0000","last_became_peered":"2024-05-22T23:57:37.795297+0000","last_unstale":"2024-05-23T00:00:33.020743+0000","last_undegraded":"2024-05-23T00:00:33.020743+0000","last_fullsized":"2024-05-23T00:00:33.020743+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:56:47.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670584+0000","last_change":"2024-05-22T23:57:32.311921+0000","last_active":"2024-05-23T00:00:32.670584+0000","last_peered":"2024-05-23T00:00:32.670584+0000","last_clean":"2024-05-23T00:00:32.670584+0000","last_became_active":"2024-05-22T23:57:32.311600+0000","last_became_peered":"2024-05-22T23:57:32.311600+0000","last_unstale":"2024-05-23T00:00:32.670584+0000","last_undegraded":"2024-05-23T00:00:32.670584+0000","last_fullsized":"2024-05-23T00:00:32.670584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:35:21.882773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024166+0000","last_change":"2024-05-22T23:57:33.321289+0000","last_active":"2024-05-23T00:00:33.024166+0000","last_peered":"2024-05-23T00:00:33.024166+0000","last_clean":"2024-05-23T00:00:33.024166+0000","last_became_active":"2024-05-22T23:57:33.320458+0000","last_became_peered":"2024-05-22T23:57:33.320458+0000","last_unstale":"2024-05-23T00:00:33.024166+0000","last_undegraded":"2024-05-23T00:00:33.024166+0000","last_fullsized":"2024-05-23T00:00:33.024166+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:02:05.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670012+0000","last_change":"2024-05-22T23:57:37.786062+0000","last_active":"2024-05-23T00:00:32.670012+0000","last_peered":"2024-05-23T00:00:32.670012+0000","last_clean":"2024-05-23T00:00:32.670012+0000","last_became_active":"2024-05-22T23:57:37.785624+0000","last_became_peered":"2024-05-22T23:57:37.785624+0000","last_unstale":"2024-05-23T00:00:32.670012+0000","last_undegraded":"2024-05-23T00:00:32.670012+0000","last_fullsized":"2024-05-23T00:00:32.670012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:38:56.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.960516+0000","last_change":"2024-05-22T23:57:32.333909+0000","last_active":"2024-05-23T00:00:32.960516+0000","last_peered":"2024-05-23T00:00:32.960516+0000","last_clean":"2024-05-23T00:00:32.960516+0000","last_became_active":"2024-05-22T23:57:32.333531+0000","last_became_peered":"2024-05-22T23:57:32.333531+0000","last_unstale":"2024-05-23T00:00:32.960516+0000","last_undegraded":"2024-05-23T00:00:32.960516+0000","last_fullsized":"2024-05-23T00:00:32.960516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:53:21.843150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023876+0000","last_change":"2024-05-22T23:57:33.334366+0000","last_active":"2024-05-23T00:00:33.023876+0000","last_peered":"2024-05-23T00:00:33.023876+0000","last_clean":"2024-05-23T00:00:33.023876+0000","last_became_active":"2024-05-22T23:57:33.334037+0000","last_became_peered":"2024-05-22T23:57:33.334037+0000","last_unstale":"2024-05-23T00:00:33.023876+0000","last_undegraded":"2024-05-23T00:00:33.023876+0000","last_fullsized":"2024-05-23T00:00:33.023876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:15:10.631087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818480+0000","last_change":"2024-05-22T23:57:37.780241+0000","last_active":"2024-05-23T00:00:32.818480+0000","last_peered":"2024-05-23T00:00:32.818480+0000","last_clean":"2024-05-23T00:00:32.818480+0000","last_became_active":"2024-05-22T23:57:37.779816+0000","last_became_peered":"2024-05-22T23:57:37.779816+0000","last_unstale":"2024-05-23T00:00:32.818480+0000","last_undegraded":"2024-05-23T00:00:32.818480+0000","last_fullsized":"2024-05-23T00:00:32.818480+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:11.397681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024119+0000","last_change":"2024-05-22T23:57:32.334829+0000","last_active":"2024-05-23T00:00:33.024119+0000","last_peered":"2024-05-23T00:00:33.024119+0000","last_clean":"2024-05-23T00:00:33.024119+0000","last_became_active":"2024-05-22T23:57:32.334368+0000","last_became_peered":"2024-05-22T23:57:32.334368+0000","last_unstale":"2024-05-23T00:00:33.024119+0000","last_undegraded":"2024-05-23T00:00:33.024119+0000","last_fullsized":"2024-05-23T00:00:33.024119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:17:42.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024133+0000","last_change":"2024-05-22T23:57:33.318740+0000","last_active":"2024-05-23T00:00:33.024133+0000","last_peered":"2024-05-23T00:00:33.024133+0000","last_clean":"2024-05-23T00:00:33.024133+0000","last_became_active":"2024-05-22T23:57:33.318363+0000","last_became_peered":"2024-05-22T23:57:33.318363+0000","last_unstale":"2024-05-23T00:00:33.024133+0000","last_undegraded":"2024-05-23T00:00:33.024133+0000","last_fullsized":"2024-05-23T00:00:33.024133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:51:42.319667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809555+0000","last_change":"2024-05-22T23:57:33.325582+0000","last_active":"2024-05-23T00:00:32.809555+0000","last_peered":"2024-05-23T00:00:32.809555+0000","last_clean":"2024-05-23T00:00:32.809555+0000","last_became_active":"2024-05-22T23:57:33.325268+0000","last_became_peered":"2024-05-22T23:57:33.325268+0000","last_unstale":"2024-05-23T00:00:32.809555+0000","last_undegraded":"2024-05-23T00:00:32.809555+0000","last_fullsized":"2024-05-23T00:00:32.809555+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:02:18.968634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669768+0000","last_change":"2024-05-22T23:58:31.109626+0000","last_active":"2024-05-23T00:00:32.669768+0000","last_peered":"2024-05-23T00:00:32.669768+0000","last_clean":"2024-05-23T00:00:32.669768+0000","last_became_active":"2024-05-22T23:58:31.109179+0000","last_became_peered":"2024-05-22T23:58:31.109179+0000","last_unstale":"2024-05-23T00:00:32.669768+0000","last_undegraded":"2024-05-23T00:00:32.669768+0000","last_fullsized":"2024-05-23T00:00:32.669768+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:36.653833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669573+0000","last_change":"2024-05-22T23:57:33.332935+0000","last_active":"2024-05-23T00:00:32.669573+0000","last_peered":"2024-05-23T00:00:32.669573+0000","last_clean":"2024-05-23T00:00:32.669573+0000","last_became_active":"2024-05-22T23:57:33.332607+0000","last_became_peered":"2024-05-22T23:57:33.332607+0000","last_unstale":"2024-05-23T00:00:32.669573+0000","last_undegraded":"2024-05-23T00:00:32.669573+0000","last_fullsized":"2024-05-23T00:00:32.669573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:09.589896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670318+0000","last_change":"2024-05-22T23:57:33.333269+0000","last_active":"2024-05-23T00:00:32.670318+0000","last_peered":"2024-05-23T00:00:32.670318+0000","last_clean":"2024-05-23T00:00:32.670318+0000","last_became_active":"2024-05-22T23:57:33.329017+0000","last_became_peered":"2024-05-22T23:57:33.329017+0000","last_unstale":"2024-05-23T00:00:32.670318+0000","last_undegraded":"2024-05-23T00:00:32.670318+0000","last_fullsized":"2024-05-23T00:00:32.670318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:35:42.156434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023462+0000","last_change":"2024-05-22T23:57:33.328721+0000","last_active":"2024-05-23T00:00:33.023462+0000","last_peered":"2024-05-23T00:00:33.023462+0000","last_clean":"2024-05-23T00:00:33.023462+0000","last_became_active":"2024-05-22T23:57:33.328318+0000","last_became_peered":"2024-05-22T23:57:33.328318+0000","last_unstale":"2024-05-23T00:00:33.023462+0000","last_undegraded":"2024-05-23T00:00:33.023462+0000","last_fullsized":"2024-05-23T00:00:33.023462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:14.772688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809824+0000","last_change":"2024-05-22T23:57:33.324777+0000","last_active":"2024-05-23T00:00:32.809824+0000","last_peered":"2024-05-23T00:00:32.809824+0000","last_clean":"2024-05-23T00:00:32.809824+0000","last_became_active":"2024-05-22T23:57:33.324446+0000","last_became_peered":"2024-05-22T23:57:33.324446+0000","last_unstale":"2024-05-23T00:00:32.809824+0000","last_undegraded":"2024-05-23T00:00:32.809824+0000","last_fullsized":"2024-05-23T00:00:32.809824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:20:42.327094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670314+0000","last_change":"2024-05-22T23:57:33.318783+0000","last_active":"2024-05-23T00:00:32.670314+0000","last_peered":"2024-05-23T00:00:32.670314+0000","last_clean":"2024-05-23T00:00:32.670314+0000","last_became_active":"2024-05-22T23:57:33.318481+0000","last_became_peered":"2024-05-22T23:57:33.318481+0000","last_unstale":"2024-05-23T00:00:32.670314+0000","last_undegraded":"2024-05-23T00:00:32.670314+0000","last_fullsized":"2024-05-23T00:00:32.670314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:14:35.745683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812167+0000","last_change":"2024-05-22T23:57:33.319939+0000","last_active":"2024-05-23T00:00:32.812167+0000","last_peered":"2024-05-23T00:00:32.812167+0000","last_clean":"2024-05-23T00:00:32.812167+0000","last_became_active":"2024-05-22T23:57:33.316029+0000","last_became_peered":"2024-05-22T23:57:33.316029+0000","last_unstale":"2024-05-23T00:00:32.812167+0000","last_undegraded":"2024-05-23T00:00:32.812167+0000","last_fullsized":"2024-05-23T00:00:32.812167+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:46:34.383953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818610+0000","last_change":"2024-05-22T23:57:33.334625+0000","last_active":"2024-05-23T00:00:32.818610+0000","last_peered":"2024-05-23T00:00:32.818610+0000","last_clean":"2024-05-23T00:00:32.818610+0000","last_became_active":"2024-05-22T23:57:33.334341+0000","last_became_peered":"2024-05-22T23:57:33.334341+0000","last_unstale":"2024-05-23T00:00:32.818610+0000","last_undegraded":"2024-05-23T00:00:32.818610+0000","last_fullsized":"2024-05-23T00:00:32.818610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:38.319721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810428+0000","last_change":"2024-05-22T23:57:33.330958+0000","last_active":"2024-05-23T00:00:32.810428+0000","last_peered":"2024-05-23T00:00:32.810428+0000","last_clean":"2024-05-23T00:00:32.810428+0000","last_became_active":"2024-05-22T23:57:33.317355+0000","last_became_peered":"2024-05-22T23:57:33.317355+0000","last_unstale":"2024-05-23T00:00:32.810428+0000","last_undegraded":"2024-05-23T00:00:32.810428+0000","last_fullsized":"2024-05-23T00:00:32.810428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:49:43.784780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021047+0000","last_change":"2024-05-22T23:57:33.319530+0000","last_active":"2024-05-23T00:00:33.021047+0000","last_peered":"2024-05-23T00:00:33.021047+0000","last_clean":"2024-05-23T00:00:33.021047+0000","last_became_active":"2024-05-22T23:57:33.319122+0000","last_became_peered":"2024-05-22T23:57:33.319122+0000","last_unstale":"2024-05-23T00:00:33.021047+0000","last_undegraded":"2024-05-23T00:00:33.021047+0000","last_fullsized":"2024-05-23T00:00:33.021047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:31:26.937804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024256+0000","last_change":"2024-05-22T23:57:33.323722+0000","last_active":"2024-05-23T00:00:33.024256+0000","last_peered":"2024-05-23T00:00:33.024256+0000","last_clean":"2024-05-23T00:00:33.024256+0000","last_became_active":"2024-05-22T23:57:33.323343+0000","last_became_peered":"2024-05-22T23:57:33.323343+0000","last_unstale":"2024-05-23T00:00:33.024256+0000","last_undegraded":"2024-05-23T00:00:33.024256+0000","last_fullsized":"2024-05-23T00:00:33.024256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:24:59.575536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809773+0000","last_change":"2024-05-22T23:57:33.320925+0000","last_active":"2024-05-23T00:00:32.809773+0000","last_peered":"2024-05-23T00:00:32.809773+0000","last_clean":"2024-05-23T00:00:32.809773+0000","last_became_active":"2024-05-22T23:57:33.320570+0000","last_became_peered":"2024-05-22T23:57:33.320570+0000","last_unstale":"2024-05-23T00:00:32.809773+0000","last_undegraded":"2024-05-23T00:00:32.809773+0000","last_fullsized":"2024-05-23T00:00:32.809773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:38:27.659025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107088,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599621,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1018,"num_read_kb":622,"num_write":295,"num_write_kb":1448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1012,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1798863,"data_compressed":80985,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318321,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":911392,"data_compressed":42724,"data_compressed_allocated":57344,"data_compressed_original":458752,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579959,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1164430,"data_compressed":57992,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743484,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":522508,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005122,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":518332,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266761,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1105414,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430284,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":503780,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":582541,"data_compressed":25271,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":514976,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":582489,"data_compressed":25271,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1038598,"data_compressed":48880,"data_compressed_allocated":65536,"data_compressed_original":581632,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888153,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":510424,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691925,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1100471,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69043,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69771,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134307,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203142,"data_compressed":7849,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-23T00:00:37.102 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-23T00:00:37.102 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-23T00:00:37.145 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:00:36 smithi174 sudo[69468]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:37.145 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:00:36 smithi174 sudo[69468]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:37.145 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:00:36 smithi174 sudo[69468]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:37.145 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:00:36 smithi174 sudo[69468]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 -- 172.21.15.103:0/2654212247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 msgr2=0x7fae58101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 --2- 172.21.15.103:0/2654212247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 0x7fae58101b30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fae48009900 tx=0x7fae4802a410 comp rx=0 tx=0).stop 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 -- 172.21.15.103:0/2654212247 shutdown_connections 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 --2- 172.21.15.103:0/2654212247 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae581032c0 0x7fae58109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 --2- 172.21.15.103:0/2654212247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae58102900 0x7fae58102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 --2- 172.21.15.103:0/2654212247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 0x7fae58101b30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 -- 172.21.15.103:0/2654212247 >> 172.21.15.103:0/2654212247 conn(0x7fae580fcc80 msgr2=0x7fae580ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 -- 172.21.15.103:0/2654212247 shutdown_connections 2024-05-23T00:00:37.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.372+0000 7fae57e00640 1 -- 172.21.15.103:0/2654212247 wait complete. 2024-05-23T00:00:37.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.374+0000 7fae57e00640 1 Processor -- start 2024-05-23T00:00:37.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.374+0000 7fae57e00640 1 -- start start 2024-05-23T00:00:37.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.374+0000 7fae57e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 0x7fae58199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:37.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.374+0000 7fae57e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae58102900 0x7fae5819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:37.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.374+0000 7fae57e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae581032c0 0x7fae581a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:37.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.374+0000 7fae57e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fae5810d390 con 0x7fae581032c0 2024-05-23T00:00:37.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.374+0000 7fae57e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fae5810d210 con 0x7fae58101710 2024-05-23T00:00:37.378 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.374+0000 7fae57e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fae5810d510 con 0x7fae58102900 2024-05-23T00:00:37.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 0x7fae58199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:37.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae58102900 0x7fae5819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:37.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 0x7fae58199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49446/0 (socket says 172.21.15.103:49446) 2024-05-23T00:00:37.379 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56a00640 1 -- 172.21.15.103:0/3820804765 learned_addr learned my addr 172.21.15.103:0/3820804765 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae57400640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae581032c0 0x7fae581a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56a00640 1 -- 172.21.15.103:0/3820804765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae58102900 msgr2=0x7fae5819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56a00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae58102900 0x7fae5819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56a00640 1 -- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae581032c0 msgr2=0x7fae581a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56a00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae581032c0 0x7fae581a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae56a00640 1 -- 172.21.15.103:0/3820804765 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae581a1850 con 0x7fae58101710 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.375+0000 7fae57400640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae581032c0 0x7fae581a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.376+0000 7fae56a00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 0x7fae58199c90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae48031610 tx=0x7fae48004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:37.380 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.376+0000 7fae5ca00640 1 -- 172.21.15.103:0/3820804765 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4803f030 con 0x7fae58101710 2024-05-23T00:00:37.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.376+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae581a1ae0 con 0x7fae58101710 2024-05-23T00:00:37.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.376+0000 7fae5ca00640 1 -- 172.21.15.103:0/3820804765 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae48003ad0 con 0x7fae58101710 2024-05-23T00:00:37.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.376+0000 7fae5ca00640 1 -- 172.21.15.103:0/3820804765 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae48035070 con 0x7fae58101710 2024-05-23T00:00:37.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.376+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae581a2020 con 0x7fae58101710 2024-05-23T00:00:37.381 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.378+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae581a21f0 con 0x7fae58101710 2024-05-23T00:00:37.382 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.379+0000 7fae5ca00640 1 -- 172.21.15.103:0/3820804765 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 24) ==== 137954+0+0 (secure 0 0 0) 0x7fae48042030 con 0x7fae58101710 2024-05-23T00:00:37.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.382+0000 7fae5ca00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fae3c0a2d00 0x7fae3c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:37.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.382+0000 7fae5ca00640 1 -- 172.21.15.103:0/3820804765 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fae480efe10 con 0x7fae58101710 2024-05-23T00:00:37.385 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.382+0000 7fae56000640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fae3c0a2d00 0x7fae3c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:37.386 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.382+0000 7fae56000640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fae3c0a2d00 0x7fae3c0a51c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fae44009bd0 tx=0x7fae44005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:37.388 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.385+0000 7fae5ca00640 1 -- 172.21.15.103:0/3820804765 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fae480f3030 con 0x7fae58101710 2024-05-23T00:00:37.547 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:00:37 smithi174 sudo[69471]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:37.547 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:00:37 smithi174 sudo[69471]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:37.547 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:00:37 smithi174 sudo[69471]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:37.548 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:00:37 smithi174 sudo[69471]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:37.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.593+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fae580751c0 con 0x7fae3c0a2d00 2024-05-23T00:00:37.610 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.607+0000 7fae5ca00640 1 -- 172.21.15.103:0/3820804765 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+339719 (secure 0 0 0) 0x7fae580751c0 con 0x7fae3c0a2d00 2024-05-23T00:00:37.615 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:37.617 INFO:teuthology.orchestra.run.smithi103.stderr:dumped all 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fae3c0a2d00 msgr2=0x7fae3c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fae3c0a2d00 0x7fae3c0a51c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fae44009bd0 tx=0x7fae44005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 msgr2=0x7fae58199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 0x7fae58199c90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae48031610 tx=0x7fae48004270 comp rx=0 tx=0).stop 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 shutdown_connections 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fae3c0a2d00 0x7fae3c0a51c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fae581032c0 0x7fae581a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fae58102900 0x7fae5819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 --2- 172.21.15.103:0/3820804765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae58101710 0x7fae58199c90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 >> 172.21.15.103:0/3820804765 conn(0x7fae580fcc80 msgr2=0x7fae58073aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:37.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 shutdown_connections 2024-05-23T00:00:37.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.613+0000 7fae57e00640 1 -- 172.21.15.103:0/3820804765 wait complete. 2024-05-23T00:00:37.640 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.640 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.640 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.640 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.640 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.640 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.640 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-05-23T00:00:37.641 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='client.25505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: Unable to set osd_memory_target on smithi105 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:37.642 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='client.25511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:37.643 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:37 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.644 INFO:teuthology.orchestra.run.smithi103.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-05-23T00:00:36.693138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1190223,"num_objects":66,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1154,"num_read_kb":741,"num_write":478,"num_write_kb":4388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1181748,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":873,"ondisk_log_size":873,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357572,"kb_used_data":12556,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584252,"statfs":{"total":1151940427776,"available":1151574274048,"internally_reserved":0,"allocated":12857344,"data_stored":9055355,"data_compressed":424676,"data_compressed_allocated":577536,"data_compressed_original":3448832,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.991324"},"pg_stats":[{"pgid":"3.3f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810114+0000","last_change":"2024-05-22T23:57:33.328075+0000","last_active":"2024-05-23T00:00:32.810114+0000","last_peered":"2024-05-23T00:00:32.810114+0000","last_clean":"2024-05-23T00:00:32.810114+0000","last_became_active":"2024-05-22T23:57:33.327704+0000","last_became_peered":"2024-05-22T23:57:33.327704+0000","last_unstale":"2024-05-23T00:00:32.810114+0000","last_undegraded":"2024-05-23T00:00:32.810114+0000","last_fullsized":"2024-05-23T00:00:32.810114+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:42:32.778586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670793+0000","last_change":"2024-05-22T23:57:33.332043+0000","last_active":"2024-05-23T00:00:32.670793+0000","last_peered":"2024-05-23T00:00:32.670793+0000","last_clean":"2024-05-23T00:00:32.670793+0000","last_became_active":"2024-05-22T23:57:33.322269+0000","last_became_peered":"2024-05-22T23:57:33.322269+0000","last_unstale":"2024-05-23T00:00:32.670793+0000","last_undegraded":"2024-05-23T00:00:32.670793+0000","last_fullsized":"2024-05-23T00:00:32.670793+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:40:23.542938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670609+0000","last_change":"2024-05-22T23:57:33.334993+0000","last_active":"2024-05-23T00:00:32.670609+0000","last_peered":"2024-05-23T00:00:32.670609+0000","last_clean":"2024-05-23T00:00:32.670609+0000","last_became_active":"2024-05-22T23:57:33.326446+0000","last_became_peered":"2024-05-22T23:57:33.326446+0000","last_unstale":"2024-05-23T00:00:32.670609+0000","last_undegraded":"2024-05-23T00:00:32.670609+0000","last_fullsized":"2024-05-23T00:00:32.670609+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:44:45.750852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023391+0000","last_change":"2024-05-22T23:57:33.320181+0000","last_active":"2024-05-23T00:00:33.023391+0000","last_peered":"2024-05-23T00:00:33.023391+0000","last_clean":"2024-05-23T00:00:33.023391+0000","last_became_active":"2024-05-22T23:57:33.319630+0000","last_became_peered":"2024-05-22T23:57:33.319630+0000","last_unstale":"2024-05-23T00:00:33.023391+0000","last_undegraded":"2024-05-23T00:00:33.023391+0000","last_fullsized":"2024-05-23T00:00:33.023391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:38:51.836241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022138+0000","last_change":"2024-05-22T23:57:33.332168+0000","last_active":"2024-05-23T00:00:33.022138+0000","last_peered":"2024-05-23T00:00:33.022138+0000","last_clean":"2024-05-23T00:00:33.022138+0000","last_became_active":"2024-05-22T23:57:33.331755+0000","last_became_peered":"2024-05-22T23:57:33.331755+0000","last_unstale":"2024-05-23T00:00:33.022138+0000","last_undegraded":"2024-05-23T00:00:33.022138+0000","last_fullsized":"2024-05-23T00:00:33.022138+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:28.013365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822749+0000","last_change":"2024-05-22T23:57:33.322449+0000","last_active":"2024-05-23T00:00:32.822749+0000","last_peered":"2024-05-23T00:00:32.822749+0000","last_clean":"2024-05-23T00:00:32.822749+0000","last_became_active":"2024-05-22T23:57:33.322099+0000","last_became_peered":"2024-05-22T23:57:33.322099+0000","last_unstale":"2024-05-23T00:00:32.822749+0000","last_undegraded":"2024-05-23T00:00:32.822749+0000","last_fullsized":"2024-05-23T00:00:32.822749+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:38.427434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023339+0000","last_change":"2024-05-22T23:57:33.322391+0000","last_active":"2024-05-23T00:00:33.023339+0000","last_peered":"2024-05-23T00:00:33.023339+0000","last_clean":"2024-05-23T00:00:33.023339+0000","last_became_active":"2024-05-22T23:57:33.322076+0000","last_became_peered":"2024-05-22T23:57:33.322076+0000","last_unstale":"2024-05-23T00:00:33.023339+0000","last_undegraded":"2024-05-23T00:00:33.023339+0000","last_fullsized":"2024-05-23T00:00:33.023339+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:45:44.857586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810020+0000","last_change":"2024-05-22T23:57:33.333005+0000","last_active":"2024-05-23T00:00:32.810020+0000","last_peered":"2024-05-23T00:00:32.810020+0000","last_clean":"2024-05-23T00:00:32.810020+0000","last_became_active":"2024-05-22T23:57:33.332669+0000","last_became_peered":"2024-05-22T23:57:33.332669+0000","last_unstale":"2024-05-23T00:00:32.810020+0000","last_undegraded":"2024-05-23T00:00:32.810020+0000","last_fullsized":"2024-05-23T00:00:32.810020+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:47:25.596106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"93'11","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021594+0000","last_change":"2024-05-22T23:58:31.104194+0000","last_active":"2024-05-23T00:00:33.021594+0000","last_peered":"2024-05-23T00:00:33.021594+0000","last_clean":"2024-05-23T00:00:33.021594+0000","last_became_active":"2024-05-22T23:58:31.103788+0000","last_became_peered":"2024-05-22T23:58:31.103788+0000","last_unstale":"2024-05-23T00:00:33.021594+0000","last_undegraded":"2024-05-23T00:00:33.021594+0000","last_fullsized":"2024-05-23T00:00:33.021594+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:41:09.215669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,11],"acting":[1,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"93'16","reported_seq":44,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822962+0000","last_change":"2024-05-22T23:57:33.324445+0000","last_active":"2024-05-23T00:00:32.822962+0000","last_peered":"2024-05-23T00:00:32.822962+0000","last_clean":"2024-05-23T00:00:32.822962+0000","last_became_active":"2024-05-22T23:57:33.324085+0000","last_became_peered":"2024-05-22T23:57:33.324085+0000","last_unstale":"2024-05-23T00:00:32.822962+0000","last_undegraded":"2024-05-23T00:00:32.822962+0000","last_fullsized":"2024-05-23T00:00:32.822962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:30:51.545667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024137+0000","last_change":"2024-05-22T23:57:33.321481+0000","last_active":"2024-05-23T00:00:33.024137+0000","last_peered":"2024-05-23T00:00:33.024137+0000","last_clean":"2024-05-23T00:00:33.024137+0000","last_became_active":"2024-05-22T23:57:33.320911+0000","last_became_peered":"2024-05-22T23:57:33.320911+0000","last_unstale":"2024-05-23T00:00:33.024137+0000","last_undegraded":"2024-05-23T00:00:33.024137+0000","last_fullsized":"2024-05-23T00:00:33.024137+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:24:24.041980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023362+0000","last_change":"2024-05-22T23:57:33.318819+0000","last_active":"2024-05-23T00:00:33.023362+0000","last_peered":"2024-05-23T00:00:33.023362+0000","last_clean":"2024-05-23T00:00:33.023362+0000","last_became_active":"2024-05-22T23:57:33.318284+0000","last_became_peered":"2024-05-22T23:57:33.318284+0000","last_unstale":"2024-05-23T00:00:33.023362+0000","last_undegraded":"2024-05-23T00:00:33.023362+0000","last_fullsized":"2024-05-23T00:00:33.023362+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:50:56.043446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024014+0000","last_change":"2024-05-22T23:57:33.321268+0000","last_active":"2024-05-23T00:00:33.024014+0000","last_peered":"2024-05-23T00:00:33.024014+0000","last_clean":"2024-05-23T00:00:33.024014+0000","last_became_active":"2024-05-22T23:57:33.320902+0000","last_became_peered":"2024-05-22T23:57:33.320902+0000","last_unstale":"2024-05-23T00:00:33.024014+0000","last_undegraded":"2024-05-23T00:00:33.024014+0000","last_fullsized":"2024-05-23T00:00:33.024014+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:51:39.716541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669962+0000","last_change":"2024-05-22T23:57:33.322638+0000","last_active":"2024-05-23T00:00:32.669962+0000","last_peered":"2024-05-23T00:00:32.669962+0000","last_clean":"2024-05-23T00:00:32.669962+0000","last_became_active":"2024-05-22T23:57:33.322318+0000","last_became_peered":"2024-05-22T23:57:33.322318+0000","last_unstale":"2024-05-23T00:00:32.669962+0000","last_undegraded":"2024-05-23T00:00:32.669962+0000","last_fullsized":"2024-05-23T00:00:32.669962+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:44:25.487618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669867+0000","last_change":"2024-05-22T23:57:33.325832+0000","last_active":"2024-05-23T00:00:32.669867+0000","last_peered":"2024-05-23T00:00:32.669867+0000","last_clean":"2024-05-23T00:00:32.669867+0000","last_became_active":"2024-05-22T23:57:33.325062+0000","last_became_peered":"2024-05-22T23:57:33.325062+0000","last_unstale":"2024-05-23T00:00:32.669867+0000","last_undegraded":"2024-05-23T00:00:32.669867+0000","last_fullsized":"2024-05-23T00:00:32.669867+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:33:04.161351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020743+0000","last_change":"2024-05-22T23:57:33.324864+0000","last_active":"2024-05-23T00:00:33.020743+0000","last_peered":"2024-05-23T00:00:33.020743+0000","last_clean":"2024-05-23T00:00:33.020743+0000","last_became_active":"2024-05-22T23:57:33.324558+0000","last_became_peered":"2024-05-22T23:57:33.324558+0000","last_unstale":"2024-05-23T00:00:33.020743+0000","last_undegraded":"2024-05-23T00:00:33.020743+0000","last_fullsized":"2024-05-23T00:00:33.020743+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:48:49.722371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809904+0000","last_change":"2024-05-22T23:57:33.337985+0000","last_active":"2024-05-23T00:00:32.809904+0000","last_peered":"2024-05-23T00:00:32.809904+0000","last_clean":"2024-05-23T00:00:32.809904+0000","last_became_active":"2024-05-22T23:57:33.337719+0000","last_became_peered":"2024-05-22T23:57:33.337719+0000","last_unstale":"2024-05-23T00:00:32.809904+0000","last_undegraded":"2024-05-23T00:00:32.809904+0000","last_fullsized":"2024-05-23T00:00:32.809904+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:50:44.282299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.673585+0000","last_change":"2024-05-22T23:57:33.325709+0000","last_active":"2024-05-23T00:00:32.673585+0000","last_peered":"2024-05-23T00:00:32.673585+0000","last_clean":"2024-05-23T00:00:32.673585+0000","last_became_active":"2024-05-22T23:57:33.325063+0000","last_became_peered":"2024-05-22T23:57:33.325063+0000","last_unstale":"2024-05-23T00:00:32.673585+0000","last_undegraded":"2024-05-23T00:00:32.673585+0000","last_fullsized":"2024-05-23T00:00:32.673585+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:39.145217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670219+0000","last_change":"2024-05-22T23:57:33.333251+0000","last_active":"2024-05-23T00:00:32.670219+0000","last_peered":"2024-05-23T00:00:32.670219+0000","last_clean":"2024-05-23T00:00:32.670219+0000","last_became_active":"2024-05-22T23:57:33.332894+0000","last_became_peered":"2024-05-22T23:57:33.332894+0000","last_unstale":"2024-05-23T00:00:32.670219+0000","last_undegraded":"2024-05-23T00:00:32.670219+0000","last_fullsized":"2024-05-23T00:00:32.670219+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:13:12.082781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669594+0000","last_change":"2024-05-22T23:57:37.788060+0000","last_active":"2024-05-23T00:00:32.669594+0000","last_peered":"2024-05-23T00:00:32.669594+0000","last_clean":"2024-05-23T00:00:32.669594+0000","last_became_active":"2024-05-22T23:57:37.787656+0000","last_became_peered":"2024-05-22T23:57:37.787656+0000","last_unstale":"2024-05-23T00:00:32.669594+0000","last_undegraded":"2024-05-23T00:00:32.669594+0000","last_fullsized":"2024-05-23T00:00:32.669594+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:39:13.718817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"91'9","reported_seq":40,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672586+0000","last_change":"2024-05-22T23:57:32.323434+0000","last_active":"2024-05-23T00:00:32.672586+0000","last_peered":"2024-05-23T00:00:32.672586+0000","last_clean":"2024-05-23T00:00:32.672586+0000","last_became_active":"2024-05-22T23:57:32.323033+0000","last_became_peered":"2024-05-22T23:57:32.323033+0000","last_unstale":"2024-05-23T00:00:32.672586+0000","last_undegraded":"2024-05-23T00:00:32.672586+0000","last_fullsized":"2024-05-23T00:00:32.672586+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:15:27.676027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812037+0000","last_change":"2024-05-22T23:57:33.323336+0000","last_active":"2024-05-23T00:00:32.812037+0000","last_peered":"2024-05-23T00:00:32.812037+0000","last_clean":"2024-05-23T00:00:32.812037+0000","last_became_active":"2024-05-22T23:57:33.323026+0000","last_became_peered":"2024-05-22T23:57:33.323026+0000","last_unstale":"2024-05-23T00:00:32.812037+0000","last_undegraded":"2024-05-23T00:00:32.812037+0000","last_fullsized":"2024-05-23T00:00:32.812037+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:02:20.142760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022462+0000","last_change":"2024-05-22T23:57:37.785137+0000","last_active":"2024-05-23T00:00:33.022462+0000","last_peered":"2024-05-23T00:00:33.022462+0000","last_clean":"2024-05-23T00:00:33.022462+0000","last_became_active":"2024-05-22T23:57:37.784791+0000","last_became_peered":"2024-05-22T23:57:37.784791+0000","last_unstale":"2024-05-23T00:00:33.022462+0000","last_undegraded":"2024-05-23T00:00:33.022462+0000","last_fullsized":"2024-05-23T00:00:33.022462+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:54:51.036334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812106+0000","last_change":"2024-05-22T23:57:32.332969+0000","last_active":"2024-05-23T00:00:32.812106+0000","last_peered":"2024-05-23T00:00:32.812106+0000","last_clean":"2024-05-23T00:00:32.812106+0000","last_became_active":"2024-05-22T23:57:32.332611+0000","last_became_peered":"2024-05-22T23:57:32.332611+0000","last_unstale":"2024-05-23T00:00:32.812106+0000","last_undegraded":"2024-05-23T00:00:32.812106+0000","last_fullsized":"2024-05-23T00:00:32.812106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:00:22.006355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670839+0000","last_change":"2024-05-22T23:57:33.332151+0000","last_active":"2024-05-23T00:00:32.670839+0000","last_peered":"2024-05-23T00:00:32.670839+0000","last_clean":"2024-05-23T00:00:32.670839+0000","last_became_active":"2024-05-22T23:57:33.329518+0000","last_became_peered":"2024-05-22T23:57:33.329518+0000","last_unstale":"2024-05-23T00:00:32.670839+0000","last_undegraded":"2024-05-23T00:00:32.670839+0000","last_fullsized":"2024-05-23T00:00:32.670839+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:10:12.083766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022602+0000","last_change":"2024-05-22T23:57:37.787524+0000","last_active":"2024-05-23T00:00:33.022602+0000","last_peered":"2024-05-23T00:00:33.022602+0000","last_clean":"2024-05-23T00:00:33.022602+0000","last_became_active":"2024-05-22T23:57:37.787103+0000","last_became_peered":"2024-05-22T23:57:37.787103+0000","last_unstale":"2024-05-23T00:00:33.022602+0000","last_undegraded":"2024-05-23T00:00:33.022602+0000","last_fullsized":"2024-05-23T00:00:33.022602+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:19:16.963159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022671+0000","last_change":"2024-05-22T23:57:32.318469+0000","last_active":"2024-05-23T00:00:33.022671+0000","last_peered":"2024-05-23T00:00:33.022671+0000","last_clean":"2024-05-23T00:00:33.022671+0000","last_became_active":"2024-05-22T23:57:32.317640+0000","last_became_peered":"2024-05-22T23:57:32.317640+0000","last_unstale":"2024-05-23T00:00:33.022671+0000","last_undegraded":"2024-05-23T00:00:33.022671+0000","last_fullsized":"2024-05-23T00:00:33.022671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:01:55.710019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021134+0000","last_change":"2024-05-22T23:57:33.318604+0000","last_active":"2024-05-23T00:00:33.021134+0000","last_peered":"2024-05-23T00:00:33.021134+0000","last_clean":"2024-05-23T00:00:33.021134+0000","last_became_active":"2024-05-22T23:57:33.318184+0000","last_became_peered":"2024-05-22T23:57:33.318184+0000","last_unstale":"2024-05-23T00:00:33.021134+0000","last_undegraded":"2024-05-23T00:00:33.021134+0000","last_fullsized":"2024-05-23T00:00:33.021134+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:16.140799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022571+0000","last_change":"2024-05-22T23:57:37.796085+0000","last_active":"2024-05-23T00:00:33.022571+0000","last_peered":"2024-05-23T00:00:33.022571+0000","last_clean":"2024-05-23T00:00:33.022571+0000","last_became_active":"2024-05-22T23:57:37.795265+0000","last_became_peered":"2024-05-22T23:57:37.795265+0000","last_unstale":"2024-05-23T00:00:33.022571+0000","last_undegraded":"2024-05-23T00:00:33.022571+0000","last_fullsized":"2024-05-23T00:00:33.022571+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:43:55.577072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022541+0000","last_change":"2024-05-22T23:57:32.316346+0000","last_active":"2024-05-23T00:00:33.022541+0000","last_peered":"2024-05-23T00:00:33.022541+0000","last_clean":"2024-05-23T00:00:33.022541+0000","last_became_active":"2024-05-22T23:57:32.316038+0000","last_became_peered":"2024-05-22T23:57:32.316038+0000","last_unstale":"2024-05-23T00:00:33.022541+0000","last_undegraded":"2024-05-23T00:00:33.022541+0000","last_fullsized":"2024-05-23T00:00:33.022541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:57:41.987250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819205+0000","last_change":"2024-05-22T23:57:33.329341+0000","last_active":"2024-05-23T00:00:32.819205+0000","last_peered":"2024-05-23T00:00:32.819205+0000","last_clean":"2024-05-23T00:00:32.819205+0000","last_became_active":"2024-05-22T23:57:33.329009+0000","last_became_peered":"2024-05-22T23:57:33.329009+0000","last_unstale":"2024-05-23T00:00:32.819205+0000","last_undegraded":"2024-05-23T00:00:32.819205+0000","last_fullsized":"2024-05-23T00:00:32.819205+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:53:07.591497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670094+0000","last_change":"2024-05-22T23:57:37.779097+0000","last_active":"2024-05-23T00:00:32.670094+0000","last_peered":"2024-05-23T00:00:32.670094+0000","last_clean":"2024-05-23T00:00:32.670094+0000","last_became_active":"2024-05-22T23:57:37.778564+0000","last_became_peered":"2024-05-22T23:57:37.778564+0000","last_unstale":"2024-05-23T00:00:32.670094+0000","last_undegraded":"2024-05-23T00:00:32.670094+0000","last_fullsized":"2024-05-23T00:00:32.670094+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:09:16.351947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"94'9","reported_seq":205,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.298705+0000","last_change":"2024-05-22T23:58:31.109013+0000","last_active":"2024-05-23T00:00:33.298705+0000","last_peered":"2024-05-23T00:00:33.298705+0000","last_clean":"2024-05-23T00:00:33.298705+0000","last_became_active":"2024-05-22T23:58:31.103858+0000","last_became_peered":"2024-05-22T23:58:31.103858+0000","last_unstale":"2024-05-23T00:00:33.298705+0000","last_undegraded":"2024-05-23T00:00:33.298705+0000","last_fullsized":"2024-05-23T00:00:33.298705+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:20:08.783189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1853,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":153,"num_read_kb":6,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022157+0000","last_change":"2024-05-22T23:57:33.324587+0000","last_active":"2024-05-23T00:00:33.022157+0000","last_peered":"2024-05-23T00:00:33.022157+0000","last_clean":"2024-05-23T00:00:33.022157+0000","last_became_active":"2024-05-22T23:57:33.323972+0000","last_became_peered":"2024-05-22T23:57:33.323972+0000","last_unstale":"2024-05-23T00:00:33.022157+0000","last_undegraded":"2024-05-23T00:00:33.022157+0000","last_fullsized":"2024-05-23T00:00:33.022157+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:23:52.551699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670431+0000","last_change":"2024-05-22T23:57:37.791726+0000","last_active":"2024-05-23T00:00:32.670431+0000","last_peered":"2024-05-23T00:00:32.670431+0000","last_clean":"2024-05-23T00:00:32.670431+0000","last_became_active":"2024-05-22T23:57:37.791185+0000","last_became_peered":"2024-05-22T23:57:37.791185+0000","last_unstale":"2024-05-23T00:00:32.670431+0000","last_undegraded":"2024-05-23T00:00:32.670431+0000","last_fullsized":"2024-05-23T00:00:32.670431+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:55:30.994272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"91'3","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822489+0000","last_change":"2024-05-22T23:58:31.116594+0000","last_active":"2024-05-23T00:00:32.822489+0000","last_peered":"2024-05-23T00:00:32.822489+0000","last_clean":"2024-05-23T00:00:32.822489+0000","last_became_active":"2024-05-22T23:58:31.113117+0000","last_became_peered":"2024-05-22T23:58:31.113117+0000","last_unstale":"2024-05-23T00:00:32.822489+0000","last_undegraded":"2024-05-23T00:00:32.822489+0000","last_fullsized":"2024-05-23T00:00:32.822489+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:05:30.758638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"93'8","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819100+0000","last_change":"2024-05-22T23:58:31.116203+0000","last_active":"2024-05-23T00:00:32.819100+0000","last_peered":"2024-05-23T00:00:32.819100+0000","last_clean":"2024-05-23T00:00:32.819100+0000","last_became_active":"2024-05-22T23:58:31.115881+0000","last_became_peered":"2024-05-22T23:58:31.115881+0000","last_unstale":"2024-05-23T00:00:32.819100+0000","last_undegraded":"2024-05-23T00:00:32.819100+0000","last_fullsized":"2024-05-23T00:00:32.819100+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:39:31.247426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809889+0000","last_change":"2024-05-22T23:57:37.787910+0000","last_active":"2024-05-23T00:00:32.809889+0000","last_peered":"2024-05-23T00:00:32.809889+0000","last_clean":"2024-05-23T00:00:32.809889+0000","last_became_active":"2024-05-22T23:57:37.787465+0000","last_became_peered":"2024-05-22T23:57:37.787465+0000","last_unstale":"2024-05-23T00:00:32.809889+0000","last_undegraded":"2024-05-23T00:00:32.809889+0000","last_fullsized":"2024-05-23T00:00:32.809889+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:54:59.202855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023402+0000","last_change":"2024-05-22T23:57:32.329365+0000","last_active":"2024-05-23T00:00:33.023402+0000","last_peered":"2024-05-23T00:00:33.023402+0000","last_clean":"2024-05-23T00:00:33.023402+0000","last_became_active":"2024-05-22T23:57:32.327864+0000","last_became_peered":"2024-05-22T23:57:32.327864+0000","last_unstale":"2024-05-23T00:00:33.023402+0000","last_undegraded":"2024-05-23T00:00:33.023402+0000","last_fullsized":"2024-05-23T00:00:33.023402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:26.998018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809992+0000","last_change":"2024-05-22T23:57:33.313930+0000","last_active":"2024-05-23T00:00:32.809992+0000","last_peered":"2024-05-23T00:00:32.809992+0000","last_clean":"2024-05-23T00:00:32.809992+0000","last_became_active":"2024-05-22T23:57:33.313612+0000","last_became_peered":"2024-05-22T23:57:33.313612+0000","last_unstale":"2024-05-23T00:00:32.809992+0000","last_undegraded":"2024-05-23T00:00:32.809992+0000","last_fullsized":"2024-05-23T00:00:32.809992+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:45:25.800338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670644+0000","last_change":"2024-05-22T23:57:37.791537+0000","last_active":"2024-05-23T00:00:32.670644+0000","last_peered":"2024-05-23T00:00:32.670644+0000","last_clean":"2024-05-23T00:00:32.670644+0000","last_became_active":"2024-05-22T23:57:37.791052+0000","last_became_peered":"2024-05-22T23:57:37.791052+0000","last_unstale":"2024-05-23T00:00:32.670644+0000","last_undegraded":"2024-05-23T00:00:32.670644+0000","last_fullsized":"2024-05-23T00:00:32.670644+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:18:11.759840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"91'1","reported_seq":31,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818596+0000","last_change":"2024-05-22T23:57:32.315907+0000","last_active":"2024-05-23T00:00:32.818596+0000","last_peered":"2024-05-23T00:00:32.818596+0000","last_clean":"2024-05-23T00:00:32.818596+0000","last_became_active":"2024-05-22T23:57:32.315576+0000","last_became_peered":"2024-05-22T23:57:32.315576+0000","last_unstale":"2024-05-23T00:00:32.818596+0000","last_undegraded":"2024-05-23T00:00:32.818596+0000","last_fullsized":"2024-05-23T00:00:32.818596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:45:48.998377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671704+0000","last_change":"2024-05-22T23:57:33.338681+0000","last_active":"2024-05-23T00:00:32.671704+0000","last_peered":"2024-05-23T00:00:32.671704+0000","last_clean":"2024-05-23T00:00:32.671704+0000","last_became_active":"2024-05-22T23:57:33.334705+0000","last_became_peered":"2024-05-22T23:57:33.334705+0000","last_unstale":"2024-05-23T00:00:32.671704+0000","last_undegraded":"2024-05-23T00:00:32.671704+0000","last_fullsized":"2024-05-23T00:00:32.671704+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:22.135137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810677+0000","last_change":"2024-05-22T23:57:37.780470+0000","last_active":"2024-05-23T00:00:32.810677+0000","last_peered":"2024-05-23T00:00:32.810677+0000","last_clean":"2024-05-23T00:00:32.810677+0000","last_became_active":"2024-05-22T23:57:37.779110+0000","last_became_peered":"2024-05-22T23:57:37.779110+0000","last_unstale":"2024-05-23T00:00:32.810677+0000","last_undegraded":"2024-05-23T00:00:32.810677+0000","last_fullsized":"2024-05-23T00:00:32.810677+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:55:45.486862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":183,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822985+0000","last_change":"2024-05-22T23:57:32.322489+0000","last_active":"2024-05-23T00:00:32.822985+0000","last_peered":"2024-05-23T00:00:32.822985+0000","last_clean":"2024-05-23T00:00:32.822985+0000","last_became_active":"2024-05-22T23:57:32.322125+0000","last_became_peered":"2024-05-22T23:57:32.322125+0000","last_unstale":"2024-05-23T00:00:32.822985+0000","last_undegraded":"2024-05-23T00:00:32.822985+0000","last_fullsized":"2024-05-23T00:00:32.822985+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:11:47.707506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":148,"num_read_kb":148,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021521+0000","last_change":"2024-05-22T23:57:33.322417+0000","last_active":"2024-05-23T00:00:33.021521+0000","last_peered":"2024-05-23T00:00:33.021521+0000","last_clean":"2024-05-23T00:00:33.021521+0000","last_became_active":"2024-05-22T23:57:33.322102+0000","last_became_peered":"2024-05-22T23:57:33.322102+0000","last_unstale":"2024-05-23T00:00:33.021521+0000","last_undegraded":"2024-05-23T00:00:33.021521+0000","last_fullsized":"2024-05-23T00:00:33.021521+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:26:43.509057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671629+0000","last_change":"2024-05-22T23:57:37.784424+0000","last_active":"2024-05-23T00:00:32.671629+0000","last_peered":"2024-05-23T00:00:32.671629+0000","last_clean":"2024-05-23T00:00:32.671629+0000","last_became_active":"2024-05-22T23:57:37.783949+0000","last_became_peered":"2024-05-22T23:57:37.783949+0000","last_unstale":"2024-05-23T00:00:32.671629+0000","last_undegraded":"2024-05-23T00:00:32.671629+0000","last_fullsized":"2024-05-23T00:00:32.671629+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:52:23.763273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"91'2","reported_seq":42,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023938+0000","last_change":"2024-05-22T23:58:31.110764+0000","last_active":"2024-05-23T00:00:33.023938+0000","last_peered":"2024-05-23T00:00:33.023938+0000","last_clean":"2024-05-23T00:00:33.023938+0000","last_became_active":"2024-05-22T23:58:31.103962+0000","last_became_peered":"2024-05-22T23:58:31.103962+0000","last_unstale":"2024-05-23T00:00:33.023938+0000","last_undegraded":"2024-05-23T00:00:33.023938+0000","last_fullsized":"2024-05-23T00:00:33.023938+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:14:24.248179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670954+0000","last_change":"2024-05-22T23:57:33.324237+0000","last_active":"2024-05-23T00:00:32.670954+0000","last_peered":"2024-05-23T00:00:32.670954+0000","last_clean":"2024-05-23T00:00:32.670954+0000","last_became_active":"2024-05-22T23:57:33.323870+0000","last_became_peered":"2024-05-22T23:57:33.323870+0000","last_unstale":"2024-05-23T00:00:32.670954+0000","last_undegraded":"2024-05-23T00:00:32.670954+0000","last_fullsized":"2024-05-23T00:00:32.670954+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:41:29.328104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022755+0000","last_change":"2024-05-22T23:57:37.787795+0000","last_active":"2024-05-23T00:00:33.022755+0000","last_peered":"2024-05-23T00:00:33.022755+0000","last_clean":"2024-05-23T00:00:33.022755+0000","last_became_active":"2024-05-22T23:57:37.787210+0000","last_became_peered":"2024-05-22T23:57:37.787210+0000","last_unstale":"2024-05-23T00:00:33.022755+0000","last_undegraded":"2024-05-23T00:00:33.022755+0000","last_fullsized":"2024-05-23T00:00:33.022755+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:19:04.616758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819178+0000","last_change":"2024-05-22T23:57:32.321415+0000","last_active":"2024-05-23T00:00:32.819178+0000","last_peered":"2024-05-23T00:00:32.819178+0000","last_clean":"2024-05-23T00:00:32.819178+0000","last_became_active":"2024-05-22T23:57:32.321055+0000","last_became_peered":"2024-05-22T23:57:32.321055+0000","last_unstale":"2024-05-23T00:00:32.819178+0000","last_undegraded":"2024-05-23T00:00:32.819178+0000","last_fullsized":"2024-05-23T00:00:32.819178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:28:58.577548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"93'11","reported_seq":45,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021090+0000","last_change":"2024-05-22T23:58:31.116991+0000","last_active":"2024-05-23T00:00:33.021090+0000","last_peered":"2024-05-23T00:00:33.021090+0000","last_clean":"2024-05-23T00:00:33.021090+0000","last_became_active":"2024-05-22T23:58:31.116526+0000","last_became_peered":"2024-05-22T23:58:31.116526+0000","last_unstale":"2024-05-23T00:00:33.021090+0000","last_undegraded":"2024-05-23T00:00:33.021090+0000","last_fullsized":"2024-05-23T00:00:33.021090+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:51:23.350871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671835+0000","last_change":"2024-05-22T23:57:37.781367+0000","last_active":"2024-05-23T00:00:32.671835+0000","last_peered":"2024-05-23T00:00:32.671835+0000","last_clean":"2024-05-23T00:00:32.671835+0000","last_became_active":"2024-05-22T23:57:37.780955+0000","last_became_peered":"2024-05-22T23:57:37.780955+0000","last_unstale":"2024-05-23T00:00:32.671835+0000","last_undegraded":"2024-05-23T00:00:32.671835+0000","last_fullsized":"2024-05-23T00:00:32.671835+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:36:18.962940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"94'42","reported_seq":380,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.298872+0000","last_change":"2024-05-22T23:57:32.313873+0000","last_active":"2024-05-23T00:00:33.298872+0000","last_peered":"2024-05-23T00:00:33.298872+0000","last_clean":"2024-05-23T00:00:33.298872+0000","last_became_active":"2024-05-22T23:57:32.313388+0000","last_became_peered":"2024-05-22T23:57:32.313388+0000","last_unstale":"2024-05-23T00:00:33.298872+0000","last_undegraded":"2024-05-23T00:00:33.298872+0000","last_fullsized":"2024-05-23T00:00:33.298872+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:39:16.383384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69043,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":308,"num_read_kb":221,"num_write":43,"num_write_kb":91,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022567+0000","last_change":"2024-05-22T23:57:33.318751+0000","last_active":"2024-05-23T00:00:33.022567+0000","last_peered":"2024-05-23T00:00:33.022567+0000","last_clean":"2024-05-23T00:00:33.022567+0000","last_became_active":"2024-05-22T23:57:33.317710+0000","last_became_peered":"2024-05-22T23:57:33.317710+0000","last_unstale":"2024-05-23T00:00:33.022567+0000","last_undegraded":"2024-05-23T00:00:33.022567+0000","last_fullsized":"2024-05-23T00:00:33.022567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:57:50.852615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819509+0000","last_change":"2024-05-22T23:57:37.790073+0000","last_active":"2024-05-23T00:00:32.819509+0000","last_peered":"2024-05-23T00:00:32.819509+0000","last_clean":"2024-05-23T00:00:32.819509+0000","last_became_active":"2024-05-22T23:57:37.789477+0000","last_became_peered":"2024-05-22T23:57:37.789477+0000","last_unstale":"2024-05-23T00:00:32.819509+0000","last_undegraded":"2024-05-23T00:00:32.819509+0000","last_fullsized":"2024-05-23T00:00:32.819509+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:11.003201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"91'6","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670178+0000","last_change":"2024-05-22T23:57:32.329846+0000","last_active":"2024-05-23T00:00:32.670178+0000","last_peered":"2024-05-23T00:00:32.670178+0000","last_clean":"2024-05-23T00:00:32.670178+0000","last_became_active":"2024-05-22T23:57:32.329414+0000","last_became_peered":"2024-05-22T23:57:32.329414+0000","last_unstale":"2024-05-23T00:00:32.670178+0000","last_undegraded":"2024-05-23T00:00:32.670178+0000","last_fullsized":"2024-05-23T00:00:32.670178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:15:38.176790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023754+0000","last_change":"2024-05-22T23:57:33.333298+0000","last_active":"2024-05-23T00:00:33.023754+0000","last_peered":"2024-05-23T00:00:33.023754+0000","last_clean":"2024-05-23T00:00:33.023754+0000","last_became_active":"2024-05-22T23:57:33.332934+0000","last_became_peered":"2024-05-22T23:57:33.332934+0000","last_unstale":"2024-05-23T00:00:33.023754+0000","last_undegraded":"2024-05-23T00:00:33.023754+0000","last_fullsized":"2024-05-23T00:00:33.023754+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:06:00.427627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809610+0000","last_change":"2024-05-22T23:57:37.795628+0000","last_active":"2024-05-23T00:00:32.809610+0000","last_peered":"2024-05-23T00:00:32.809610+0000","last_clean":"2024-05-23T00:00:32.809610+0000","last_became_active":"2024-05-22T23:57:37.795208+0000","last_became_peered":"2024-05-22T23:57:37.795208+0000","last_unstale":"2024-05-23T00:00:32.809610+0000","last_undegraded":"2024-05-23T00:00:32.809610+0000","last_fullsized":"2024-05-23T00:00:32.809610+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:03:12.684474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818880+0000","last_change":"2024-05-22T23:57:32.334039+0000","last_active":"2024-05-23T00:00:32.818880+0000","last_peered":"2024-05-23T00:00:32.818880+0000","last_clean":"2024-05-23T00:00:32.818880+0000","last_became_active":"2024-05-22T23:57:32.333709+0000","last_became_peered":"2024-05-22T23:57:32.333709+0000","last_unstale":"2024-05-23T00:00:32.818880+0000","last_undegraded":"2024-05-23T00:00:32.818880+0000","last_fullsized":"2024-05-23T00:00:32.818880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:41:43.319094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023784+0000","last_change":"2024-05-22T23:57:33.328056+0000","last_active":"2024-05-23T00:00:33.023784+0000","last_peered":"2024-05-23T00:00:33.023784+0000","last_clean":"2024-05-23T00:00:33.023784+0000","last_became_active":"2024-05-22T23:57:33.326559+0000","last_became_peered":"2024-05-22T23:57:33.326559+0000","last_unstale":"2024-05-23T00:00:33.023784+0000","last_undegraded":"2024-05-23T00:00:33.023784+0000","last_fullsized":"2024-05-23T00:00:33.023784+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:36:06.463837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672875+0000","last_change":"2024-05-22T23:57:37.795892+0000","last_active":"2024-05-23T00:00:32.672875+0000","last_peered":"2024-05-23T00:00:32.672875+0000","last_clean":"2024-05-23T00:00:32.672875+0000","last_became_active":"2024-05-22T23:57:37.792246+0000","last_became_peered":"2024-05-22T23:57:37.792246+0000","last_unstale":"2024-05-23T00:00:32.672875+0000","last_undegraded":"2024-05-23T00:00:32.672875+0000","last_fullsized":"2024-05-23T00:00:32.672875+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:18.462267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"94'6","reported_seq":190,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.560988+0000","last_change":"2024-05-22T23:57:32.319379+0000","last_active":"2024-05-23T00:00:33.560988+0000","last_peered":"2024-05-23T00:00:33.560988+0000","last_clean":"2024-05-23T00:00:33.560988+0000","last_became_active":"2024-05-22T23:57:32.318803+0000","last_became_peered":"2024-05-22T23:57:32.318803+0000","last_unstale":"2024-05-23T00:00:33.560988+0000","last_undegraded":"2024-05-23T00:00:33.560988+0000","last_fullsized":"2024-05-23T00:00:33.560988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:59:15.689119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1509,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023586+0000","last_change":"2024-05-22T23:57:33.325294+0000","last_active":"2024-05-23T00:00:33.023586+0000","last_peered":"2024-05-23T00:00:33.023586+0000","last_clean":"2024-05-23T00:00:33.023586+0000","last_became_active":"2024-05-22T23:57:33.324339+0000","last_became_peered":"2024-05-22T23:57:33.324339+0000","last_unstale":"2024-05-23T00:00:33.023586+0000","last_undegraded":"2024-05-23T00:00:33.023586+0000","last_fullsized":"2024-05-23T00:00:33.023586+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:29:47.591828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024152+0000","last_change":"2024-05-22T23:57:37.797961+0000","last_active":"2024-05-23T00:00:33.024152+0000","last_peered":"2024-05-23T00:00:33.024152+0000","last_clean":"2024-05-23T00:00:33.024152+0000","last_became_active":"2024-05-22T23:57:37.797532+0000","last_became_peered":"2024-05-22T23:57:37.797532+0000","last_unstale":"2024-05-23T00:00:33.024152+0000","last_undegraded":"2024-05-23T00:00:33.024152+0000","last_fullsized":"2024-05-23T00:00:33.024152+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:51:12.997674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"91'15","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022065+0000","last_change":"2024-05-22T23:57:32.317434+0000","last_active":"2024-05-23T00:00:33.022065+0000","last_peered":"2024-05-23T00:00:33.022065+0000","last_clean":"2024-05-23T00:00:33.022065+0000","last_became_active":"2024-05-22T23:57:32.316987+0000","last_became_peered":"2024-05-22T23:57:32.316987+0000","last_unstale":"2024-05-23T00:00:33.022065+0000","last_undegraded":"2024-05-23T00:00:33.022065+0000","last_fullsized":"2024-05-23T00:00:33.022065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:46:50.306756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818865+0000","last_change":"2024-05-22T23:57:33.328670+0000","last_active":"2024-05-23T00:00:32.818865+0000","last_peered":"2024-05-23T00:00:32.818865+0000","last_clean":"2024-05-23T00:00:32.818865+0000","last_became_active":"2024-05-22T23:57:33.328294+0000","last_became_peered":"2024-05-22T23:57:33.328294+0000","last_unstale":"2024-05-23T00:00:32.818865+0000","last_undegraded":"2024-05-23T00:00:32.818865+0000","last_fullsized":"2024-05-23T00:00:32.818865+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:09:10.807844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818693+0000","last_change":"2024-05-22T23:57:37.788885+0000","last_active":"2024-05-23T00:00:32.818693+0000","last_peered":"2024-05-23T00:00:32.818693+0000","last_clean":"2024-05-23T00:00:32.818693+0000","last_became_active":"2024-05-22T23:57:37.788145+0000","last_became_peered":"2024-05-22T23:57:37.788145+0000","last_unstale":"2024-05-23T00:00:32.818693+0000","last_undegraded":"2024-05-23T00:00:32.818693+0000","last_fullsized":"2024-05-23T00:00:32.818693+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:14:50.027785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'2","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022537+0000","last_change":"2024-05-22T23:57:32.317855+0000","last_active":"2024-05-23T00:00:33.022537+0000","last_peered":"2024-05-23T00:00:33.022537+0000","last_clean":"2024-05-23T00:00:33.022537+0000","last_became_active":"2024-05-22T23:57:32.317087+0000","last_became_peered":"2024-05-22T23:57:32.317087+0000","last_unstale":"2024-05-23T00:00:33.022537+0000","last_undegraded":"2024-05-23T00:00:33.022537+0000","last_fullsized":"2024-05-23T00:00:33.022537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:49:40.486222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809847+0000","last_change":"2024-05-22T23:57:33.323897+0000","last_active":"2024-05-23T00:00:32.809847+0000","last_peered":"2024-05-23T00:00:32.809847+0000","last_clean":"2024-05-23T00:00:32.809847+0000","last_became_active":"2024-05-22T23:57:33.323567+0000","last_became_peered":"2024-05-22T23:57:33.323567+0000","last_unstale":"2024-05-23T00:00:32.809847+0000","last_undegraded":"2024-05-23T00:00:32.809847+0000","last_fullsized":"2024-05-23T00:00:32.809847+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:17:48.125829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022150+0000","last_change":"2024-05-22T23:57:38.218313+0000","last_active":"2024-05-23T00:00:33.022150+0000","last_peered":"2024-05-23T00:00:33.022150+0000","last_clean":"2024-05-23T00:00:33.022150+0000","last_became_active":"2024-05-22T23:57:38.217756+0000","last_became_peered":"2024-05-22T23:57:38.217756+0000","last_unstale":"2024-05-23T00:00:33.022150+0000","last_undegraded":"2024-05-23T00:00:33.022150+0000","last_fullsized":"2024-05-23T00:00:33.022150+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:07:55.122260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672176+0000","last_change":"2024-05-22T23:57:32.325729+0000","last_active":"2024-05-23T00:00:32.672176+0000","last_peered":"2024-05-23T00:00:32.672176+0000","last_clean":"2024-05-23T00:00:32.672176+0000","last_became_active":"2024-05-22T23:57:32.325383+0000","last_became_peered":"2024-05-22T23:57:32.325383+0000","last_unstale":"2024-05-23T00:00:32.672176+0000","last_undegraded":"2024-05-23T00:00:32.672176+0000","last_fullsized":"2024-05-23T00:00:32.672176+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-23T23:58:57.978806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822720+0000","last_change":"2024-05-22T23:57:33.337499+0000","last_active":"2024-05-23T00:00:32.822720+0000","last_peered":"2024-05-23T00:00:32.822720+0000","last_clean":"2024-05-23T00:00:32.822720+0000","last_became_active":"2024-05-22T23:57:33.337271+0000","last_became_peered":"2024-05-22T23:57:33.337271+0000","last_unstale":"2024-05-23T00:00:32.822720+0000","last_undegraded":"2024-05-23T00:00:32.822720+0000","last_fullsized":"2024-05-23T00:00:32.822720+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T02:32:39.133984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024205+0000","last_change":"2024-05-22T23:57:37.792707+0000","last_active":"2024-05-23T00:00:33.024205+0000","last_peered":"2024-05-23T00:00:33.024205+0000","last_clean":"2024-05-23T00:00:33.024205+0000","last_became_active":"2024-05-22T23:57:37.792209+0000","last_became_peered":"2024-05-22T23:57:37.792209+0000","last_unstale":"2024-05-23T00:00:33.024205+0000","last_undegraded":"2024-05-23T00:00:33.024205+0000","last_fullsized":"2024-05-23T00:00:33.024205+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:36:34.089424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"91'1","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809543+0000","last_change":"2024-05-22T23:58:31.112612+0000","last_active":"2024-05-23T00:00:32.809543+0000","last_peered":"2024-05-23T00:00:32.809543+0000","last_clean":"2024-05-23T00:00:32.809543+0000","last_became_active":"2024-05-22T23:58:31.112181+0000","last_became_peered":"2024-05-22T23:58:31.112181+0000","last_unstale":"2024-05-23T00:00:32.809543+0000","last_undegraded":"2024-05-23T00:00:32.809543+0000","last_fullsized":"2024-05-23T00:00:32.809543+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:25:18.717294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.822488+0000","last_change":"2024-05-22T23:57:33.328129+0000","last_active":"2024-05-23T00:00:32.822488+0000","last_peered":"2024-05-23T00:00:32.822488+0000","last_clean":"2024-05-23T00:00:32.822488+0000","last_became_active":"2024-05-22T23:57:33.327732+0000","last_became_peered":"2024-05-22T23:57:33.327732+0000","last_unstale":"2024-05-23T00:00:32.822488+0000","last_undegraded":"2024-05-23T00:00:32.822488+0000","last_fullsized":"2024-05-23T00:00:32.822488+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:25:14.105235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024320+0000","last_change":"2024-05-22T23:57:38.220884+0000","last_active":"2024-05-23T00:00:33.024320+0000","last_peered":"2024-05-23T00:00:33.024320+0000","last_clean":"2024-05-23T00:00:33.024320+0000","last_became_active":"2024-05-22T23:57:38.219365+0000","last_became_peered":"2024-05-22T23:57:38.219365+0000","last_unstale":"2024-05-23T00:00:33.024320+0000","last_undegraded":"2024-05-23T00:00:33.024320+0000","last_fullsized":"2024-05-23T00:00:33.024320+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:54:56.455743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818771+0000","last_change":"2024-05-22T23:57:33.328272+0000","last_active":"2024-05-23T00:00:32.818771+0000","last_peered":"2024-05-23T00:00:32.818771+0000","last_clean":"2024-05-23T00:00:32.818771+0000","last_became_active":"2024-05-22T23:57:33.327915+0000","last_became_peered":"2024-05-22T23:57:33.327915+0000","last_unstale":"2024-05-23T00:00:32.818771+0000","last_undegraded":"2024-05-23T00:00:32.818771+0000","last_fullsized":"2024-05-23T00:00:32.818771+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:59:09.085649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"94'6","reported_seq":185,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.031020+0000","last_change":"2024-05-22T23:57:32.331979+0000","last_active":"2024-05-23T00:00:33.031020+0000","last_peered":"2024-05-23T00:00:33.031020+0000","last_clean":"2024-05-23T00:00:33.031020+0000","last_became_active":"2024-05-22T23:57:32.330983+0000","last_became_peered":"2024-05-22T23:57:32.330983+0000","last_unstale":"2024-05-23T00:00:33.031020+0000","last_undegraded":"2024-05-23T00:00:33.031020+0000","last_fullsized":"2024-05-23T00:00:33.031020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:07:19.137652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023940+0000","last_change":"2024-05-22T23:57:37.787935+0000","last_active":"2024-05-23T00:00:33.023940+0000","last_peered":"2024-05-23T00:00:33.023940+0000","last_clean":"2024-05-23T00:00:33.023940+0000","last_became_active":"2024-05-22T23:57:37.787495+0000","last_became_peered":"2024-05-22T23:57:37.787495+0000","last_unstale":"2024-05-23T00:00:33.023940+0000","last_undegraded":"2024-05-23T00:00:33.023940+0000","last_fullsized":"2024-05-23T00:00:33.023940+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:07:37.682711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810969+0000","last_change":"2024-05-22T23:57:33.329451+0000","last_active":"2024-05-23T00:00:32.810969+0000","last_peered":"2024-05-23T00:00:32.810969+0000","last_clean":"2024-05-23T00:00:32.810969+0000","last_became_active":"2024-05-22T23:57:33.329107+0000","last_became_peered":"2024-05-22T23:57:33.329107+0000","last_unstale":"2024-05-23T00:00:32.810969+0000","last_undegraded":"2024-05-23T00:00:32.810969+0000","last_fullsized":"2024-05-23T00:00:32.810969+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:44:50.198040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"91'3","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022720+0000","last_change":"2024-05-22T23:57:32.321873+0000","last_active":"2024-05-23T00:00:33.022720+0000","last_peered":"2024-05-23T00:00:33.022720+0000","last_clean":"2024-05-23T00:00:33.022720+0000","last_became_active":"2024-05-22T23:57:32.321550+0000","last_became_peered":"2024-05-22T23:57:32.321550+0000","last_unstale":"2024-05-23T00:00:33.022720+0000","last_undegraded":"2024-05-23T00:00:33.022720+0000","last_fullsized":"2024-05-23T00:00:33.022720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:30:17.479238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":363,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023381+0000","last_change":"2024-05-22T23:56:11.327788+0000","last_active":"2024-05-23T00:00:33.023381+0000","last_peered":"2024-05-23T00:00:33.023381+0000","last_clean":"2024-05-23T00:00:33.023381+0000","last_became_active":"2024-05-22T23:56:11.327232+0000","last_became_peered":"2024-05-22T23:56:11.327232+0000","last_unstale":"2024-05-23T00:00:33.023381+0000","last_undegraded":"2024-05-23T00:00:33.023381+0000","last_fullsized":"2024-05-23T00:00:33.023381+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:52:16.719643+0000","last_clean_scrub_stamp":"2024-05-22T23:52:16.719643+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:26:13.077470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.819294+0000","last_change":"2024-05-22T23:57:38.219326+0000","last_active":"2024-05-23T00:00:32.819294+0000","last_peered":"2024-05-23T00:00:32.819294+0000","last_clean":"2024-05-23T00:00:32.819294+0000","last_became_active":"2024-05-22T23:57:38.218835+0000","last_became_peered":"2024-05-22T23:57:38.218835+0000","last_unstale":"2024-05-23T00:00:32.819294+0000","last_undegraded":"2024-05-23T00:00:32.819294+0000","last_fullsized":"2024-05-23T00:00:32.819294+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:22:29.722862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023039+0000","last_change":"2024-05-22T23:57:33.316760+0000","last_active":"2024-05-23T00:00:33.023039+0000","last_peered":"2024-05-23T00:00:33.023039+0000","last_clean":"2024-05-23T00:00:33.023039+0000","last_became_active":"2024-05-22T23:57:33.316328+0000","last_became_peered":"2024-05-22T23:57:33.316328+0000","last_unstale":"2024-05-23T00:00:33.023039+0000","last_undegraded":"2024-05-23T00:00:33.023039+0000","last_fullsized":"2024-05-23T00:00:33.023039+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:53:11.592134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023557+0000","last_change":"2024-05-22T23:57:32.319154+0000","last_active":"2024-05-23T00:00:33.023557+0000","last_peered":"2024-05-23T00:00:33.023557+0000","last_clean":"2024-05-23T00:00:33.023557+0000","last_became_active":"2024-05-22T23:57:32.318808+0000","last_became_peered":"2024-05-22T23:57:32.318808+0000","last_unstale":"2024-05-23T00:00:33.023557+0000","last_undegraded":"2024-05-23T00:00:33.023557+0000","last_fullsized":"2024-05-23T00:00:33.023557+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:37:47.271767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809550+0000","last_change":"2024-05-22T23:57:37.791464+0000","last_active":"2024-05-23T00:00:32.809550+0000","last_peered":"2024-05-23T00:00:32.809550+0000","last_clean":"2024-05-23T00:00:32.809550+0000","last_became_active":"2024-05-22T23:57:37.791044+0000","last_became_peered":"2024-05-22T23:57:37.791044+0000","last_unstale":"2024-05-23T00:00:32.809550+0000","last_undegraded":"2024-05-23T00:00:32.809550+0000","last_fullsized":"2024-05-23T00:00:32.809550+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:47:00.698303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"91'1","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022492+0000","last_change":"2024-05-22T23:57:32.322740+0000","last_active":"2024-05-23T00:00:33.022492+0000","last_peered":"2024-05-23T00:00:33.022492+0000","last_clean":"2024-05-23T00:00:33.022492+0000","last_became_active":"2024-05-22T23:57:32.322425+0000","last_became_peered":"2024-05-22T23:57:32.322425+0000","last_unstale":"2024-05-23T00:00:33.022492+0000","last_undegraded":"2024-05-23T00:00:33.022492+0000","last_fullsized":"2024-05-23T00:00:33.022492+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:05:10.522462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023588+0000","last_change":"2024-05-22T23:57:33.324667+0000","last_active":"2024-05-23T00:00:33.023588+0000","last_peered":"2024-05-23T00:00:33.023588+0000","last_clean":"2024-05-23T00:00:33.023588+0000","last_became_active":"2024-05-22T23:57:33.324251+0000","last_became_peered":"2024-05-22T23:57:33.324251+0000","last_unstale":"2024-05-23T00:00:33.023588+0000","last_undegraded":"2024-05-23T00:00:33.023588+0000","last_fullsized":"2024-05-23T00:00:33.023588+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:49:57.203996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809807+0000","last_change":"2024-05-22T23:57:38.222346+0000","last_active":"2024-05-23T00:00:32.809807+0000","last_peered":"2024-05-23T00:00:32.809807+0000","last_clean":"2024-05-23T00:00:32.809807+0000","last_became_active":"2024-05-22T23:57:38.220248+0000","last_became_peered":"2024-05-22T23:57:38.220248+0000","last_unstale":"2024-05-23T00:00:32.809807+0000","last_undegraded":"2024-05-23T00:00:32.809807+0000","last_fullsized":"2024-05-23T00:00:32.809807+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:26:44.026641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"94'44","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.963348+0000","last_change":"2024-05-22T23:58:31.126056+0000","last_active":"2024-05-23T00:00:32.963348+0000","last_peered":"2024-05-23T00:00:32.963348+0000","last_clean":"2024-05-23T00:00:32.963348+0000","last_became_active":"2024-05-22T23:58:31.117186+0000","last_became_peered":"2024-05-22T23:58:31.117186+0000","last_unstale":"2024-05-23T00:00:32.963348+0000","last_undegraded":"2024-05-23T00:00:32.963348+0000","last_fullsized":"2024-05-23T00:00:32.963348+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:21:34.156655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":79,"num_write_kb":836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"93'5","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.671990+0000","last_change":"2024-05-22T23:58:31.116649+0000","last_active":"2024-05-23T00:00:32.671990+0000","last_peered":"2024-05-23T00:00:32.671990+0000","last_clean":"2024-05-23T00:00:32.671990+0000","last_became_active":"2024-05-22T23:58:31.116009+0000","last_became_peered":"2024-05-22T23:58:31.116009+0000","last_unstale":"2024-05-23T00:00:32.671990+0000","last_undegraded":"2024-05-23T00:00:32.671990+0000","last_fullsized":"2024-05-23T00:00:32.671990+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:00:29.964565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669481+0000","last_change":"2024-05-22T23:57:37.778524+0000","last_active":"2024-05-23T00:00:32.669481+0000","last_peered":"2024-05-23T00:00:32.669481+0000","last_clean":"2024-05-23T00:00:32.669481+0000","last_became_active":"2024-05-22T23:57:37.777827+0000","last_became_peered":"2024-05-22T23:57:37.777827+0000","last_unstale":"2024-05-23T00:00:32.669481+0000","last_undegraded":"2024-05-23T00:00:32.669481+0000","last_fullsized":"2024-05-23T00:00:32.669481+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:30:59.154921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"91'4","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023206+0000","last_change":"2024-05-22T23:57:32.317649+0000","last_active":"2024-05-23T00:00:33.023206+0000","last_peered":"2024-05-23T00:00:33.023206+0000","last_clean":"2024-05-23T00:00:33.023206+0000","last_became_active":"2024-05-22T23:57:32.317280+0000","last_became_peered":"2024-05-22T23:57:32.317280+0000","last_unstale":"2024-05-23T00:00:33.023206+0000","last_undegraded":"2024-05-23T00:00:33.023206+0000","last_fullsized":"2024-05-23T00:00:33.023206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:01:16.917079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670710+0000","last_change":"2024-05-22T23:57:33.339296+0000","last_active":"2024-05-23T00:00:32.670710+0000","last_peered":"2024-05-23T00:00:32.670710+0000","last_clean":"2024-05-23T00:00:32.670710+0000","last_became_active":"2024-05-22T23:57:33.338457+0000","last_became_peered":"2024-05-22T23:57:33.338457+0000","last_unstale":"2024-05-23T00:00:32.670710+0000","last_undegraded":"2024-05-23T00:00:32.670710+0000","last_fullsized":"2024-05-23T00:00:32.670710+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:13:05.927893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023012+0000","last_change":"2024-05-22T23:57:38.218649+0000","last_active":"2024-05-23T00:00:33.023012+0000","last_peered":"2024-05-23T00:00:33.023012+0000","last_clean":"2024-05-23T00:00:33.023012+0000","last_became_active":"2024-05-22T23:57:38.218189+0000","last_became_peered":"2024-05-22T23:57:38.218189+0000","last_unstale":"2024-05-23T00:00:33.023012+0000","last_undegraded":"2024-05-23T00:00:33.023012+0000","last_fullsized":"2024-05-23T00:00:33.023012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:23:30.747293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"91'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020655+0000","last_change":"2024-05-22T23:57:32.309379+0000","last_active":"2024-05-23T00:00:33.020655+0000","last_peered":"2024-05-23T00:00:33.020655+0000","last_clean":"2024-05-23T00:00:33.020655+0000","last_became_active":"2024-05-22T23:57:32.309068+0000","last_became_peered":"2024-05-22T23:57:32.309068+0000","last_unstale":"2024-05-23T00:00:33.020655+0000","last_undegraded":"2024-05-23T00:00:33.020655+0000","last_fullsized":"2024-05-23T00:00:33.020655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:43:54.787996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022150+0000","last_change":"2024-05-22T23:57:33.320253+0000","last_active":"2024-05-23T00:00:33.022150+0000","last_peered":"2024-05-23T00:00:33.022150+0000","last_clean":"2024-05-23T00:00:33.022150+0000","last_became_active":"2024-05-22T23:57:33.319604+0000","last_became_peered":"2024-05-22T23:57:33.319604+0000","last_unstale":"2024-05-23T00:00:33.022150+0000","last_undegraded":"2024-05-23T00:00:33.022150+0000","last_fullsized":"2024-05-23T00:00:33.022150+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:23:51.745040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.672719+0000","last_change":"2024-05-22T23:57:37.795368+0000","last_active":"2024-05-23T00:00:32.672719+0000","last_peered":"2024-05-23T00:00:32.672719+0000","last_clean":"2024-05-23T00:00:32.672719+0000","last_became_active":"2024-05-22T23:57:37.792848+0000","last_became_peered":"2024-05-22T23:57:37.792848+0000","last_unstale":"2024-05-23T00:00:32.672719+0000","last_undegraded":"2024-05-23T00:00:32.672719+0000","last_fullsized":"2024-05-23T00:00:32.672719+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:45.859867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"91'3","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810299+0000","last_change":"2024-05-22T23:57:32.329147+0000","last_active":"2024-05-23T00:00:32.810299+0000","last_peered":"2024-05-23T00:00:32.810299+0000","last_clean":"2024-05-23T00:00:32.810299+0000","last_became_active":"2024-05-22T23:57:32.327254+0000","last_became_peered":"2024-05-22T23:57:32.327254+0000","last_unstale":"2024-05-23T00:00:32.810299+0000","last_undegraded":"2024-05-23T00:00:32.810299+0000","last_fullsized":"2024-05-23T00:00:32.810299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:55:24.915817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809447+0000","last_change":"2024-05-22T23:57:33.332843+0000","last_active":"2024-05-23T00:00:32.809447+0000","last_peered":"2024-05-23T00:00:32.809447+0000","last_clean":"2024-05-23T00:00:32.809447+0000","last_became_active":"2024-05-22T23:57:33.332497+0000","last_became_peered":"2024-05-22T23:57:33.332497+0000","last_unstale":"2024-05-23T00:00:32.809447+0000","last_undegraded":"2024-05-23T00:00:32.809447+0000","last_fullsized":"2024-05-23T00:00:32.809447+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:55:16.109676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"91'1","reported_seq":22,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023022+0000","last_change":"2024-05-22T23:57:37.787609+0000","last_active":"2024-05-23T00:00:33.023022+0000","last_peered":"2024-05-23T00:00:33.023022+0000","last_clean":"2024-05-23T00:00:33.023022+0000","last_became_active":"2024-05-22T23:57:37.787153+0000","last_became_peered":"2024-05-22T23:57:37.787153+0000","last_unstale":"2024-05-23T00:00:33.023022+0000","last_undegraded":"2024-05-23T00:00:33.023022+0000","last_fullsized":"2024-05-23T00:00:33.023022+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:57:51.640037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"94'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669838+0000","last_change":"2024-05-22T23:57:32.329784+0000","last_active":"2024-05-23T00:00:32.669838+0000","last_peered":"2024-05-23T00:00:32.669838+0000","last_clean":"2024-05-23T00:00:32.669838+0000","last_became_active":"2024-05-22T23:57:32.329393+0000","last_became_peered":"2024-05-22T23:57:32.329393+0000","last_unstale":"2024-05-23T00:00:32.669838+0000","last_undegraded":"2024-05-23T00:00:32.669838+0000","last_fullsized":"2024-05-23T00:00:32.669838+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:17:54.811962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.811747+0000","last_change":"2024-05-22T23:57:33.318803+0000","last_active":"2024-05-23T00:00:32.811747+0000","last_peered":"2024-05-23T00:00:32.811747+0000","last_clean":"2024-05-23T00:00:32.811747+0000","last_became_active":"2024-05-22T23:57:33.318170+0000","last_became_peered":"2024-05-22T23:57:33.318170+0000","last_unstale":"2024-05-23T00:00:32.811747+0000","last_undegraded":"2024-05-23T00:00:32.811747+0000","last_fullsized":"2024-05-23T00:00:32.811747+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:56:56.808305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023028+0000","last_change":"2024-05-22T23:57:38.218583+0000","last_active":"2024-05-23T00:00:33.023028+0000","last_peered":"2024-05-23T00:00:33.023028+0000","last_clean":"2024-05-23T00:00:33.023028+0000","last_became_active":"2024-05-22T23:57:38.218048+0000","last_became_peered":"2024-05-22T23:57:38.218048+0000","last_unstale":"2024-05-23T00:00:33.023028+0000","last_undegraded":"2024-05-23T00:00:33.023028+0000","last_fullsized":"2024-05-23T00:00:33.023028+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:10:57.579656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"91'4","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.022209+0000","last_change":"2024-05-22T23:57:32.317908+0000","last_active":"2024-05-23T00:00:33.022209+0000","last_peered":"2024-05-23T00:00:33.022209+0000","last_clean":"2024-05-23T00:00:33.022209+0000","last_became_active":"2024-05-22T23:57:32.317580+0000","last_became_peered":"2024-05-22T23:57:32.317580+0000","last_unstale":"2024-05-23T00:00:33.022209+0000","last_undegraded":"2024-05-23T00:00:33.022209+0000","last_fullsized":"2024-05-23T00:00:33.022209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:50:02.044817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"93'5","reported_seq":33,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023705+0000","last_change":"2024-05-22T23:57:33.321184+0000","last_active":"2024-05-23T00:00:33.023705+0000","last_peered":"2024-05-23T00:00:33.023705+0000","last_clean":"2024-05-23T00:00:33.023705+0000","last_became_active":"2024-05-22T23:57:33.320418+0000","last_became_peered":"2024-05-22T23:57:33.320418+0000","last_unstale":"2024-05-23T00:00:33.023705+0000","last_undegraded":"2024-05-23T00:00:33.023705+0000","last_fullsized":"2024-05-23T00:00:33.023705+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:22:43.019809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.020743+0000","last_change":"2024-05-22T23:57:37.795718+0000","last_active":"2024-05-23T00:00:33.020743+0000","last_peered":"2024-05-23T00:00:33.020743+0000","last_clean":"2024-05-23T00:00:33.020743+0000","last_became_active":"2024-05-22T23:57:37.795297+0000","last_became_peered":"2024-05-22T23:57:37.795297+0000","last_unstale":"2024-05-23T00:00:33.020743+0000","last_undegraded":"2024-05-23T00:00:33.020743+0000","last_fullsized":"2024-05-23T00:00:33.020743+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T09:56:47.415023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"91'3","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670584+0000","last_change":"2024-05-22T23:57:32.311921+0000","last_active":"2024-05-23T00:00:32.670584+0000","last_peered":"2024-05-23T00:00:32.670584+0000","last_clean":"2024-05-23T00:00:32.670584+0000","last_became_active":"2024-05-22T23:57:32.311600+0000","last_became_peered":"2024-05-22T23:57:32.311600+0000","last_unstale":"2024-05-23T00:00:32.670584+0000","last_undegraded":"2024-05-23T00:00:32.670584+0000","last_fullsized":"2024-05-23T00:00:32.670584+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:35:21.882773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024166+0000","last_change":"2024-05-22T23:57:33.321289+0000","last_active":"2024-05-23T00:00:33.024166+0000","last_peered":"2024-05-23T00:00:33.024166+0000","last_clean":"2024-05-23T00:00:33.024166+0000","last_became_active":"2024-05-22T23:57:33.320458+0000","last_became_peered":"2024-05-22T23:57:33.320458+0000","last_unstale":"2024-05-23T00:00:33.024166+0000","last_undegraded":"2024-05-23T00:00:33.024166+0000","last_fullsized":"2024-05-23T00:00:33.024166+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:02:05.065266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670012+0000","last_change":"2024-05-22T23:57:37.786062+0000","last_active":"2024-05-23T00:00:32.670012+0000","last_peered":"2024-05-23T00:00:32.670012+0000","last_clean":"2024-05-23T00:00:32.670012+0000","last_became_active":"2024-05-22T23:57:37.785624+0000","last_became_peered":"2024-05-22T23:57:37.785624+0000","last_unstale":"2024-05-23T00:00:32.670012+0000","last_undegraded":"2024-05-23T00:00:32.670012+0000","last_fullsized":"2024-05-23T00:00:32.670012+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T07:38:56.186155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"94'39","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.960516+0000","last_change":"2024-05-22T23:57:32.333909+0000","last_active":"2024-05-23T00:00:32.960516+0000","last_peered":"2024-05-23T00:00:32.960516+0000","last_clean":"2024-05-23T00:00:32.960516+0000","last_became_active":"2024-05-22T23:57:32.333531+0000","last_became_peered":"2024-05-22T23:57:32.333531+0000","last_unstale":"2024-05-23T00:00:32.960516+0000","last_undegraded":"2024-05-23T00:00:32.960516+0000","last_fullsized":"2024-05-23T00:00:32.960516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:53:21.843150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023876+0000","last_change":"2024-05-22T23:57:33.334366+0000","last_active":"2024-05-23T00:00:33.023876+0000","last_peered":"2024-05-23T00:00:33.023876+0000","last_clean":"2024-05-23T00:00:33.023876+0000","last_became_active":"2024-05-22T23:57:33.334037+0000","last_became_peered":"2024-05-22T23:57:33.334037+0000","last_unstale":"2024-05-23T00:00:33.023876+0000","last_undegraded":"2024-05-23T00:00:33.023876+0000","last_fullsized":"2024-05-23T00:00:33.023876+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:15:10.631087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818480+0000","last_change":"2024-05-22T23:57:37.780241+0000","last_active":"2024-05-23T00:00:32.818480+0000","last_peered":"2024-05-23T00:00:32.818480+0000","last_clean":"2024-05-23T00:00:32.818480+0000","last_became_active":"2024-05-22T23:57:37.779816+0000","last_became_peered":"2024-05-22T23:57:37.779816+0000","last_unstale":"2024-05-23T00:00:32.818480+0000","last_undegraded":"2024-05-23T00:00:32.818480+0000","last_fullsized":"2024-05-23T00:00:32.818480+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:36.736820+0000","last_clean_scrub_stamp":"2024-05-22T23:57:36.736820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:01:11.397681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'2","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024119+0000","last_change":"2024-05-22T23:57:32.334829+0000","last_active":"2024-05-23T00:00:33.024119+0000","last_peered":"2024-05-23T00:00:33.024119+0000","last_clean":"2024-05-23T00:00:33.024119+0000","last_became_active":"2024-05-22T23:57:32.334368+0000","last_became_peered":"2024-05-22T23:57:32.334368+0000","last_unstale":"2024-05-23T00:00:33.024119+0000","last_undegraded":"2024-05-23T00:00:33.024119+0000","last_fullsized":"2024-05-23T00:00:33.024119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:31.286941+0000","last_clean_scrub_stamp":"2024-05-22T23:57:31.286941+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:17:42.319441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024133+0000","last_change":"2024-05-22T23:57:33.318740+0000","last_active":"2024-05-23T00:00:33.024133+0000","last_peered":"2024-05-23T00:00:33.024133+0000","last_clean":"2024-05-23T00:00:33.024133+0000","last_became_active":"2024-05-22T23:57:33.318363+0000","last_became_peered":"2024-05-22T23:57:33.318363+0000","last_unstale":"2024-05-23T00:00:33.024133+0000","last_undegraded":"2024-05-23T00:00:33.024133+0000","last_fullsized":"2024-05-23T00:00:33.024133+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:51:42.319667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"93'9","reported_seq":37,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809555+0000","last_change":"2024-05-22T23:57:33.325582+0000","last_active":"2024-05-23T00:00:32.809555+0000","last_peered":"2024-05-23T00:00:32.809555+0000","last_clean":"2024-05-23T00:00:32.809555+0000","last_became_active":"2024-05-22T23:57:33.325268+0000","last_became_peered":"2024-05-22T23:57:33.325268+0000","last_unstale":"2024-05-23T00:00:32.809555+0000","last_undegraded":"2024-05-23T00:00:32.809555+0000","last_fullsized":"2024-05-23T00:00:32.809555+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:02:18.968634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"93'12","reported_seq":46,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669768+0000","last_change":"2024-05-22T23:58:31.109626+0000","last_active":"2024-05-23T00:00:32.669768+0000","last_peered":"2024-05-23T00:00:32.669768+0000","last_clean":"2024-05-23T00:00:32.669768+0000","last_became_active":"2024-05-22T23:58:31.109179+0000","last_became_peered":"2024-05-22T23:58:31.109179+0000","last_unstale":"2024-05-23T00:00:32.669768+0000","last_undegraded":"2024-05-23T00:00:32.669768+0000","last_fullsized":"2024-05-23T00:00:32.669768+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T01:01:36.653833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,8],"acting":[2,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.669573+0000","last_change":"2024-05-22T23:57:33.332935+0000","last_active":"2024-05-23T00:00:32.669573+0000","last_peered":"2024-05-23T00:00:32.669573+0000","last_clean":"2024-05-23T00:00:32.669573+0000","last_became_active":"2024-05-22T23:57:33.332607+0000","last_became_peered":"2024-05-22T23:57:33.332607+0000","last_unstale":"2024-05-23T00:00:32.669573+0000","last_undegraded":"2024-05-23T00:00:32.669573+0000","last_fullsized":"2024-05-23T00:00:32.669573+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T04:23:09.589896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"93'7","reported_seq":35,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670318+0000","last_change":"2024-05-22T23:57:33.333269+0000","last_active":"2024-05-23T00:00:32.670318+0000","last_peered":"2024-05-23T00:00:32.670318+0000","last_clean":"2024-05-23T00:00:32.670318+0000","last_became_active":"2024-05-22T23:57:33.329017+0000","last_became_peered":"2024-05-22T23:57:33.329017+0000","last_unstale":"2024-05-23T00:00:32.670318+0000","last_undegraded":"2024-05-23T00:00:32.670318+0000","last_fullsized":"2024-05-23T00:00:32.670318+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:35:42.156434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"93'4","reported_seq":32,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.023462+0000","last_change":"2024-05-22T23:57:33.328721+0000","last_active":"2024-05-23T00:00:33.023462+0000","last_peered":"2024-05-23T00:00:33.023462+0000","last_clean":"2024-05-23T00:00:33.023462+0000","last_became_active":"2024-05-22T23:57:33.328318+0000","last_became_peered":"2024-05-22T23:57:33.328318+0000","last_unstale":"2024-05-23T00:00:33.023462+0000","last_undegraded":"2024-05-23T00:00:33.023462+0000","last_fullsized":"2024-05-23T00:00:33.023462+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:48:14.772688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809824+0000","last_change":"2024-05-22T23:57:33.324777+0000","last_active":"2024-05-23T00:00:32.809824+0000","last_peered":"2024-05-23T00:00:32.809824+0000","last_clean":"2024-05-23T00:00:32.809824+0000","last_became_active":"2024-05-22T23:57:33.324446+0000","last_became_peered":"2024-05-22T23:57:33.324446+0000","last_unstale":"2024-05-23T00:00:32.809824+0000","last_undegraded":"2024-05-23T00:00:32.809824+0000","last_fullsized":"2024-05-23T00:00:32.809824+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T05:20:42.327094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"93'15","reported_seq":43,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.670314+0000","last_change":"2024-05-22T23:57:33.318783+0000","last_active":"2024-05-23T00:00:32.670314+0000","last_peered":"2024-05-23T00:00:32.670314+0000","last_clean":"2024-05-23T00:00:32.670314+0000","last_became_active":"2024-05-22T23:57:33.318481+0000","last_became_peered":"2024-05-22T23:57:33.318481+0000","last_unstale":"2024-05-23T00:00:32.670314+0000","last_undegraded":"2024-05-23T00:00:32.670314+0000","last_fullsized":"2024-05-23T00:00:32.670314+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T08:14:35.745683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"93'2","reported_seq":30,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.812167+0000","last_change":"2024-05-22T23:57:33.319939+0000","last_active":"2024-05-23T00:00:32.812167+0000","last_peered":"2024-05-23T00:00:32.812167+0000","last_clean":"2024-05-23T00:00:32.812167+0000","last_became_active":"2024-05-22T23:57:33.316029+0000","last_became_peered":"2024-05-22T23:57:33.316029+0000","last_unstale":"2024-05-23T00:00:32.812167+0000","last_undegraded":"2024-05-23T00:00:32.812167+0000","last_fullsized":"2024-05-23T00:00:32.812167+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:46:34.383953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"93'6","reported_seq":34,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.818610+0000","last_change":"2024-05-22T23:57:33.334625+0000","last_active":"2024-05-23T00:00:32.818610+0000","last_peered":"2024-05-23T00:00:32.818610+0000","last_clean":"2024-05-23T00:00:32.818610+0000","last_became_active":"2024-05-22T23:57:33.334341+0000","last_became_peered":"2024-05-22T23:57:33.334341+0000","last_unstale":"2024-05-23T00:00:32.818610+0000","last_undegraded":"2024-05-23T00:00:32.818610+0000","last_fullsized":"2024-05-23T00:00:32.818610+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T00:40:38.319721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.810428+0000","last_change":"2024-05-22T23:57:33.330958+0000","last_active":"2024-05-23T00:00:32.810428+0000","last_peered":"2024-05-23T00:00:32.810428+0000","last_clean":"2024-05-23T00:00:32.810428+0000","last_became_active":"2024-05-22T23:57:33.317355+0000","last_became_peered":"2024-05-22T23:57:33.317355+0000","last_unstale":"2024-05-23T00:00:32.810428+0000","last_undegraded":"2024-05-23T00:00:32.810428+0000","last_fullsized":"2024-05-23T00:00:32.810428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T10:49:43.784780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"93'10","reported_seq":38,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.021047+0000","last_change":"2024-05-22T23:57:33.319530+0000","last_active":"2024-05-23T00:00:33.021047+0000","last_peered":"2024-05-23T00:00:33.021047+0000","last_clean":"2024-05-23T00:00:33.021047+0000","last_became_active":"2024-05-22T23:57:33.319122+0000","last_became_peered":"2024-05-22T23:57:33.319122+0000","last_unstale":"2024-05-23T00:00:33.021047+0000","last_undegraded":"2024-05-23T00:00:33.021047+0000","last_fullsized":"2024-05-23T00:00:33.021047+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T03:31:26.937804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"93'11","reported_seq":39,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:33.024256+0000","last_change":"2024-05-22T23:57:33.323722+0000","last_active":"2024-05-23T00:00:33.024256+0000","last_peered":"2024-05-23T00:00:33.024256+0000","last_clean":"2024-05-23T00:00:33.024256+0000","last_became_active":"2024-05-22T23:57:33.323343+0000","last_became_peered":"2024-05-22T23:57:33.323343+0000","last_unstale":"2024-05-23T00:00:33.024256+0000","last_undegraded":"2024-05-23T00:00:33.024256+0000","last_fullsized":"2024-05-23T00:00:33.024256+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T06:24:59.575536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"93'8","reported_seq":36,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-05-23T00:00:32.809773+0000","last_change":"2024-05-22T23:57:33.320925+0000","last_active":"2024-05-23T00:00:32.809773+0000","last_peered":"2024-05-23T00:00:32.809773+0000","last_clean":"2024-05-23T00:00:32.809773+0000","last_became_active":"2024-05-22T23:57:33.320570+0000","last_became_peered":"2024-05-22T23:57:33.320570+0000","last_unstale":"2024-05-23T00:00:32.809773+0000","last_undegraded":"2024-05-23T00:00:32.809773+0000","last_fullsized":"2024-05-23T00:00:32.809773+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-22T23:57:32.291401+0000","last_clean_scrub_stamp":"2024-05-22T23:57:32.291401+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-24T11:38:27.659025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":107088,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":599621,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1018,"num_read_kb":622,"num_write":295,"num_write_kb":1448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1012,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1798863,"data_compressed":80985,"data_compressed_allocated":110592,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},"log_size":234,"ondisk_log_size":234,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":82,"seq":352187318321,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1052672,"data_stored":911392,"data_compressed":42724,"data_compressed_allocated":57344,"data_compressed_original":458752,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":74,"seq":317827579959,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1164430,"data_compressed":57992,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":69,"seq":296352743484,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712948,"statfs":{"total":95995035648,"available":95962058752,"internally_reserved":0,"allocated":1048576,"data_stored":522508,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":61,"seq":261993005122,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":518332,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266761,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1105414,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430284,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":503780,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":582541,"data_compressed":25271,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":514976,"data_compressed":23578,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1110016,"data_stored":582489,"data_compressed":25271,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149791,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32232,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712920,"statfs":{"total":95995035648,"available":95962030080,"internally_reserved":0,"allocated":1081344,"data_stored":1038598,"data_compressed":48880,"data_compressed_allocated":65536,"data_compressed_original":581632,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888153,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":510424,"data_compressed":20603,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691925,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32240,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712912,"statfs":{"total":95995035648,"available":95962021888,"internally_reserved":0,"allocated":1089536,"data_stored":1100471,"data_compressed":56299,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35696,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":69043,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69771,"data_compressed":1693,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":3418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134307,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":203142,"data_compressed":7849,"data_compressed_allocated":12288,"data_compressed_original":131072,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":393798,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-23T00:00:37.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:00:37.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='client.25505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: Unable to set osd_memory_target on smithi105 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-05-23T00:00:37.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:37.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='client.25511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:37.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:37 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 -- 172.21.15.103:0/4224870399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde84101710 msgr2=0x7fde84101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 --2- 172.21.15.103:0/4224870399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde84101710 0x7fde84101b30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fde74009860 tx=0x7fde7402a340 comp rx=0 tx=0).stop 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 -- 172.21.15.103:0/4224870399 shutdown_connections 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 --2- 172.21.15.103:0/4224870399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde841032c0 0x7fde84109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 --2- 172.21.15.103:0/4224870399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde84102900 0x7fde84102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 --2- 172.21.15.103:0/4224870399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde84101710 0x7fde84101b30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 -- 172.21.15.103:0/4224870399 >> 172.21.15.103:0/4224870399 conn(0x7fde840fcc80 msgr2=0x7fde840ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 -- 172.21.15.103:0/4224870399 shutdown_connections 2024-05-23T00:00:37.864 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.860+0000 7fde89a00640 1 -- 172.21.15.103:0/4224870399 wait complete. 2024-05-23T00:00:37.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.861+0000 7fde89a00640 1 Processor -- start 2024-05-23T00:00:37.865 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.861+0000 7fde89a00640 1 -- start start 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde89a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde84101710 0x7fde84075620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde89a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde84102900 0x7fde84073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde89a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde841032c0 0x7fde840741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde89a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fde8410d360 con 0x7fde84101710 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde89a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fde8410d1e0 con 0x7fde841032c0 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde89a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fde8410d4e0 con 0x7fde84102900 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde83e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde84101710 0x7fde84075620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde89000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde841032c0 0x7fde840741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde83400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde84102900 0x7fde84073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde83e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde84101710 0x7fde84075620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38704/0 (socket says 172.21.15.103:38704) 2024-05-23T00:00:37.866 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.862+0000 7fde83e00640 1 -- 172.21.15.103:0/3724968782 learned_addr learned my addr 172.21.15.103:0/3724968782 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde89000640 1 -- 172.21.15.103:0/3724968782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde84102900 msgr2=0x7fde84073c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde89000640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde84102900 0x7fde84073c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde89000640 1 -- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde84101710 msgr2=0x7fde84075620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde89000640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde84101710 0x7fde84075620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde89000640 1 -- 172.21.15.103:0/3724968782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde84074a30 con 0x7fde841032c0 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde89000640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde841032c0 0x7fde840741a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fde7c00ef00 tx=0x7fde7c00c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde80c00640 1 -- 172.21.15.103:0/3724968782 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c004b20 con 0x7fde841032c0 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde80c00640 1 -- 172.21.15.103:0/3724968782 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde7c003e50 con 0x7fde841032c0 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde841a1b30 con 0x7fde841032c0 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.863+0000 7fde80c00640 1 -- 172.21.15.103:0/3724968782 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c00b570 con 0x7fde841032c0 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.864+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde841a2080 con 0x7fde841032c0 2024-05-23T00:00:37.868 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.865+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde84108ab0 con 0x7fde841032c0 2024-05-23T00:00:37.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='client.25505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: Unable to set osd_memory_target on smithi105 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-05-23T00:00:37.870 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:00:37.871 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.871 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.871 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='client.25511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:37.871 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:37 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:37.871 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.868+0000 7fde80c00640 1 -- 172.21.15.103:0/3724968782 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7fde7c01d020 con 0x7fde841032c0 2024-05-23T00:00:37.872 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.869+0000 7fde80c00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fde680a2e10 0x7fde680a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:37.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.869+0000 7fde80c00640 1 -- 172.21.15.103:0/3724968782 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fde7c0cf790 con 0x7fde841032c0 2024-05-23T00:00:37.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.869+0000 7fde83e00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fde680a2e10 0x7fde680a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:37.873 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.870+0000 7fde83e00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fde680a2e10 0x7fde680a52d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fde7402a930 tx=0x7fde740058b0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:37.877 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:37.874+0000 7fde80c00640 1 -- 172.21.15.103:0/3724968782 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fde7c09bea0 con 0x7fde841032c0 2024-05-23T00:00:38.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.073+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde8402d450 con 0x7fde841032c0 2024-05-23T00:00:38.079 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.075+0000 7fde80c00640 1 -- 172.21.15.103:0/3724968782 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25041 (secure 0 0 0) 0x7fde7c09b7a0 con 0x7fde841032c0 2024-05-23T00:00:38.079 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:38.079 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":94,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-23T00:00:32.661164+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/284924627":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3929139004":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3827978298":"2024-05-24T00:00:32.661117+0000","172.21.15.103:6801/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/2531882208":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:00:38.080 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:00:37 smithi103 sudo[86839]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:38.080 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:00:37 smithi103 sudo[86839]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:38.080 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:00:37 smithi103 sudo[86839]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:38.080 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:00:37 smithi103 sudo[86839]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:38.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fde680a2e10 msgr2=0x7fde680a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.081 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fde680a2e10 0x7fde680a52d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fde7402a930 tx=0x7fde740058b0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde841032c0 msgr2=0x7fde840741a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde841032c0 0x7fde840741a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fde7c00ef00 tx=0x7fde7c00c740 comp rx=0 tx=0).stop 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 shutdown_connections 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fde680a2e10 0x7fde680a52d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde841032c0 0x7fde840741a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde84102900 0x7fde84073c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 --2- 172.21.15.103:0/3724968782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fde84101710 0x7fde84075620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 >> 172.21.15.103:0/3724968782 conn(0x7fde840fcc80 msgr2=0x7fde84105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 shutdown_connections 2024-05-23T00:00:38.082 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.078+0000 7fde89a00640 1 -- 172.21.15.103:0/3724968782 wait complete. 2024-05-23T00:00:38.099 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-23T00:00:38.100 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 -- 172.21.15.103:0/4037227141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0430102990 msgr2=0x7f0430102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 --2- 172.21.15.103:0/4037227141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0430102990 0x7f0430102e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f041c009860 tx=0x7f041c02a340 comp rx=0 tx=0).stop 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 -- 172.21.15.103:0/4037227141 shutdown_connections 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 --2- 172.21.15.103:0/4037227141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0430103350 0x7f0430109be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 --2- 172.21.15.103:0/4037227141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0430102990 0x7f0430102e10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 --2- 172.21.15.103:0/4037227141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f04301017a0 0x7f0430101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 -- 172.21.15.103:0/4037227141 >> 172.21.15.103:0/4037227141 conn(0x7f04300fcd10 msgr2=0x7f04300ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 -- 172.21.15.103:0/4037227141 shutdown_connections 2024-05-23T00:00:38.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.284+0000 7f0434c00640 1 -- 172.21.15.103:0/4037227141 wait complete. 2024-05-23T00:00:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.285+0000 7f0434c00640 1 Processor -- start 2024-05-23T00:00:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f0434c00640 1 -- start start 2024-05-23T00:00:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f0434c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04301017a0 0x7f0430197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f0434c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0430102990 0x7f0430197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f0434c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0430103350 0x7f043019f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f0434c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f043010d420 con 0x7f0430102990 2024-05-23T00:00:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f0434c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f043010d2a0 con 0x7f04301017a0 2024-05-23T00:00:38.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f0434c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f043010d5a0 con 0x7f0430103350 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f042ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0430102990 0x7f0430197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f042ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0430102990 0x7f0430197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38720/0 (socket says 172.21.15.103:38720) 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f042ea00640 1 -- 172.21.15.103:0/1347366785 learned_addr learned my addr 172.21.15.103:0/1347366785 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.286+0000 7f042f400640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04301017a0 0x7f0430197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.287+0000 7f042fe00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0430103350 0x7f043019f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.287+0000 7f042ea00640 1 -- 172.21.15.103:0/1347366785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0430103350 msgr2=0x7f043019f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.287+0000 7f042ea00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0430103350 0x7f043019f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.287+0000 7f042ea00640 1 -- 172.21.15.103:0/1347366785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04301017a0 msgr2=0x7f0430197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.287+0000 7f042ea00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04301017a0 0x7f0430197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.287+0000 7f042ea00640 1 -- 172.21.15.103:0/1347366785 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f043019f680 con 0x7f0430102990 2024-05-23T00:00:38.291 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.287+0000 7f042f400640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04301017a0 0x7f0430197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:38.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.287+0000 7f042ea00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0430102990 0x7f0430197fa0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f041c005940 tx=0x7f041c033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:38.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.288+0000 7f0427e00640 1 -- 172.21.15.103:0/1347366785 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c03f020 con 0x7f0430102990 2024-05-23T00:00:38.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.288+0000 7f0427e00640 1 -- 172.21.15.103:0/1347366785 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f041c030850 con 0x7f0430102990 2024-05-23T00:00:38.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.288+0000 7f0427e00640 1 -- 172.21.15.103:0/1347366785 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f041c037040 con 0x7f0430102990 2024-05-23T00:00:38.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.288+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f043019f910 con 0x7f0430102990 2024-05-23T00:00:38.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.288+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f043019fdb0 con 0x7f0430102990 2024-05-23T00:00:38.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.289+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0430108a60 con 0x7f0430102990 2024-05-23T00:00:38.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.294+0000 7f0427e00640 1 -- 172.21.15.103:0/1347366785 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f041c042020 con 0x7f0430102990 2024-05-23T00:00:38.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.296+0000 7f0427e00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f04140a2d90 0x7f04140a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:38.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.297+0000 7f0427e00640 1 -- 172.21.15.103:0/1347366785 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f041c0efb30 con 0x7f0430102990 2024-05-23T00:00:38.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.297+0000 7f042f400640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f04140a2d90 0x7f04140a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:38.300 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.297+0000 7f0427e00640 1 -- 172.21.15.103:0/1347366785 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f041c049030 con 0x7f0430102990 2024-05-23T00:00:38.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.298+0000 7f042f400640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f04140a2d90 0x7f04140a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04301027f0 tx=0x7f0420009260 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:38.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.500+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f043002d390 con 0x7f0430102990 2024-05-23T00:00:38.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.501+0000 7f0427e00640 1 -- 172.21.15.103:0/1347366785 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) ==== 93+0+10 (secure 0 0 0) 0x7f041c0bc3b0 con 0x7f0430102990 2024-05-23T00:00:38.504 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f04140a2d90 msgr2=0x7f04140a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f04140a2d90 0x7f04140a5250 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f04301027f0 tx=0x7f0420009260 comp rx=0 tx=0).stop 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0430102990 msgr2=0x7f0430197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0430102990 0x7f0430197fa0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f041c005940 tx=0x7f041c033d90 comp rx=0 tx=0).stop 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 shutdown_connections 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f04140a2d90 0x7f04140a5250 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0430103350 0x7f043019f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0430102990 0x7f0430197fa0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 --2- 172.21.15.103:0/1347366785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04301017a0 0x7f0430197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 >> 172.21.15.103:0/1347366785 conn(0x7f04300fcd10 msgr2=0x7f0430105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:38.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 shutdown_connections 2024-05-23T00:00:38.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.503+0000 7f0434c00640 1 -- 172.21.15.103:0/1347366785 wait complete. 2024-05-23T00:00:38.531 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-23T00:00:38.651 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:00:38 smithi103 sudo[86968]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:38.651 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:00:38 smithi103 sudo[86968]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:38.651 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:00:38 smithi103 sudo[86968]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:38.651 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:00:38 smithi103 sudo[86968]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 -- 172.21.15.103:0/3665756954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4103ad0 msgr2=0x7f2ad4103ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 --2- 172.21.15.103:0/3665756954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4103ad0 0x7f2ad4103ef0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8009900 tx=0x7f2ac802a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 -- 172.21.15.103:0/3665756954 shutdown_connections 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 --2- 172.21.15.103:0/3665756954 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ad4105680 0x7f2ad410bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 --2- 172.21.15.103:0/3665756954 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ad4104cc0 0x7f2ad4105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 --2- 172.21.15.103:0/3665756954 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4103ad0 0x7f2ad4103ef0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 -- 172.21.15.103:0/3665756954 >> 172.21.15.103:0/3665756954 conn(0x7f2ad40ff060 msgr2=0x7f2ad4101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 -- 172.21.15.103:0/3665756954 shutdown_connections 2024-05-23T00:00:38.716 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.712+0000 7f2adb600640 1 -- 172.21.15.103:0/3665756954 wait complete. 2024-05-23T00:00:38.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.713+0000 7f2adb600640 1 Processor -- start 2024-05-23T00:00:38.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2adb600640 1 -- start start 2024-05-23T00:00:38.717 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2adb600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ad4103ad0 0x7f2ad4199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2adb600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4104cc0 0x7f2ad419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2adb600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ad4105680 0x7f2ad41a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2adb600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410f720 con 0x7f2ad4104cc0 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2adb600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410f5a0 con 0x7f2ad4105680 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2adb600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2ad410f8a0 con 0x7f2ad4103ad0 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2ad3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4104cc0 0x7f2ad419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2ad9600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ad4105680 0x7f2ad41a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2ad9600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ad4105680 0x7f2ad41a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49498/0 (socket says 172.21.15.103:49498) 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2ad9600640 1 -- 172.21.15.103:0/2108343118 learned_addr learned my addr 172.21.15.103:0/2108343118 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:38.718 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.714+0000 7f2ad8c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ad4103ad0 0x7f2ad4199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:38.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad9600640 1 -- 172.21.15.103:0/2108343118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ad4103ad0 msgr2=0x7f2ad4199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad9600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ad4103ad0 0x7f2ad4199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad9600640 1 -- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4104cc0 msgr2=0x7f2ad419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.719 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad9600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4104cc0 0x7f2ad419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad9600640 1 -- 172.21.15.103:0/2108343118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad41a19c0 con 0x7f2ad4105680 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad3e00640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4104cc0 0x7f2ad419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad9600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ad4105680 0x7f2ad41a1320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00efd0 tx=0x7f2acc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad1600640 1 -- 172.21.15.103:0/2108343118 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc003b70 con 0x7f2ad4105680 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad1600640 1 -- 172.21.15.103:0/2108343118 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2acc003d10 con 0x7f2ad4105680 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ad41a1cb0 con 0x7f2ad4105680 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2ad1600640 1 -- 172.21.15.103:0/2108343118 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2acc0108d0 con 0x7f2ad4105680 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.715+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ad41a21d0 con 0x7f2ad4105680 2024-05-23T00:00:38.720 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.717+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ad410adf0 con 0x7f2ad4105680 2024-05-23T00:00:38.722 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.719+0000 7f2ad1600640 1 -- 172.21.15.103:0/2108343118 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f2acc004020 con 0x7f2ad4105680 2024-05-23T00:00:38.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.721+0000 7f2ad1600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2abc0a2e10 0x7f2abc0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:38.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.721+0000 7f2ad1600640 1 -- 172.21.15.103:0/2108343118 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2acc0cfdb0 con 0x7f2ad4105680 2024-05-23T00:00:38.724 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.721+0000 7f2ad8c00640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2abc0a2e10 0x7f2abc0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:38.725 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.722+0000 7f2ad8c00640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2abc0a2e10 0x7f2abc0a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8005ab0 tx=0x7f2ac8000bc0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:38.727 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.724+0000 7f2ad1600640 1 -- 172.21.15.103:0/2108343118 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f2acc0d4070 con 0x7f2ad4105680 2024-05-23T00:00:38.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:38 smithi105 ceph-mon[36426]: from='client.25517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:38.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:38 smithi105 ceph-mon[36426]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-23T00:00:38.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:38 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3724968782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:38.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:38 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1347366785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:38.889 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:38 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:38.890 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:38 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:38.890 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:38 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:38.890 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:38 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:38.890 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:00:38 smithi105 sudo[69040]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:38.890 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:00:38 smithi105 sudo[69040]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:38.890 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:00:38 smithi105 sudo[69040]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:38.890 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:00:38 smithi105 sudo[69040]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:38.951 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.948+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2ad402d450 con 0x7f2ad4105680 2024-05-23T00:00:38.952 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.949+0000 7f2ad1600640 1 -- 172.21.15.103:0/2108343118 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) ==== 104+0+11 (secure 0 0 0) 0x7f2acc09c4c0 con 0x7f2ad4105680 2024-05-23T00:00:38.953 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:38.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2abc0a2e10 msgr2=0x7f2abc0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2abc0a2e10 0x7f2abc0a52d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8005ab0 tx=0x7f2ac8000bc0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ad4105680 msgr2=0x7f2ad41a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:38.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ad4105680 0x7f2ad41a1320 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2acc00efd0 tx=0x7f2acc00c6a0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 shutdown_connections 2024-05-23T00:00:38.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2abc0a2e10 0x7f2abc0a52d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2ad4105680 0x7f2ad41a1320 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ad4104cc0 0x7f2ad419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 --2- 172.21.15.103:0/2108343118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2ad4103ad0 0x7f2ad4199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:38.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 >> 172.21.15.103:0/2108343118 conn(0x7f2ad40ff060 msgr2=0x7f2ad4107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:38.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 shutdown_connections 2024-05-23T00:00:38.956 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:38.952+0000 7f2adb600640 1 -- 172.21.15.103:0/2108343118 wait complete. 2024-05-23T00:00:38.972 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:38 smithi103 ceph-mon[31370]: from='client.25517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:38.972 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:38 smithi103 ceph-mon[31370]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-23T00:00:38.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:38 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3724968782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:38.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:38 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1347366785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:38.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:38 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:38.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:38 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:38.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:38 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:38.973 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:38 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:38.975 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-23T00:00:39.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:38 smithi174 ceph-mon[36179]: from='client.25517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-23T00:00:39.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:38 smithi174 ceph-mon[36179]: mgrmap e25: x(active, since 4s), standbys: z, y 2024-05-23T00:00:39.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:38 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3724968782' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:39.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:38 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1347366785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:00:39.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:38 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:39.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:38 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:38 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:39.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:38 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:39.152 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 -- 172.21.15.103:0/96496888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f290c109a20 msgr2=0x7f290c110340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 --2- 172.21.15.103:0/96496888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f290c109a20 0x7f290c110340 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2904009ca0 tx=0x7f290402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 -- 172.21.15.103:0/96496888 shutdown_connections 2024-05-23T00:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 --2- 172.21.15.103:0/96496888 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f290c109a20 0x7f290c110340 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 --2- 172.21.15.103:0/96496888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f290c109060 0x7f290c1094e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 --2- 172.21.15.103:0/96496888 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c107e80 0x7f290c1082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 -- 172.21.15.103:0/96496888 >> 172.21.15.103:0/96496888 conn(0x7f290c076e70 msgr2=0x7f290c0792b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 -- 172.21.15.103:0/96496888 shutdown_connections 2024-05-23T00:00:39.153 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.149+0000 7f2911400640 1 -- 172.21.15.103:0/96496888 wait complete. 2024-05-23T00:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.150+0000 7f2911400640 1 Processor -- start 2024-05-23T00:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.150+0000 7f2911400640 1 -- start start 2024-05-23T00:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.150+0000 7f2911400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f290c107e80 0x7f290c19e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.150+0000 7f2911400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f290c109060 0x7f290c19e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:39.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.150+0000 7f2911400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c109a20 0x7f290c1a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.150+0000 7f2911400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f290c113b80 con 0x7f290c109a20 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.150+0000 7f2911400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f290c113a00 con 0x7f290c109060 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.150+0000 7f2911400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f290c113d00 con 0x7f290c107e80 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f290c107e80 0x7f290c19e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f2909e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f290c109060 0x7f290c19e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f290c107e80 0x7f290c19e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41656/0 (socket says 172.21.15.103:41656) 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 -- 172.21.15.103:0/746487096 learned_addr learned my addr 172.21.15.103:0/746487096 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290b200640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c109a20 0x7f290c1a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 -- 172.21.15.103:0/746487096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f290c109060 msgr2=0x7f290c19e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f290c109060 0x7f290c19e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 -- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c109a20 msgr2=0x7f290c1a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.156 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c109a20 0x7f290c1a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 -- 172.21.15.103:0/746487096 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f290c1a5d80 con 0x7f290c107e80 2024-05-23T00:00:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f290a800640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f290c107e80 0x7f290c19e160 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00e9c0 tx=0x7f28fc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.151+0000 7f2903400640 1 -- 172.21.15.103:0/746487096 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc00cd80 con 0x7f290c107e80 2024-05-23T00:00:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.152+0000 7f2903400640 1 -- 172.21.15.103:0/746487096 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28fc0035f0 con 0x7f290c107e80 2024-05-23T00:00:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.152+0000 7f2903400640 1 -- 172.21.15.103:0/746487096 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28fc010690 con 0x7f290c107e80 2024-05-23T00:00:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.152+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f290c1a6070 con 0x7f290c107e80 2024-05-23T00:00:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.152+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f290c1a6610 con 0x7f290c107e80 2024-05-23T00:00:39.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.153+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f290c10f1c0 con 0x7f290c107e80 2024-05-23T00:00:39.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.155+0000 7f2903400640 1 -- 172.21.15.103:0/746487096 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f28fc0037a0 con 0x7f290c107e80 2024-05-23T00:00:39.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.157+0000 7f2903400640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f28f00a2e10 0x7f28f00a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:39.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.157+0000 7f2903400640 1 -- 172.21.15.103:0/746487096 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f28fc0cfda0 con 0x7f290c107e80 2024-05-23T00:00:39.160 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.157+0000 7f2909e00640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f28f00a2e10 0x7f28f00a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:39.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.158+0000 7f2909e00640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f28f00a2e10 0x7f28f00a52d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f290c19f670 tx=0x7f28f8005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:39.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.160+0000 7f2903400640 1 -- 172.21.15.103:0/746487096 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f28fc09c4b0 con 0x7f290c107e80 2024-05-23T00:00:39.308 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:00:38 smithi105 sudo[69220]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:39.308 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:00:38 smithi105 sudo[69220]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:39.308 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:00:38 smithi105 sudo[69220]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:39.308 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:00:38 smithi105 sudo[69220]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:39.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.367+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f290c0624a0 con 0x7f290c107e80 2024-05-23T00:00:39.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.368+0000 7f2903400640 1 -- 172.21.15.103:0/746487096 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) ==== 100+0+11 (secure 0 0 0) 0x7f28fc09bdb0 con 0x7f290c107e80 2024-05-23T00:00:39.371 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-23T00:00:39.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f28f00a2e10 msgr2=0x7f28f00a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f28f00a2e10 0x7f28f00a52d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f290c19f670 tx=0x7f28f8005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:39.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f290c107e80 msgr2=0x7f290c19e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f290c107e80 0x7f290c19e160 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f28fc00e9c0 tx=0x7f28fc00ee80 comp rx=0 tx=0).stop 2024-05-23T00:00:39.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 shutdown_connections 2024-05-23T00:00:39.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f28f00a2e10 0x7f28f00a52d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f290c109a20 0x7f290c1a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f290c109060 0x7f290c19e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 --2- 172.21.15.103:0/746487096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f290c107e80 0x7f290c19e160 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 >> 172.21.15.103:0/746487096 conn(0x7f290c076e70 msgr2=0x7f290c10c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:39.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 shutdown_connections 2024-05-23T00:00:39.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.372+0000 7f2911400640 1 -- 172.21.15.103:0/746487096 wait complete. 2024-05-23T00:00:39.393 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-23T00:00:39.413 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:00:39 smithi174 sudo[69842]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:39.413 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:00:39 smithi174 sudo[69842]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:39.413 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:00:39 smithi174 sudo[69842]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:39.413 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:00:39 smithi174 sudo[69842]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:39.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 -- 172.21.15.103:0/3350529548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 msgr2=0x7f3b1c102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 --2- 172.21.15.103:0/3350529548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 0x7f3b1c102f10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3b04009900 tx=0x7f3b0402a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 -- 172.21.15.103:0/3350529548 shutdown_connections 2024-05-23T00:00:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 --2- 172.21.15.103:0/3350529548 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b1c103450 0x7f3b1c109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 --2- 172.21.15.103:0/3350529548 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 0x7f3b1c102f10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 --2- 172.21.15.103:0/3350529548 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c1018a0 0x7f3b1c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 -- 172.21.15.103:0/3350529548 >> 172.21.15.103:0/3350529548 conn(0x7f3b1c0fce30 msgr2=0x7f3b1c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 -- 172.21.15.103:0/3350529548 shutdown_connections 2024-05-23T00:00:39.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.587+0000 7f3b21200640 1 -- 172.21.15.103:0/3350529548 wait complete. 2024-05-23T00:00:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b21200640 1 Processor -- start 2024-05-23T00:00:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b21200640 1 -- start start 2024-05-23T00:00:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b21200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b1c1018a0 0x7f3b1c075620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b21200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 0x7f3b1c073c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b21200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c103450 0x7f3b1c0741a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b21200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10d550 con 0x7f3b1c1018a0 2024-05-23T00:00:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b21200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10d3d0 con 0x7f3b1c102a90 2024-05-23T00:00:39.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b21200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10d6d0 con 0x7f3b1c103450 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b13e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 0x7f3b1c073c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b13e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 0x7f3b1c073c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49540/0 (socket says 172.21.15.103:49540) 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b13e00640 1 -- 172.21.15.103:0/2377021989 learned_addr learned my addr 172.21.15.103:0/2377021989 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b19e00640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b1c1018a0 0x7f3b1c075620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.588+0000 7f3b1a800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c103450 0x7f3b1c0741a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b13e00640 1 -- 172.21.15.103:0/2377021989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c103450 msgr2=0x7f3b1c0741a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b13e00640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c103450 0x7f3b1c0741a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b13e00640 1 -- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b1c1018a0 msgr2=0x7f3b1c075620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b13e00640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b1c1018a0 0x7f3b1c075620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b13e00640 1 -- 172.21.15.103:0/2377021989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c0746e0 con 0x7f3b1c102a90 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b13e00640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 0x7f3b1c073c60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3b04031440 tx=0x7f3b04032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:39.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b12a00640 1 -- 172.21.15.103:0/2377021989 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0403f030 con 0x7f3b1c102a90 2024-05-23T00:00:39.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.589+0000 7f3b12a00640 1 -- 172.21.15.103:0/2377021989 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b04003ce0 con 0x7f3b1c102a90 2024-05-23T00:00:39.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.590+0000 7f3b12a00640 1 -- 172.21.15.103:0/2377021989 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b04035070 con 0x7f3b1c102a90 2024-05-23T00:00:39.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.590+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1c074970 con 0x7f3b1c102a90 2024-05-23T00:00:39.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.590+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1c1a6590 con 0x7f3b1c102a90 2024-05-23T00:00:39.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.591+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b1c108bc0 con 0x7f3b1c102a90 2024-05-23T00:00:39.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.594+0000 7f3b12a00640 1 -- 172.21.15.103:0/2377021989 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f3b04042030 con 0x7f3b1c102a90 2024-05-23T00:00:39.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.595+0000 7f3b12a00640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3b000a2e10 0x7f3b000a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:39.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.595+0000 7f3b12a00640 1 -- 172.21.15.103:0/2377021989 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f3b040efda0 con 0x7f3b1c102a90 2024-05-23T00:00:39.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.596+0000 7f3b19e00640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3b000a2e10 0x7f3b000a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:39.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.596+0000 7f3b19e00640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3b000a2e10 0x7f3b000a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0095f0 tx=0x7f3b0c005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:39.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.599+0000 7f3b12a00640 1 -- 172.21.15.103:0/2377021989 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3b040bbe60 con 0x7f3b1c102a90 2024-05-23T00:00:39.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.834+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3b1c02d420 con 0x7f3b1c102a90 2024-05-23T00:00:39.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.835+0000 7f3b12a00640 1 -- 172.21.15.103:0/2377021989 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) ==== 103+0+11 (secure 0 0 0) 0x7f3b040c0160 con 0x7f3b1c102a90 2024-05-23T00:00:39.838 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3b000a2e10 msgr2=0x7f3b000a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3b000a2e10 0x7f3b000a52d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0095f0 tx=0x7f3b0c005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 msgr2=0x7f3b1c073c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 0x7f3b1c073c60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3b04031440 tx=0x7f3b04032ce0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 shutdown_connections 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3b000a2e10 0x7f3b000a52d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b1c103450 0x7f3b1c0741a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b1c102a90 0x7f3b1c073c60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 --2- 172.21.15.103:0/2377021989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b1c1018a0 0x7f3b1c075620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 >> 172.21.15.103:0/2377021989 conn(0x7f3b1c0fce30 msgr2=0x7f3b1c105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 shutdown_connections 2024-05-23T00:00:39.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:39.837+0000 7f3b21200640 1 -- 172.21.15.103:0/2377021989 wait complete. 2024-05-23T00:00:39.860 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:00:39.867 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:00:39 smithi174 sudo[70053]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:39.867 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:00:39 smithi174 sudo[70053]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:39.867 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:00:39 smithi174 sudo[70053]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:39.867 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:00:39 smithi174 sudo[70053]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:39.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi103 to 176.4M 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Unable to set osd_memory_target on smithi103 to 185066291: error parsing value: Value '185066291' is below minimum 939524096 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi174:/etc/ceph/ceph.conf 2024-05-23T00:00:39.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 236 KiB/s wr, 49 op/s 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2108343118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/746487096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:39.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:39 smithi174 ceph-mon[36179]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:39.912 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi103 to 176.4M 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Unable to set osd_memory_target on smithi103 to 185066291: error parsing value: Value '185066291' is below minimum 939524096 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi174:/etc/ceph/ceph.conf 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 236 KiB/s wr, 49 op/s 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:39.913 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:39.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2108343118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:39.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:39.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:39.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:39.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/746487096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:39.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:39.914 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:39 smithi103 ceph-mon[31370]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.026+0000 7f644c800640 1 -- 172.21.15.103:0/3917041561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 msgr2=0x7f644410bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.026+0000 7f644c800640 1 --2- 172.21.15.103:0/3917041561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 0x7f644410bf10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6440009be0 tx=0x7f644002a6c0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.026+0000 7f644c800640 1 -- 172.21.15.103:0/3917041561 shutdown_connections 2024-05-23T00:00:40.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.026+0000 7f644c800640 1 --2- 172.21.15.103:0/3917041561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 0x7f644410bf10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.026+0000 7f644c800640 1 --2- 172.21.15.103:0/3917041561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6444104cc0 0x7f6444105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.026+0000 7f644c800640 1 --2- 172.21.15.103:0/3917041561 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6444103ad0 0x7f6444103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.026+0000 7f644c800640 1 -- 172.21.15.103:0/3917041561 >> 172.21.15.103:0/3917041561 conn(0x7f64440ff060 msgr2=0x7f6444101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.027+0000 7f644c800640 1 -- 172.21.15.103:0/3917041561 shutdown_connections 2024-05-23T00:00:40.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.027+0000 7f644c800640 1 -- 172.21.15.103:0/3917041561 wait complete. 2024-05-23T00:00:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.027+0000 7f644c800640 1 Processor -- start 2024-05-23T00:00:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f644c800640 1 -- start start 2024-05-23T00:00:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f644c800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6444103ad0 0x7f6444199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f644c800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6444104cc0 0x7f644419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f644c800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 0x7f64441a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f644c800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f644410f700 con 0x7f6444105680 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f644c800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f644410f580 con 0x7f6444104cc0 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f644c800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f644410f880 con 0x7f6444103ad0 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f6449e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6444103ad0 0x7f6444199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f6449e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6444103ad0 0x7f6444199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41700/0 (socket says 172.21.15.103:41700) 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f6449e00640 1 -- 172.21.15.103:0/894772836 learned_addr learned my addr 172.21.15.103:0/894772836 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f6449400640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6444104cc0 0x7f644419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.028+0000 7f644a800640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 0x7f64441a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f6449e00640 1 -- 172.21.15.103:0/894772836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6444104cc0 msgr2=0x7f644419a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f6449e00640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6444104cc0 0x7f644419a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f6449e00640 1 -- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 msgr2=0x7f64441a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f6449e00640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 0x7f64441a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f6449e00640 1 -- 172.21.15.103:0/894772836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64441a1990 con 0x7f6444103ad0 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f644a800640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 0x7f64441a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f6449400640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6444104cc0 0x7f644419a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f6449e00640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6444103ad0 0x7f6444199d70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f643800d950 tx=0x7f643800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f643ea00640 1 -- 172.21.15.103:0/894772836 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438017020 con 0x7f6444103ad0 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f643ea00640 1 -- 172.21.15.103:0/894772836 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6438009c30 con 0x7f6444103ad0 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f643ea00640 1 -- 172.21.15.103:0/894772836 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643800f720 con 0x7f6444103ad0 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.029+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64441a1c80 con 0x7f6444103ad0 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.030+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64441a2180 con 0x7f6444103ad0 2024-05-23T00:00:40.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.031+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f644410adf0 con 0x7f6444103ad0 2024-05-23T00:00:40.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.033+0000 7f643ea00640 1 -- 172.21.15.103:0/894772836 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f643801a020 con 0x7f6444103ad0 2024-05-23T00:00:40.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.035+0000 7f643ea00640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f642c0a2e10 0x7f642c0a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.036+0000 7f643ea00640 1 -- 172.21.15.103:0/894772836 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f64380cf870 con 0x7f6444103ad0 2024-05-23T00:00:40.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.036+0000 7f6449400640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f642c0a2e10 0x7f642c0a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.036+0000 7f6449400640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f642c0a2e10 0x7f642c0a52d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f644419b280 tx=0x7f6434005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.039+0000 7f643ea00640 1 -- 172.21.15.103:0/894772836 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6438021020 con 0x7f6444103ad0 2024-05-23T00:00:40.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:40.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:00:40.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi103 to 176.4M 2024-05-23T00:00:40.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Unable to set osd_memory_target on smithi103 to 185066291: error parsing value: Value '185066291' is below minimum 939524096 2024-05-23T00:00:40.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:40.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:40.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-23T00:00:40.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi105:/etc/ceph/ceph.conf 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi174:/etc/ceph/ceph.conf 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 236 KiB/s wr, 49 op/s 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2108343118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.conf 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/746487096' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:40.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:39 smithi105 ceph-mon[36426]: Updating smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:40.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.294+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f644402d200 con 0x7f6444103ad0 2024-05-23T00:00:40.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.295+0000 7f643ea00640 1 -- 172.21.15.103:0/894772836 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11896 (secure 0 0 0) 0x7f6438009de0 con 0x7f6444103ad0 2024-05-23T00:00:40.298 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:40.301 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-23T00:00:31.011025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24559,24563]},"id":1}]} 2024-05-23T00:00:40.302 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 25 2024-05-23T00:00:40.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f642c0a2e10 msgr2=0x7f642c0a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f642c0a2e10 0x7f642c0a52d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f644419b280 tx=0x7f6434005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:40.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6444103ad0 msgr2=0x7f6444199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6444103ad0 0x7f6444199d70 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f643800d950 tx=0x7f643800de10 comp rx=0 tx=0).stop 2024-05-23T00:00:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 shutdown_connections 2024-05-23T00:00:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f642c0a2e10 0x7f642c0a52d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6444105680 0x7f64441a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6444104cc0 0x7f644419a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 --2- 172.21.15.103:0/894772836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6444103ad0 0x7f6444199d70 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 >> 172.21.15.103:0/894772836 conn(0x7f64440ff060 msgr2=0x7f6444107fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 shutdown_connections 2024-05-23T00:00:40.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.297+0000 7f644c800640 1 -- 172.21.15.103:0/894772836 wait complete. 2024-05-23T00:00:40.315 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:00:40 smithi103 sudo[87583]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:40.315 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:00:40 smithi103 sudo[87583]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:40.315 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:00:40 smithi103 sudo[87583]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:40.315 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:00:40 smithi103 sudo[87583]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:40.317 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:00:40.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 -- 172.21.15.103:0/687972841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a20074540 msgr2=0x7f1a2010dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 --2- 172.21.15.103:0/687972841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a20074540 0x7f1a2010dbc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c009900 tx=0x7f1a0c02a480 comp rx=0 tx=0).stop 2024-05-23T00:00:40.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 -- 172.21.15.103:0/687972841 shutdown_connections 2024-05-23T00:00:40.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 --2- 172.21.15.103:0/687972841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a2010e100 0x7f1a201104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 --2- 172.21.15.103:0/687972841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a20074540 0x7f1a2010dbc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 --2- 172.21.15.103:0/687972841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a20073c20 0x7f1a20074000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 -- 172.21.15.103:0/687972841 >> 172.21.15.103:0/687972841 conn(0x7f1a20078c30 msgr2=0x7f1a20079040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:40.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 -- 172.21.15.103:0/687972841 shutdown_connections 2024-05-23T00:00:40.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.488+0000 7f1a26e00640 1 -- 172.21.15.103:0/687972841 wait complete. 2024-05-23T00:00:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.489+0000 7f1a26e00640 1 Processor -- start 2024-05-23T00:00:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.489+0000 7f1a26e00640 1 -- start start 2024-05-23T00:00:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.489+0000 7f1a26e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a20073c20 0x7f1a2019a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.489+0000 7f1a26e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a20074540 0x7f1a2019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.489+0000 7f1a26e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a2010e100 0x7f1a2019ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.489+0000 7f1a26e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a20113e70 con 0x7f1a2010e100 2024-05-23T00:00:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.489+0000 7f1a26e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1a20113cf0 con 0x7f1a20074540 2024-05-23T00:00:40.494 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.489+0000 7f1a26e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a20113ff0 con 0x7f1a20073c20 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a2010e100 0x7f1a2019ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a1f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a20074540 0x7f1a2019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a2010e100 0x7f1a2019ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38802/0 (socket says 172.21.15.103:38802) 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 -- 172.21.15.103:0/3546795665 learned_addr learned my addr 172.21.15.103:0/3546795665 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a1fe00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a20073c20 0x7f1a2019a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 -- 172.21.15.103:0/3546795665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a20073c20 msgr2=0x7f1a2019a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a20073c20 0x7f1a2019a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 -- 172.21.15.103:0/3546795665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a20074540 msgr2=0x7f1a2019a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a20074540 0x7f1a2019a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 -- 172.21.15.103:0/3546795665 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2019f360 con 0x7f1a2010e100 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.490+0000 7f1a24e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a2010e100 0x7f1a2019ec30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800eb10 tx=0x7f1a1800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.491+0000 7f1a1cc00640 1 -- 172.21.15.103:0/3546795665 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18004b20 con 0x7f1a2010e100 2024-05-23T00:00:40.495 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.491+0000 7f1a1cc00640 1 -- 172.21.15.103:0/3546795665 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a18004cc0 con 0x7f1a2010e100 2024-05-23T00:00:40.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.491+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a2019f650 con 0x7f1a2010e100 2024-05-23T00:00:40.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.491+0000 7f1a1cc00640 1 -- 172.21.15.103:0/3546795665 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a18010760 con 0x7f1a2010e100 2024-05-23T00:00:40.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.491+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a201a6ee0 con 0x7f1a2010e100 2024-05-23T00:00:40.496 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.492+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a20068ae0 con 0x7f1a2010e100 2024-05-23T00:00:40.497 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.494+0000 7f1a1cc00640 1 -- 172.21.15.103:0/3546795665 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f1a18005020 con 0x7f1a2010e100 2024-05-23T00:00:40.498 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.495+0000 7f1a1cc00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a040a2d90 0x7f1a040a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.495+0000 7f1a1cc00640 1 -- 172.21.15.103:0/3546795665 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f1a180cfef0 con 0x7f1a2010e100 2024-05-23T00:00:40.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.496+0000 7f1a1fe00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a040a2d90 0x7f1a040a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.499 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.496+0000 7f1a1fe00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a040a2d90 0x7f1a040a5250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009b60 tx=0x7f1a10005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.500+0000 7f1a1cc00640 1 -- 172.21.15.103:0/3546795665 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f1a1809c640 con 0x7f1a2010e100 2024-05-23T00:00:40.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.727+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a20108cf0 con 0x7f1a2010e100 2024-05-23T00:00:40.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.729+0000 7f1a1cc00640 1 -- 172.21.15.103:0/3546795665 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) ==== 74+0+25041 (secure 0 0 0) 0x7f1a1809bf40 con 0x7f1a2010e100 2024-05-23T00:00:40.733 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:00:40.733 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":94,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-23T00:00:32.661164+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/284924627":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3929139004":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3827978298":"2024-05-24T00:00:32.661117+0000","172.21.15.103:6801/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/2531882208":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:00:40.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a040a2d90 msgr2=0x7f1a040a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a040a2d90 0x7f1a040a5250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1a10009b60 tx=0x7f1a10005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a2010e100 msgr2=0x7f1a2019ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a2010e100 0x7f1a2019ec30 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1a1800eb10 tx=0x7f1a1800c2d0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 shutdown_connections 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a040a2d90 0x7f1a040a5250 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a2010e100 0x7f1a2019ec30 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a20074540 0x7f1a2019a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 --2- 172.21.15.103:0/3546795665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a20073c20 0x7f1a2019a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 >> 172.21.15.103:0/3546795665 conn(0x7f1a20078c30 msgr2=0x7f1a200fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 shutdown_connections 2024-05-23T00:00:40.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.732+0000 7f1a26e00640 1 -- 172.21.15.103:0/3546795665 wait complete. 2024-05-23T00:00:40.755 DEBUG:tasks.fwd_scrub:Yielding 2024-05-23T00:00:40.755 INFO:teuthology.run_tasks:Running task exec... 2024-05-23T00:00:40.783 INFO:teuthology.task.exec:Executing custom commands... 2024-05-23T00:00:40.784 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:00:40.784 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/log_level debug' 2024-05-23T00:00:40.785 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-23T00:00:40.785 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-23T00:00:40.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:00:40.785 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-23T00:00:40.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 -- 172.21.15.103:0/1219168574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 msgr2=0x7fdad8109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 --2- 172.21.15.103:0/1219168574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 0x7fdad8109e50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009900 tx=0x7fdacc02a410 comp rx=0 tx=0).stop 2024-05-23T00:00:40.962 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 -- 172.21.15.103:0/1219168574 shutdown_connections 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 --2- 172.21.15.103:0/1219168574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad8104420 0x7fdad8110930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 --2- 172.21.15.103:0/1219168574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad8103a60 0x7fdad8103ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 --2- 172.21.15.103:0/1219168574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 0x7fdad8109e50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 -- 172.21.15.103:0/1219168574 >> 172.21.15.103:0/1219168574 conn(0x7fdad80ff530 msgr2=0x7fdad8101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 -- 172.21.15.103:0/1219168574 shutdown_connections 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 -- 172.21.15.103:0/1219168574 wait complete. 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 Processor -- start 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.957+0000 7fdadfe00640 1 -- start start 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadfe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad8103a60 0x7fdad819a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad8104420 0x7fdad819a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 0x7fdad819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadfe00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdad8113ea0 con 0x7fdad8104420 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadfe00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdad8113d20 con 0x7fdad8109a70 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadfe00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdad8114020 con 0x7fdad8103a60 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadd400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad8103a60 0x7fdad819a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadde00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 0x7fdad819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad8104420 0x7fdad819a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadde00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 0x7fdad819ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49592/0 (socket says 172.21.15.103:49592) 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadd400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad8103a60 0x7fdad819a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41734/0 (socket says 172.21.15.103:41734) 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.958+0000 7fdadde00640 1 -- 172.21.15.103:0/2887205923 learned_addr learned my addr 172.21.15.103:0/2887205923 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdadde00640 1 -- 172.21.15.103:0/2887205923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad8103a60 msgr2=0x7fdad819a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdadde00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad8103a60 0x7fdad819a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdadde00640 1 -- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad8104420 msgr2=0x7fdad819a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdadde00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad8104420 0x7fdad819a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdadde00640 1 -- 172.21.15.103:0/2887205923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad819f300 con 0x7fdad8109a70 2024-05-23T00:00:40.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdadde00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 0x7fdad819ebd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdad000eb10 tx=0x7fdad000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdad6000640 1 -- 172.21.15.103:0/2887205923 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0004b20 con 0x7fdad8109a70 2024-05-23T00:00:40.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdad6000640 1 -- 172.21.15.103:0/2887205923 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdad0004cc0 con 0x7fdad8109a70 2024-05-23T00:00:40.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdad6000640 1 -- 172.21.15.103:0/2887205923 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdad0010760 con 0x7fdad8109a70 2024-05-23T00:00:40.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad819f5f0 con 0x7fdad8109a70 2024-05-23T00:00:40.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.959+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad81a6e80 con 0x7fdad8109a70 2024-05-23T00:00:40.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.960+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad804f630 con 0x7fdad8109a70 2024-05-23T00:00:40.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.962+0000 7fdad6000640 1 -- 172.21.15.103:0/2887205923 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7fdad0005020 con 0x7fdad8109a70 2024-05-23T00:00:40.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.963+0000 7fdad6000640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdac00a2e40 0x7fdac00a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.964+0000 7fdad6000640 1 -- 172.21.15.103:0/2887205923 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fdad00cfe80 con 0x7fdad8109a70 2024-05-23T00:00:40.967 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.964+0000 7fdadd400640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdac00a2e40 0x7fdac00a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.964+0000 7fdadd400640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdac00a2e40 0x7fdac00a5300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdacc000c00 tx=0x7fdacc005b50 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.967+0000 7fdad6000640 1 -- 172.21.15.103:0/2887205923 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fdad009c590 con 0x7fdad8109a70 2024-05-23T00:00:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.974+0000 7f510da00640 1 -- 172.21.15.103:0/4104322851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51081078a0 msgr2=0x7f5108107c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.974+0000 7f510da00640 1 --2- 172.21.15.103:0/4104322851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51081078a0 0x7f5108107c80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f50f8009900 tx=0x7f50f802a410 comp rx=0 tx=0).stop 2024-05-23T00:00:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.975+0000 7f510da00640 1 -- 172.21.15.103:0/4104322851 shutdown_connections 2024-05-23T00:00:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.975+0000 7f510da00640 1 --2- 172.21.15.103:0/4104322851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5108102250 0x7f510810e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.975+0000 7f510da00640 1 --2- 172.21.15.103:0/4104322851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108101890 0x7f5108101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.975+0000 7f510da00640 1 --2- 172.21.15.103:0/4104322851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51081078a0 0x7f5108107c80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.978 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.975+0000 7f510da00640 1 -- 172.21.15.103:0/4104322851 >> 172.21.15.103:0/4104322851 conn(0x7f51080fd380 msgr2=0x7f51080ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:40.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.975+0000 7f510da00640 1 -- 172.21.15.103:0/4104322851 shutdown_connections 2024-05-23T00:00:40.979 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.975+0000 7f510da00640 1 -- 172.21.15.103:0/4104322851 wait complete. 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.976+0000 7f510da00640 1 Processor -- start 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.976+0000 7f510da00640 1 -- start start 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.976+0000 7f510da00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5108101890 0x7f510819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.976+0000 7f510da00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108102250 0x7f510819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.976+0000 7f510da00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51081078a0 0x7f510819ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.976+0000 7f510da00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5108111d00 con 0x7f5108101890 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.976+0000 7f510da00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f5108111b80 con 0x7f51081078a0 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.976+0000 7f510da00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5108111e80 con 0x7f5108102250 2024-05-23T00:00:40.980 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5108101890 0x7f510819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108102250 0x7f510819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5107400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51081078a0 0x7f510819ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5108101890 0x7f510819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38822/0 (socket says 172.21.15.103:38822) 2024-05-23T00:00:40.981 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106a00640 1 -- 172.21.15.103:0/4199353578 learned_addr learned my addr 172.21.15.103:0/4199353578 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106000640 1 -- 172.21.15.103:0/4199353578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51081078a0 msgr2=0x7f510819ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106000640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51081078a0 0x7f510819ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106000640 1 -- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5108101890 msgr2=0x7f510819a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106000640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5108101890 0x7f510819a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106000640 1 -- 172.21.15.103:0/4199353578 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510819f430 con 0x7f5108102250 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.977+0000 7f5106000640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108102250 0x7f510819a980 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50f400e9f0 tx=0x7f50f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.978+0000 7f50ff400640 1 -- 172.21.15.103:0/4199353578 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f400ccc0 con 0x7f5108102250 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.978+0000 7f50ff400640 1 -- 172.21.15.103:0/4199353578 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f4005020 con 0x7f5108102250 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.978+0000 7f510da00640 1 -- 172.21.15.103:0/4199353578 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510819f720 con 0x7f5108102250 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.978+0000 7f50ff400640 1 -- 172.21.15.103:0/4199353578 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f40049e0 con 0x7f5108102250 2024-05-23T00:00:40.983 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.978+0000 7f510da00640 1 -- 172.21.15.103:0/4199353578 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51081086f0 con 0x7f5108102250 2024-05-23T00:00:40.984 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.981+0000 7f51107f0740 1 Processor -- start 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.981+0000 7f51107f0740 1 -- start start 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f51107f0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207550 0x564826207930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f51107f0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826212000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f51107f0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826212540 0x564826214930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f51107f0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56482616b7c0 con 0x564826207e70 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f51107f0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564826113270 con 0x564826212540 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f51107f0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564826167780 con 0x564826207550 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f50ff400640 1 -- 172.21.15.103:0/4199353578 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f50f4005560 con 0x7f5108102250 2024-05-23T00:00:40.986 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5107400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207550 0x564826207930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5106a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826212000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5107400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207550 0x564826207930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41758/0 (socket says 172.21.15.103:41758) 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5107400640 1 -- 172.21.15.103:0/3253076185 learned_addr learned my addr 172.21.15.103:0/3253076185 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5106000640 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826212540 0x564826214930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5107400640 1 -- 172.21.15.103:0/3253076185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826212540 msgr2=0x564826214930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5107400640 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826212540 0x564826214930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5107400640 1 -- 172.21.15.103:0/3253076185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 msgr2=0x564826212000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.982+0000 7f5107400640 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826212000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f5107400640 1 -- 172.21.15.103:0/3253076185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564825f477e0 con 0x564826207550 2024-05-23T00:00:40.987 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f5106000640 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826212540 0x564826214930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:40.988 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f5106a00640 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826212000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:40.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f5107400640 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207550 0x564826207930 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5100009aa0 tx=0x7f5100009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d79909b7a23964d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.995 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f50f3e00640 1 -- 172.21.15.103:0/3253076185 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5100004b20 con 0x564826207550 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f50ff400640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f50ec0a2d90 0x7f50ec0a5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f50f3e00640 1 -- 172.21.15.103:0/3253076185 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5100004cc0 con 0x564826207550 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f50f3e00640 1 -- 172.21.15.103:0/3253076185 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5100004250 con 0x564826207550 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.983+0000 7f50ff400640 1 -- 172.21.15.103:0/4199353578 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f50f40cf900 con 0x7f5108102250 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f5106a00640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f50ec0a2d90 0x7f50ec0a5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 -- 172.21.15.103:0/3253076185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207550 msgr2=0x564826207930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207550 0x564826207930 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5100009aa0 tx=0x7f5100009500 comp rx=0 tx=0).stop 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f5106a00640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f50ec0a2d90 0x7f50ec0a5250 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x564826208b10 tx=0x7f50f80046c0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:40.996 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 -- 172.21.15.103:0/3253076185 shutdown_connections 2024-05-23T00:00:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826212540 0x564826214930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826212000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 --2- 172.21.15.103:0/3253076185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207550 0x564826207930 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 -- 172.21.15.103:0/3253076185 >> 172.21.15.103:0/3253076185 conn(0x564826203870 msgr2=0x564826203c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 -- 172.21.15.103:0/3253076185 shutdown_connections 2024-05-23T00:00:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 -- 172.21.15.103:0/3253076185 wait complete. 2024-05-23T00:00:40.997 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 -- start start 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.985+0000 7f51107f0740 1 Processor -- start 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f51107f0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482620f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f51107f0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x56482620fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f51107f0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 0x56482620ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f51107f0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564826215330 con 0x564826212540 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f51107f0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564826126620 con 0x564826207550 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f51107f0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5648262154b0 con 0x564826207e70 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f51107f0740 10 client.0 _ll_register_callbacks cb 0x5648261148b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f5106a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482620f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.998 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f5106a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482620f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49612/0 (socket says 172.21.15.103:49612) 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f5106a00640 1 -- 172.21.15.103:0/1342388927 learned_addr learned my addr 172.21.15.103:0/1342388927 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f5106000640 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x56482620fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.986+0000 7f5107400640 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 0x56482620ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f5106000640 1 -- 172.21.15.103:0/1342388927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 msgr2=0x56482620f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f5106000640 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482620f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f5106000640 1 -- 172.21.15.103:0/1342388927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 msgr2=0x56482620ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f5106000640 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 0x56482620ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f5106000640 1 -- 172.21.15.103:0/1342388927 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564825fa7f70 con 0x564826207e70 2024-05-23T00:00:40.999 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f5106a00640 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482620f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:00:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f5106000640 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x56482620fa10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f50f40ccbf0 tx=0x7f50f409b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=344c5aa4307519ff server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f50f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:00:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f50f0c00640 1 -- 172.21.15.103:0/1342388927 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f401c070 con 0x564826207e70 2024-05-23T00:00:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f51107f0740 1 -- 172.21.15.103:0/1342388927 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5100004cc0 con 0x564826207e70 2024-05-23T00:00:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.987+0000 7f50f0c00640 1 -- 172.21.15.103:0/1342388927 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f50f40ccea0 con 0x564826207e70 2024-05-23T00:00:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f50f0c00640 1 -- 172.21.15.103:0/1342388927 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f50f40051d0 con 0x564826207e70 2024-05-23T00:00:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f51107f0740 10 client.25585 fetch_fsmap learned FSMap version 25 2024-05-23T00:00:41.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f51107f0740 1 -- 172.21.15.103:0/1342388927 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564825e3f170 con 0x564826207e70 2024-05-23T00:00:41.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f50f0c00640 1 -- 172.21.15.103:0/1342388927 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5100004cc0 con 0x564826207e70 2024-05-23T00:00:41.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f50f0c00640 1 -- 172.21.15.103:0/1342388927 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f50f4086070 con 0x564826207e70 2024-05-23T00:00:41.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f51107f0740 10 client.25585 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:00:41.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f51107f0740 10 client.25585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:00:41.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f51107f0740 20 client.25585 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:41.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f51107f0740 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5648262228c0 0x564826224cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f51107f0740 4 client.25585 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:00:41.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.988+0000 7f51107f0740 1 -- 172.21.15.103:0/1342388927 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56482617f1e0 con 0x5648262228c0 2024-05-23T00:00:41.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.989+0000 7f5106a00640 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5648262228c0 0x564826224cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.989+0000 7f50f0c00640 10 client.25585 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:41.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.989+0000 7f5106a00640 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5648262228c0 0x564826224cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=744f7715c8ea6de9 server_cookie=c2b3f5d85ed5f0a4 in_seq=0 out_seq=0 2024-05-23T00:00:41.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.990+0000 7f50f0c00640 1 -- 172.21.15.103:0/1342388927 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56482617f1e0 con 0x5648262228c0 2024-05-23T00:00:41.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.990+0000 7f50f0c00640 10 client.25585 handle_command_reply: tid=0 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.990+0000 7f51107f0740 1 client.25585 shutdown 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 -- 172.21.15.103:0/1342388927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 msgr2=0x56482620fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x56482620fa10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f50f40ccbf0 tx=0x7f50f409b7a0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 -- 172.21.15.103:0/1342388927 shutdown_connections 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5648262228c0 0x564826224cb0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 0x56482620ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x56482620fa10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 --2- 172.21.15.103:0/1342388927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482620f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f50f0c00640 0 client.25585 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 -- 172.21.15.103:0/1342388927 >> 172.21.15.103:0/1342388927 conn(0x564826203870 msgr2=0x564826209670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 -- 172.21.15.103:0/1342388927 shutdown_connections 2024-05-23T00:00:41.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 1 -- 172.21.15.103:0/1342388927 wait complete. 2024-05-23T00:00:41.005 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:40.991+0000 7f51107f0740 20 client.25585 trim_cache size 0 max 16384 2024-05-23T00:00:41.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.010+0000 7f51107f0740 1 Processor -- start 2024-05-23T00:00:41.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.010+0000 7f51107f0740 1 -- start start 2024-05-23T00:00:41.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.010+0000 7f51107f0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.010+0000 7f51107f0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x564826209cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.010+0000 7f51107f0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 0x56482622a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.010+0000 7f51107f0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51081af260 con 0x564826212540 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.010+0000 7f51107f0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f51081acb30 con 0x564826207550 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.010+0000 7f51107f0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51081aa9c0 con 0x564826207e70 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49618/0 (socket says 172.21.15.103:49618) 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 -- 172.21.15.103:0/2672547731 learned_addr learned my addr 172.21.15.103:0/2672547731 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5107400640 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 0x56482622a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106000640 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x564826209cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.016 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 -- 172.21.15.103:0/2672547731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 msgr2=0x564826209cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x564826209cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 -- 172.21.15.103:0/2672547731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 msgr2=0x56482622a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 0x56482622a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 -- 172.21.15.103:0/2672547731 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564826221510 con 0x564826207550 2024-05-23T00:00:41.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f5106a00640 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621e1f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f50f802aa00 tx=0x7f50f803f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db8a84b157f4a17e server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.011+0000 7f50f3400640 1 -- 172.21.15.103:0/2672547731 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8035040 con 0x564826207550 2024-05-23T00:00:41.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f50f3400640 1 -- 172.21.15.103:0/2672547731 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f8058070 con 0x564826207550 2024-05-23T00:00:41.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f50f3400640 1 -- 172.21.15.103:0/2672547731 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8031df0 con 0x564826207550 2024-05-23T00:00:41.021 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f51107f0740 1 -- 172.21.15.103:0/2672547731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 msgr2=0x56482621e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f51107f0740 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621e1f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f50f802aa00 tx=0x7f50f803f000 comp rx=0 tx=0).stop 2024-05-23T00:00:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f51107f0740 1 -- 172.21.15.103:0/2672547731 shutdown_connections 2024-05-23T00:00:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f51107f0740 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826212540 0x56482622a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f51107f0740 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826207e70 0x564826209cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f51107f0740 1 --2- 172.21.15.103:0/2672547731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621e1f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.012+0000 7f51107f0740 1 -- 172.21.15.103:0/2672547731 >> 172.21.15.103:0/2672547731 conn(0x564826203870 msgr2=0x5648262029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.013+0000 7f51107f0740 1 -- 172.21.15.103:0/2672547731 shutdown_connections 2024-05-23T00:00:41.022 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.013+0000 7f51107f0740 1 -- 172.21.15.103:0/2672547731 wait complete. 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.013+0000 7f51107f0740 1 -- start start 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.013+0000 7f51107f0740 1 Processor -- start 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f51107f0740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f51107f0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826216b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f51107f0740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826212540 0x564826217090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f51107f0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51081ab010 con 0x564826207e70 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f51107f0740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f51081ac460 con 0x564826207550 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f51107f0740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51081ab6c0 con 0x564826212540 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f51107f0740 10 client.0 _ll_register_callbacks cb 0x7f50f40d5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f5106a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f5106a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49632/0 (socket says 172.21.15.103:49632) 2024-05-23T00:00:41.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f5106a00640 1 -- 172.21.15.103:0/1085601524 learned_addr learned my addr 172.21.15.103:0/1085601524 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f5106000640 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826216b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.014+0000 7f5107400640 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826212540 0x564826217090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f5106a00640 1 -- 172.21.15.103:0/1085601524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826212540 msgr2=0x564826217090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f5106a00640 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826212540 0x564826217090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f5106a00640 1 -- 172.21.15.103:0/1085601524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 msgr2=0x564826216b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f5106a00640 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826216b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f5106a00640 1 -- 172.21.15.103:0/1085601524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648262175d0 con 0x564826207550 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f5106000640 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826216b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:41.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f5106a00640 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621a7d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f50f8031650 tx=0x7f50f8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b9426560872dddd server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f50f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:00:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f50f0c00640 1 -- 172.21.15.103:0/1085601524 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8061020 con 0x564826207550 2024-05-23T00:00:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.015+0000 7f50f0c00640 1 -- 172.21.15.103:0/1085601524 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f50f8031a70 con 0x564826207550 2024-05-23T00:00:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 1 -- 172.21.15.103:0/1085601524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50f8058070 con 0x564826207550 2024-05-23T00:00:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f50f0c00640 1 -- 172.21.15.103:0/1085601524 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f50f806c050 con 0x564826207550 2024-05-23T00:00:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 10 client.25583 fetch_fsmap learned FSMap version 25 2024-05-23T00:00:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 1 -- 172.21.15.103:0/1085601524 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5648262899a0 con 0x564826207550 2024-05-23T00:00:41.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f50f0c00640 1 -- 172.21.15.103:0/1085601524 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f50f8058070 con 0x564826207550 2024-05-23T00:00:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f50f0c00640 1 -- 172.21.15.103:0/1085601524 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f50f803d5e0 con 0x564826207550 2024-05-23T00:00:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 10 client.25583 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:00:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 10 client.25583 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:00:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 20 client.25583 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56482628a0e0 0x56482628c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 4 client.25583 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-23T00:00:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.016+0000 7f51107f0740 1 -- 172.21.15.103:0/1085601524 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564826178ef0 con 0x56482628a0e0 2024-05-23T00:00:41.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.017+0000 7f5106000640 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56482628a0e0 0x56482628c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.017+0000 7f50f0c00640 10 client.25583 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.017+0000 7f5106000640 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56482628a0e0 0x56482628c500 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc3f44f5ca969f65 server_cookie=27d540f023e04188 in_seq=0 out_seq=0 2024-05-23T00:00:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.018+0000 7f50f0c00640 1 -- 172.21.15.103:0/1085601524 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564826178ef0 con 0x56482628a0e0 2024-05-23T00:00:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.018+0000 7f50f0c00640 10 client.25583 handle_command_reply: tid=0 2024-05-23T00:00:41.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.018+0000 7f51107f0740 1 client.25583 shutdown 2024-05-23T00:00:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 -- 172.21.15.103:0/1085601524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 msgr2=0x56482621a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621a7d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f50f8031650 tx=0x7f50f8005fd0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.028 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 -- 172.21.15.103:0/1085601524 shutdown_connections 2024-05-23T00:00:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56482628a0e0 0x56482628c500 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564826212540 0x564826217090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564826207e70 0x564826216b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 --2- 172.21.15.103:0/1085601524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564826207550 0x56482621a7d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 -- 172.21.15.103:0/1085601524 >> 172.21.15.103:0/1085601524 conn(0x564826203870 msgr2=0x564826210d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f50f0c00640 0 client.25583 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 -- 172.21.15.103:0/1085601524 shutdown_connections 2024-05-23T00:00:41.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 1 -- 172.21.15.103:0/1085601524 wait complete. 2024-05-23T00:00:41.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.019+0000 7f51107f0740 20 client.25583 trim_cache size 0 max 16384 2024-05-23T00:00:41.030 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:00:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: "return_code": 0, 2024-05-23T00:00:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_tag": "2024c2e0-4890-42df-91e6-999eaab35664", 2024-05-23T00:00:41.030 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": "asynchronous" 2024-05-23T00:00:41.030 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:00:41.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 -- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f50ec0a2d90 msgr2=0x7f50ec0a5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f50ec0a2d90 0x7f50ec0a5250 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x564826208b10 tx=0x7f50f80046c0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 -- 172.21.15.103:0/4199353578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108102250 msgr2=0x7f510819a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108102250 0x7f510819a980 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f50f400e9f0 tx=0x7f50f400eeb0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 -- 172.21.15.103:0/4199353578 shutdown_connections 2024-05-23T00:00:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f50ec0a2d90 0x7f50ec0a5250 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f51081078a0 0x7f510819ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5108102250 0x7f510819a980 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 --2- 172.21.15.103:0/4199353578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5108101890 0x7f510819a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 -- 172.21.15.103:0/4199353578 >> 172.21.15.103:0/4199353578 conn(0x7f51080fd380 msgr2=0x7f51080fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 -- 172.21.15.103:0/4199353578 shutdown_connections 2024-05-23T00:00:41.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.021+0000 7f510da00640 1 -- 172.21.15.103:0/4199353578 wait complete. 2024-05-23T00:00:41.037 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:00:40 smithi103 sudo[87869]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:41.038 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:00:40 smithi103 sudo[87869]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:41.038 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:00:40 smithi103 sudo[87869]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:41.038 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:00:40 smithi103 sudo[87869]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:41.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2377021989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:41.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:41.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/894772836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:41.039 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:41.040 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:40 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3546795665' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:41.041 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-23T00:00:41.041 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-23T00:00:41.058 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:00:40 smithi105 sudo[69878]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:41.058 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:00:40 smithi105 sudo[69878]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:41.058 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:00:40 smithi105 sudo[69878]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:41.058 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:00:40 smithi105 sudo[69878]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2377021989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/894772836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:41.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:40 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3546795665' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:41.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:41.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2377021989' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:00:41.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-05-23T00:00:41.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/894772836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:00:41.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: Updating smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/config/ceph.client.admin.keyring 2024-05-23T00:00:41.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:40 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3546795665' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:00:41.191 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.189+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/log_level}] v 0) -- 0x7fdad810ba40 con 0x7fdad8109a70 2024-05-23T00:00:41.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.195+0000 7fdad6000640 1 -- 172.21.15.103:0/2887205923 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/log_level}]=0 v28)=0 v28) ==== 129+0+0 (secure 0 0 0) 0x7fdad009be90 con 0x7fdad8109a70 2024-05-23T00:00:41.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdac00a2e40 msgr2=0x7fdac00a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdac00a2e40 0x7fdac00a5300 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdacc000c00 tx=0x7fdacc005b50 comp rx=0 tx=0).stop 2024-05-23T00:00:41.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 msgr2=0x7fdad819ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 0x7fdad819ebd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdad000eb10 tx=0x7fdad000c2d0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 shutdown_connections 2024-05-23T00:00:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdac00a2e40 0x7fdac00a5300 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdad8109a70 0x7fdad819ebd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdad8104420 0x7fdad819a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 --2- 172.21.15.103:0/2887205923 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdad8103a60 0x7fdad819a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.199+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 >> 172.21.15.103:0/2887205923 conn(0x7fdad80ff530 msgr2=0x7fdad80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.200+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 shutdown_connections 2024-05-23T00:00:41.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.200+0000 7fdadfe00640 1 -- 172.21.15.103:0/2887205923 wait complete. 2024-05-23T00:00:41.222 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-05-23T00:00:41.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 -- 172.21.15.103:0/3132969372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe01043f0 msgr2=0x7f8fe0110900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.229 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 --2- 172.21.15.103:0/3132969372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe01043f0 0x7f8fe0110900 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8009cd0 tx=0x7f8fd802a8f0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 -- 172.21.15.103:0/3132969372 shutdown_connections 2024-05-23T00:00:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 --2- 172.21.15.103:0/3132969372 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe01043f0 0x7f8fe0110900 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 --2- 172.21.15.103:0/3132969372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fe0103a30 0x7f8fe0103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 --2- 172.21.15.103:0/3132969372 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fe0109a40 0x7f8fe0109e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 -- 172.21.15.103:0/3132969372 >> 172.21.15.103:0/3132969372 conn(0x7f8fe00ff520 msgr2=0x7f8fe0101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 -- 172.21.15.103:0/3132969372 shutdown_connections 2024-05-23T00:00:41.230 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.227+0000 7f8fe7800640 1 -- 172.21.15.103:0/3132969372 wait complete. 2024-05-23T00:00:41.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.228+0000 7f8fe7800640 1 Processor -- start 2024-05-23T00:00:41.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe7800640 1 -- start start 2024-05-23T00:00:41.231 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe7800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fe0103a30 0x7f8fe019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe7800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fe01043f0 0x7f8fe019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe7800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0109a40 0x7f8fe019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe7800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0113ea0 con 0x7f8fe0103a30 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe7800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0113d20 con 0x7f8fe01043f0 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe7800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe0114020 con 0x7f8fe0109a40 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe5800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0109a40 0x7f8fe019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fdfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fe01043f0 0x7f8fe019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe5800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0109a40 0x7f8fe019eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41804/0 (socket says 172.21.15.103:41804) 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe5800640 1 -- 172.21.15.103:0/185348808 learned_addr learned my addr 172.21.15.103:0/185348808 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.232 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.229+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fe0103a30 0x7f8fe019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdfe00640 1 -- 172.21.15.103:0/185348808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0109a40 msgr2=0x7f8fe019eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0109a40 0x7f8fe019eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdfe00640 1 -- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fe0103a30 msgr2=0x7f8fe019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fe0103a30 0x7f8fe019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdfe00640 1 -- 172.21.15.103:0/185348808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe019f2d0 con 0x7f8fe01043f0 2024-05-23T00:00:41.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fe01043f0 0x7f8fe019a820 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00e9f0 tx=0x7f8fcc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdd600640 1 -- 172.21.15.103:0/185348808 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc00ccc0 con 0x7f8fe01043f0 2024-05-23T00:00:41.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdd600640 1 -- 172.21.15.103:0/185348808 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fcc005020 con 0x7f8fe01043f0 2024-05-23T00:00:41.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fdd600640 1 -- 172.21.15.103:0/185348808 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc0049e0 con 0x7f8fe01043f0 2024-05-23T00:00:41.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.230+0000 7f8fe7800640 1 -- 172.21.15.103:0/185348808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fe019f5c0 con 0x7f8fe01043f0 2024-05-23T00:00:41.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.231+0000 7f8fe7800640 1 -- 172.21.15.103:0/185348808 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe01a6e50 con 0x7f8fe01043f0 2024-05-23T00:00:41.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.235+0000 7f8fea7b4740 1 Processor -- start 2024-05-23T00:00:41.237 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.235+0000 7f8fea7b4740 1 -- start start 2024-05-23T00:00:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.235+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ec99a0 0x556ac8ec9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.235+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8eca2c0 0x556ac8ed4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 0x556ac8ed6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556ac8e8d320 con 0x556ac8ed4990 2024-05-23T00:00:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556ac8e2ecb0 con 0x556ac8ec99a0 2024-05-23T00:00:41.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556ac8dd5fc0 con 0x556ac8eca2c0 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fdd600640 1 -- 172.21.15.103:0/185348808 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f8fcc005560 con 0x7f8fe01043f0 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe4e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8eca2c0 0x556ac8ed4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe4e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8eca2c0 0x556ac8ed4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41808/0 (socket says 172.21.15.103:41808) 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe4e00640 1 -- 172.21.15.103:0/4052834264 learned_addr learned my addr 172.21.15.103:0/4052834264 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 0x556ac8ed6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe5800640 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ec99a0 0x556ac8ec9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe4e00640 1 -- 172.21.15.103:0/4052834264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ec99a0 msgr2=0x556ac8ec9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ec99a0 0x556ac8ec9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe4e00640 1 -- 172.21.15.103:0/4052834264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 msgr2=0x556ac8ed6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 0x556ac8ed6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe4e00640 1 -- 172.21.15.103:0/4052834264 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac8c0c7d0 con 0x556ac8eca2c0 2024-05-23T00:00:41.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.236+0000 7f8fe5800640 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ec99a0 0x556ac8ec9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:00:41.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.237+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8eca2c0 0x556ac8ed4450 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009c70 tx=0x7f8fd4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a054d12abf8fe76f server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.237+0000 7f8fd2000640 1 -- 172.21.15.103:0/4052834264 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4004230 con 0x556ac8eca2c0 2024-05-23T00:00:41.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.237+0000 7f8fd2000640 1 -- 172.21.15.103:0/4052834264 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fd40043d0 con 0x556ac8eca2c0 2024-05-23T00:00:41.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.237+0000 7f8fd2000640 1 -- 172.21.15.103:0/4052834264 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4019820 con 0x556ac8eca2c0 2024-05-23T00:00:41.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 -- 172.21.15.103:0/4052834264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8eca2c0 msgr2=0x556ac8ed4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8eca2c0 0x556ac8ed4450 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009c70 tx=0x7f8fd4011810 comp rx=0 tx=0).stop 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fdd600640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8fc80a2e10 0x7f8fc80a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fdd600640 1 -- 172.21.15.103:0/185348808 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8fcc0cf840 con 0x7f8fe01043f0 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 -- 172.21.15.103:0/4052834264 shutdown_connections 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 0x556ac8ed6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8eca2c0 0x556ac8ed4450 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/4052834264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ec99a0 0x556ac8ec9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 -- 172.21.15.103:0/4052834264 >> 172.21.15.103:0/4052834264 conn(0x556ac8ec5cc0 msgr2=0x556ac8ec60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fe5800640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8fc80a2e10 0x7f8fc80a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 -- 172.21.15.103:0/4052834264 shutdown_connections 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.238+0000 7f8fea7b4740 1 -- 172.21.15.103:0/4052834264 wait complete. 2024-05-23T00:00:41.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.239+0000 7f8fea7b4740 1 -- start start 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.239+0000 7f8fe5800640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8fc80a2e10 0x7f8fc80a52d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8009a60 tx=0x7f8fd8005d80 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.239+0000 7f8fea7b4740 1 Processor -- start 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.239+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ed1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.239+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 0x556ac8ed1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 0x556ac8ed2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556ac8de9410 con 0x556ac8eca2c0 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556ac8e2ac70 con 0x556ac8ed4990 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556ac8ed7780 con 0x556ac8ec99a0 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fea7b4740 10 client.0 _ll_register_callbacks cb 0x556ac8e250d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ed1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ed1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41812/0 (socket says 172.21.15.103:41812) 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 -- 172.21.15.103:0/1162071714 learned_addr learned my addr 172.21.15.103:0/1162071714 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe5800640 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 0x556ac8ed2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 0x556ac8ed1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 -- 172.21.15.103:0/1162071714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 msgr2=0x556ac8ed2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 0x556ac8ed2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 -- 172.21.15.103:0/1162071714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 msgr2=0x556ac8ed1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 0x556ac8ed1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 -- 172.21.15.103:0/1162071714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac8b423f0 con 0x556ac8ec99a0 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ed1910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009560 tx=0x7f8fd4009ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c97291a1dbad024 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.240+0000 7f8fc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.241+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1162071714 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4027070 con 0x556ac8ec99a0 2024-05-23T00:00:41.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.241+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1162071714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd40043d0 con 0x556ac8ec99a0 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.241+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1162071714 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8fd4003800 con 0x556ac8ec99a0 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.241+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1162071714 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8fd4033050 con 0x556ac8ec99a0 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.241+0000 7f8fea7b4740 10 client.25603 fetch_fsmap learned FSMap version 25 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.241+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1162071714 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556ac8b020f0 con 0x556ac8ec99a0 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.241+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1162071714 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fd40043d0 con 0x556ac8ec99a0 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.242+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1162071714 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8fd4018070 con 0x556ac8ec99a0 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.242+0000 7f8fea7b4740 10 client.25603 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.242+0000 7f8fea7b4740 10 client.25603 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.242+0000 7f8fea7b4740 20 client.25603 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.242+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556ac8ee4a20 0x556ac8ee6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.242+0000 7f8fea7b4740 4 client.25603 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.242+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1162071714 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ac8dede00 con 0x556ac8ee4a20 2024-05-23T00:00:41.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.242+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556ac8ee4a20 0x556ac8ee6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.243+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556ac8ee4a20 0x556ac8ee6e10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40df736898a71ed9 server_cookie=f5c611a3664b9200 in_seq=0 out_seq=0 2024-05-23T00:00:41.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.243+0000 7f8fc6a00640 10 client.25603 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.243+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1162071714 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ac8dede00 con 0x556ac8ee4a20 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.243+0000 7f8fc6a00640 10 client.25603 handle_command_reply: tid=0 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.243+0000 7f8fea7b4740 1 client.25603 shutdown 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1162071714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 msgr2=0x556ac8ed1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ed1910 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4009560 tx=0x7f8fd4009ee0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1162071714 shutdown_connections 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556ac8ee4a20 0x556ac8ee6e10 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 0x556ac8ed2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 0x556ac8ed1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fc6a00640 0 client.25603 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:41.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1162071714 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ed1910 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1162071714 >> 172.21.15.103:0/1162071714 conn(0x556ac8ec5cc0 msgr2=0x556ac8ecbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1162071714 shutdown_connections 2024-05-23T00:00:41.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1162071714 wait complete. 2024-05-23T00:00:41.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.244+0000 7f8fea7b4740 20 client.25603 trim_cache size 0 max 16384 2024-05-23T00:00:41.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fea7b4740 1 Processor -- start 2024-05-23T00:00:41.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fea7b4740 1 -- start start 2024-05-23T00:00:41.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ee0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8eca2c0 0x556ac8ed0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 0x556ac8ec4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01b3700 con 0x556ac8ed4990 2024-05-23T00:00:41.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01b0fd0 con 0x556ac8eca2c0 2024-05-23T00:00:41.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01aee20 con 0x556ac8ec99a0 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fe4e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ee0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fe4e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ee0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41814/0 (socket says 172.21.15.103:41814) 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fe4e00640 1 -- 172.21.15.103:0/2919490847 learned_addr learned my addr 172.21.15.103:0/2919490847 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.266+0000 7f8fe5800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 0x556ac8ec4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8eca2c0 0x556ac8ed0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fe4e00640 1 -- 172.21.15.103:0/2919490847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8eca2c0 msgr2=0x556ac8ed0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8eca2c0 0x556ac8ed0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fe4e00640 1 -- 172.21.15.103:0/2919490847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 msgr2=0x556ac8ec4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 0x556ac8ec4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fe4e00640 1 -- 172.21.15.103:0/2919490847 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac8eca7c0 con 0x556ac8ec99a0 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8eca2c0 0x556ac8ed0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:41.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ee0510 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40058c0 tx=0x7f8fd4005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8fc740c685f602d server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fd1600640 1 -- 172.21.15.103:0/2919490847 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4027070 con 0x556ac8ec99a0 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fd1600640 1 -- 172.21.15.103:0/2919490847 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fd4015030 con 0x556ac8ec99a0 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.267+0000 7f8fd1600640 1 -- 172.21.15.103:0/2919490847 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd402c9b0 con 0x556ac8ec99a0 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 -- 172.21.15.103:0/2919490847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 msgr2=0x556ac8ee0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ee0510 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40058c0 tx=0x7f8fd4005850 comp rx=0 tx=0).stop 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 -- 172.21.15.103:0/2919490847 shutdown_connections 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8ed4990 0x556ac8ec4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8eca2c0 0x556ac8ed0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/2919490847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8ee0510 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 -- 172.21.15.103:0/2919490847 >> 172.21.15.103:0/2919490847 conn(0x556ac8ec5cc0 msgr2=0x556ac8ec4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 -- 172.21.15.103:0/2919490847 shutdown_connections 2024-05-23T00:00:41.272 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.268+0000 7f8fea7b4740 1 -- 172.21.15.103:0/2919490847 wait complete. 2024-05-23T00:00:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 1 -- start start 2024-05-23T00:00:41.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 1 Processor -- start 2024-05-23T00:00:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8edccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 0x556ac8ece1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 0x556ac8ece720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01af490 con 0x556ac8eca2c0 2024-05-23T00:00:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01b0900 con 0x556ac8ed4990 2024-05-23T00:00:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe01afb60 con 0x556ac8ec99a0 2024-05-23T00:00:41.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.269+0000 7f8fea7b4740 10 client.0 _ll_register_callbacks cb 0x7f8fd4009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8edccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8edccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41824/0 (socket says 172.21.15.103:41824) 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 -- 172.21.15.103:0/1104689102 learned_addr learned my addr 172.21.15.103:0/1104689102 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 0x556ac8ece1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe5800640 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 0x556ac8ece720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 -- 172.21.15.103:0/1104689102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 msgr2=0x556ac8ece720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 0x556ac8ece720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 -- 172.21.15.103:0/1104689102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 msgr2=0x556ac8ece1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 0x556ac8ece1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 -- 172.21.15.103:0/1104689102 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac8ecece0 con 0x556ac8ec99a0 2024-05-23T00:00:41.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fe4e00640 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8edccc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40059f0 tx=0x7f8fd4009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7285296e3eb34328 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1104689102 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4027040 con 0x556ac8ec99a0 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1104689102 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8fd4015030 con 0x556ac8ec99a0 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.270+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1104689102 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8fd40097a0 con 0x556ac8ec99a0 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1104689102 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8fd4045050 con 0x556ac8ec99a0 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fea7b4740 10 client.25609 fetch_fsmap learned FSMap version 25 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1104689102 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556ac8f4ac70 con 0x556ac8ec99a0 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1104689102 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8fd4015030 con 0x556ac8ec99a0 2024-05-23T00:00:41.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1104689102 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8fd4018070 con 0x556ac8ec99a0 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fea7b4740 10 client.25609 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fea7b4740 10 client.25609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fea7b4740 20 client.25609 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556ac8f4b380 0x556ac8f4d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fea7b4740 4 client.25609 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1104689102 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556ac8e3c3e0 con 0x556ac8f4b380 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.271+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556ac8f4b380 0x556ac8f4d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fdfe00640 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556ac8f4b380 0x556ac8f4d7a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d4a40e576ded5b server_cookie=e246be8a1bf76c21 in_seq=0 out_seq=0 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fc6a00640 10 client.25609 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fc6a00640 1 -- 172.21.15.103:0/1104689102 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556ac8e3c3e0 con 0x556ac8f4b380 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fc6a00640 10 client.25609 handle_command_reply: tid=0 2024-05-23T00:00:41.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 client.25609 shutdown 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1104689102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 msgr2=0x556ac8edccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8edccc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40059f0 tx=0x7f8fd4009f70 comp rx=0 tx=0).stop 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1104689102 shutdown_connections 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556ac8f4b380 0x556ac8f4d7a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556ac8ed4990 0x556ac8ece720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556ac8eca2c0 0x556ac8ece1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 --2- 172.21.15.103:0/1104689102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556ac8ec99a0 0x556ac8edccc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fc6a00640 0 client.25609 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1104689102 >> 172.21.15.103:0/1104689102 conn(0x556ac8ec5cc0 msgr2=0x556ac8ed0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1104689102 shutdown_connections 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 1 -- 172.21.15.103:0/1104689102 wait complete. 2024-05-23T00:00:41.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.272+0000 7f8fea7b4740 20 client.25609 trim_cache size 0 max 16384 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "scrubs": { 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "2024c2e0-4890-42df-91e6-999eaab35664": { 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "path": "/", 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "tag": "2024c2e0-4890-42df-91e6-999eaab35664", 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: "options": "recursive,force" 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:00:41.281 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:00:41.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 -- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8fc80a2e10 msgr2=0x7f8fc80a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8fc80a2e10 0x7f8fc80a52d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8fd8009a60 tx=0x7f8fd8005d80 comp rx=0 tx=0).stop 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 -- 172.21.15.103:0/185348808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fe01043f0 msgr2=0x7f8fe019a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fe01043f0 0x7f8fe019a820 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00e9f0 tx=0x7f8fcc00eeb0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 -- 172.21.15.103:0/185348808 shutdown_connections 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8fc80a2e10 0x7f8fc80a52d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8fe0109a40 0x7f8fe019eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fe01043f0 0x7f8fe019a820 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 --2- 172.21.15.103:0/185348808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8fe0103a30 0x7f8fe019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 -- 172.21.15.103:0/185348808 >> 172.21.15.103:0/185348808 conn(0x7f8fe00ff520 msgr2=0x7f8fe00ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 -- 172.21.15.103:0/185348808 shutdown_connections 2024-05-23T00:00:41.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.273+0000 7f8fe7800640 1 -- 172.21.15.103:0/185348808 wait complete. 2024-05-23T00:00:41.292 INFO:tasks.cephfs.filesystem:scrub status for tag:2024c2e0-4890-42df-91e6-999eaab35664 - {'path': '/', 'tag': '2024c2e0-4890-42df-91e6-999eaab35664', 'options': 'recursive,force'} 2024-05-23T00:00:41.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 -- 172.21.15.103:0/100888535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101670 msgr2=0x7f7794101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 --2- 172.21.15.103:0/100888535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101670 0x7f7794101af0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f777c009930 tx=0x7f777c02a420 comp rx=0 tx=0).stop 2024-05-23T00:00:41.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 -- 172.21.15.103:0/100888535 shutdown_connections 2024-05-23T00:00:41.403 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 --2- 172.21.15.103:0/100888535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7794102030 0x7f779410e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 --2- 172.21.15.103:0/100888535 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794101670 0x7f7794101af0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 --2- 172.21.15.103:0/100888535 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7794107680 0x7f7794107a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 -- 172.21.15.103:0/100888535 >> 172.21.15.103:0/100888535 conn(0x7f77940fd1c0 msgr2=0x7f77940ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 -- 172.21.15.103:0/100888535 shutdown_connections 2024-05-23T00:00:41.404 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.400+0000 7f7793200640 1 -- 172.21.15.103:0/100888535 wait complete. 2024-05-23T00:00:41.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.402+0000 7f7793200640 1 Processor -- start 2024-05-23T00:00:41.406 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.402+0000 7f7793200640 1 -- start start 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.402+0000 7f7793200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7794101670 0x7f7794197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.402+0000 7f7793200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794102030 0x7f77941984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.402+0000 7f7793200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7794107680 0x7f779419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.402+0000 7f7793200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7794111a90 con 0x7f7794107680 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.402+0000 7f7793200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7794111910 con 0x7f7794101670 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.402+0000 7f7793200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7794111c10 con 0x7f7794102030 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7794101670 0x7f7794197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7794101670 0x7f7794197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49694/0 (socket says 172.21.15.103:49694) 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 -- 172.21.15.103:0/464786929 learned_addr learned my addr 172.21.15.103:0/464786929 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.407 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7789400640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794102030 0x7f77941984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7792800640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7794107680 0x7f779419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 -- 172.21.15.103:0/464786929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794102030 msgr2=0x7f77941984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794102030 0x7f77941984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 -- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7794107680 msgr2=0x7f779419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7794107680 0x7f779419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 -- 172.21.15.103:0/464786929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779419cfa0 con 0x7f7794101670 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.403+0000 7f7791e00640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7794101670 0x7f7794197fb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f778400db80 tx=0x7f778400b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.404+0000 7f778b400640 1 -- 172.21.15.103:0/464786929 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7784017070 con 0x7f7794101670 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.404+0000 7f778b400640 1 -- 172.21.15.103:0/464786929 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7784003b70 con 0x7f7794101670 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.404+0000 7f778b400640 1 -- 172.21.15.103:0/464786929 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77840106c0 con 0x7f7794101670 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.404+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f779419d290 con 0x7f7794101670 2024-05-23T00:00:41.408 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.404+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77941a4b20 con 0x7f7794101670 2024-05-23T00:00:41.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.405+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7794102db0 con 0x7f7794101670 2024-05-23T00:00:41.409 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.407+0000 7f778b400640 1 -- 172.21.15.103:0/464786929 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f7784004020 con 0x7f7794101670 2024-05-23T00:00:41.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.408+0000 7f778b400640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f77780a2e00 0x7f77780a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.408+0000 7f778b400640 1 -- 172.21.15.103:0/464786929 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f77840cf5e0 con 0x7f7794101670 2024-05-23T00:00:41.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.408+0000 7f7789400640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f77780a2e00 0x7f77780a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.409+0000 7f7789400640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f77780a2e00 0x7f77780a52c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f777c009900 tx=0x7f777c0058d0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.414 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.412+0000 7f778b400640 1 -- 172.21.15.103:0/464786929 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7784020070 con 0x7f7794101670 2024-05-23T00:00:41.558 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:00:41 smithi105 sudo[69881]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:41.558 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:00:41 smithi105 sudo[69881]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:41.558 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:00:41 smithi105 sudo[69881]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:41.558 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:00:41 smithi105 sudo[69881]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:41.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.614+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f7794109c40 con 0x7f7794101670 2024-05-23T00:00:41.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.623+0000 7f778b400640 1 -- 172.21.15.103:0/464786929 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) ==== 138+0+0 (secure 0 0 0) 0x7f778409bcf0 con 0x7f7794101670 2024-05-23T00:00:41.625 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:00:41 smithi174 sudo[70344]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:41.626 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:00:41 smithi174 sudo[70344]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:41.626 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:00:41 smithi174 sudo[70344]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:41.626 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:00:41 smithi174 sudo[70344]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f77780a2e00 msgr2=0x7f77780a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f77780a2e00 0x7f77780a52c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f777c009900 tx=0x7f777c0058d0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7794101670 msgr2=0x7f7794197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7794101670 0x7f7794197fb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f778400db80 tx=0x7f778400b630 comp rx=0 tx=0).stop 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 shutdown_connections 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f77780a2e00 0x7f77780a52c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7794107680 0x7f779419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7794102030 0x7f77941984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.629 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 --2- 172.21.15.103:0/464786929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7794101670 0x7f7794197fb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 >> 172.21.15.103:0/464786929 conn(0x7f77940fd1c0 msgr2=0x7f77940fda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 shutdown_connections 2024-05-23T00:00:41.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.626+0000 7f7793200640 1 -- 172.21.15.103:0/464786929 wait complete. 2024-05-23T00:00:41.649 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-05-23T00:00:41.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 -- 172.21.15.103:0/2772467141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 msgr2=0x7f52dc10b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 --2- 172.21.15.103:0/2772467141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 0x7f52dc10b940 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009900 tx=0x7f52c402a480 comp rx=0 tx=0).stop 2024-05-23T00:00:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 -- 172.21.15.103:0/2772467141 shutdown_connections 2024-05-23T00:00:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 --2- 172.21.15.103:0/2772467141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52dc10be80 0x7f52dc10e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 --2- 172.21.15.103:0/2772467141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 0x7f52dc10b940 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 --2- 172.21.15.103:0/2772467141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52dc102b40 0x7f52dc102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 -- 172.21.15.103:0/2772467141 >> 172.21.15.103:0/2772467141 conn(0x7f52dc0778c0 msgr2=0x7f52dc077cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 -- 172.21.15.103:0/2772467141 shutdown_connections 2024-05-23T00:00:41.831 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.828+0000 7f52e1600640 1 -- 172.21.15.103:0/2772467141 wait complete. 2024-05-23T00:00:41.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.829+0000 7f52e1600640 1 Processor -- start 2024-05-23T00:00:41.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.829+0000 7f52e1600640 1 -- start start 2024-05-23T00:00:41.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.829+0000 7f52e1600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52dc102b40 0x7f52dc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.829+0000 7f52e1600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 0x7f52dc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.829+0000 7f52e1600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52dc10be80 0x7f52dc19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.829+0000 7f52e1600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc111c60 con 0x7f52dc102b40 2024-05-23T00:00:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.829+0000 7f52e1600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc111ae0 con 0x7f52dc103460 2024-05-23T00:00:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.829+0000 7f52e1600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52dc111de0 con 0x7f52dc10be80 2024-05-23T00:00:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 0x7f52dc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52daa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52dc102b40 0x7f52dc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 0x7f52dc19a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49706/0 (socket says 172.21.15.103:49706) 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 -- 172.21.15.103:0/2230782546 learned_addr learned my addr 172.21.15.103:0/2230782546 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52db400640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52dc10be80 0x7f52dc19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 -- 172.21.15.103:0/2230782546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52dc10be80 msgr2=0x7f52dc19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52dc10be80 0x7f52dc19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 -- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52dc102b40 msgr2=0x7f52dc19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52dc102b40 0x7f52dc19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 -- 172.21.15.103:0/2230782546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52dc19f270 con 0x7f52dc103460 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52daa00640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52dc102b40 0x7f52dc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.830+0000 7f52d2000640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 0x7f52dc19a7c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f52c4006010 tx=0x7f52c40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.831+0000 7f52d3e00640 1 -- 172.21.15.103:0/2230782546 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c403f030 con 0x7f52dc103460 2024-05-23T00:00:41.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.831+0000 7f52d3e00640 1 -- 172.21.15.103:0/2230782546 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52c4003ce0 con 0x7f52dc103460 2024-05-23T00:00:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.831+0000 7f52d3e00640 1 -- 172.21.15.103:0/2230782546 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4035070 con 0x7f52dc103460 2024-05-23T00:00:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.831+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52dc19f500 con 0x7f52dc103460 2024-05-23T00:00:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.831+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52dc1a6df0 con 0x7f52dc103460 2024-05-23T00:00:41.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.832+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52dc06a280 con 0x7f52dc103460 2024-05-23T00:00:41.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.834+0000 7f52d3e00640 1 -- 172.21.15.103:0/2230782546 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f52c4042030 con 0x7f52dc103460 2024-05-23T00:00:41.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.835+0000 7f52d3e00640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f52c00a2ec0 0x7f52c00a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:41.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.835+0000 7f52d3e00640 1 -- 172.21.15.103:0/2230782546 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f52c40efda0 con 0x7f52dc103460 2024-05-23T00:00:41.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.835+0000 7f52daa00640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f52c00a2ec0 0x7f52c00a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:41.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.836+0000 7f52daa00640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f52c00a2ec0 0x7f52c00a5380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009fd0 tx=0x7f52cc009760 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:41.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:41.839+0000 7f52d3e00640 1 -- 172.21.15.103:0/2230782546 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f52c40bbf20 con 0x7f52dc103460 2024-05-23T00:00:42.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.029+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7f52dc1a6fc0 con 0x7f52dc103460 2024-05-23T00:00:42.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.036+0000 7f52d3e00640 1 -- 172.21.15.103:0/2230782546 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) ==== 133+0+0 (secure 0 0 0) 0x7f52c40c0220 con 0x7f52dc103460 2024-05-23T00:00:42.041 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:00:41 smithi174 sudo[70379]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:42.041 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:00:41 smithi174 sudo[70379]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:42.041 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:00:41 smithi174 sudo[70379]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:42.041 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:00:41 smithi174 sudo[70379]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:42.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.040+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f52c00a2ec0 msgr2=0x7f52c00a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.040+0000 7f52e1600640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f52c00a2ec0 0x7f52c00a5380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009fd0 tx=0x7f52cc009760 comp rx=0 tx=0).stop 2024-05-23T00:00:42.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.040+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 msgr2=0x7f52dc19a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.040+0000 7f52e1600640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 0x7f52dc19a7c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f52c4006010 tx=0x7f52c40042f0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.040+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 shutdown_connections 2024-05-23T00:00:42.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.040+0000 7f52e1600640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f52c00a2ec0 0x7f52c00a5380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.040+0000 7f52e1600640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52dc10be80 0x7f52dc19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.041+0000 7f52e1600640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52dc103460 0x7f52dc19a7c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.041+0000 7f52e1600640 1 --2- 172.21.15.103:0/2230782546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f52dc102b40 0x7f52dc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.041+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 >> 172.21.15.103:0/2230782546 conn(0x7f52dc0778c0 msgr2=0x7f52dc107cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.041+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 shutdown_connections 2024-05-23T00:00:42.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.041+0000 7f52e1600640 1 -- 172.21.15.103:0/2230782546 wait complete. 2024-05-23T00:00:42.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 -- 172.21.15.103:0/692990425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 msgr2=0x7fd36010da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 --2- 172.21.15.103:0/692990425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 0x7fd36010da10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd34c009900 tx=0x7fd34c02a480 comp rx=0 tx=0).stop 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 -- 172.21.15.103:0/692990425 shutdown_connections 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 --2- 172.21.15.103:0/692990425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd36010df50 0x7fd360110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 --2- 172.21.15.103:0/692990425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 0x7fd36010da10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 --2- 172.21.15.103:0/692990425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd360073c20 0x7fd360074000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 -- 172.21.15.103:0/692990425 >> 172.21.15.103:0/692990425 conn(0x7fd3600fd1c0 msgr2=0x7fd3600ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 -- 172.21.15.103:0/692990425 shutdown_connections 2024-05-23T00:00:42.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.232+0000 7fd365400640 1 -- 172.21.15.103:0/692990425 wait complete. 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.233+0000 7fd365400640 1 Processor -- start 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.233+0000 7fd365400640 1 -- start start 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd365400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd360073c20 0x7fd36019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd365400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 0x7fd36019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd365400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd36010df50 0x7fd36019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd365400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd360113cc0 con 0x7fd360073c20 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd365400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fd360113b40 con 0x7fd360074540 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd365400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd360113e40 con 0x7fd36010df50 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd35fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd360073c20 0x7fd36019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:42.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd35f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 0x7fd36019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd35fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd360073c20 0x7fd36019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39002/0 (socket says 172.21.15.103:39002) 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd35fe00640 1 -- 172.21.15.103:0/1775037140 learned_addr learned my addr 172.21.15.103:0/1775037140 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd364a00640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd36010df50 0x7fd36019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd35fe00640 1 -- 172.21.15.103:0/1775037140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd36010df50 msgr2=0x7fd36019ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd35fe00640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd36010df50 0x7fd36019ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd35fe00640 1 -- 172.21.15.103:0/1775037140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 msgr2=0x7fd36019a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.234+0000 7fd35fe00640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 0x7fd36019a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.235+0000 7fd35fe00640 1 -- 172.21.15.103:0/1775037140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36019f180 con 0x7fd360073c20 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.235+0000 7fd35f400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 0x7fd36019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.235+0000 7fd35fe00640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd360073c20 0x7fd36019a1e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd35000ebc0 tx=0x7fd35000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.235+0000 7fd35cc00640 1 -- 172.21.15.103:0/1775037140 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350010630 con 0x7fd360073c20 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.235+0000 7fd35cc00640 1 -- 172.21.15.103:0/1775037140 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd350010cb0 con 0x7fd360073c20 2024-05-23T00:00:42.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.235+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36019f470 con 0x7fd360073c20 2024-05-23T00:00:42.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.235+0000 7fd35cc00640 1 -- 172.21.15.103:0/1775037140 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350018bd0 con 0x7fd360073c20 2024-05-23T00:00:42.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.235+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3601a6d00 con 0x7fd360073c20 2024-05-23T00:00:42.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.237+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36004fe80 con 0x7fd360073c20 2024-05-23T00:00:42.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.238+0000 7fd35cc00640 1 -- 172.21.15.103:0/1775037140 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7fd3500107d0 con 0x7fd360073c20 2024-05-23T00:00:42.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.240+0000 7fd35cc00640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fd3440a2e00 0x7fd3440a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:42.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.240+0000 7fd35cc00640 1 -- 172.21.15.103:0/1775037140 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fd3500cff30 con 0x7fd360073c20 2024-05-23T00:00:42.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.240+0000 7fd35f400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fd3440a2e00 0x7fd3440a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:42.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.241+0000 7fd35f400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fd3440a2e00 0x7fd3440a52c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd34c000c00 tx=0x7fd34c031510 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:42.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.244+0000 7fd35cc00640 1 -- 172.21.15.103:0/1775037140 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fd35009bff0 con 0x7fd360073c20 2024-05-23T00:00:42.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 168 KiB/s wr, 36 op/s 2024-05-23T00:00:42.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: scrub queued for path: / 2024-05-23T00:00:42.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: scrub summary: idle+waiting paths [/] 2024-05-23T00:00:42.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: scrub summary: active paths [/] 2024-05-23T00:00:42.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: scrub summary: idle+waiting paths [/] 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:42.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:42 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.329 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:00:42 smithi103 sudo[88426]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:42.329 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:00:42 smithi103 sudo[88426]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:42.329 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:00:42 smithi103 sudo[88426]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:42.329 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:00:42 smithi103 sudo[88426]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:42.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 168 KiB/s wr, 36 op/s 2024-05-23T00:00:42.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: scrub queued for path: / 2024-05-23T00:00:42.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: scrub summary: idle+waiting paths [/] 2024-05-23T00:00:42.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: scrub summary: active paths [/] 2024-05-23T00:00:42.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: scrub summary: idle+waiting paths [/] 2024-05-23T00:00:42.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:42.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:42 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: pgmap v7: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 168 KiB/s wr, 36 op/s 2024-05-23T00:00:42.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: scrub queued for path: / 2024-05-23T00:00:42.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: scrub summary: idle+waiting paths [/] 2024-05-23T00:00:42.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: scrub summary: active paths [/] 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: scrub summary: idle+waiting paths [/] 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:42.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:42.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:42 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:42.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.462+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7fd360061cb0 con 0x7fd3440a2e00 2024-05-23T00:00:42.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.588+0000 7fd35cc00640 1 -- 172.21.15.103:0/1775037140 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd360061cb0 con 0x7fd3440a2e00 2024-05-23T00:00:42.590 INFO:teuthology.orchestra.run.smithi103.stdout:Schedule set for path / 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fd3440a2e00 msgr2=0x7fd3440a52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fd3440a2e00 0x7fd3440a52c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd34c000c00 tx=0x7fd34c031510 comp rx=0 tx=0).stop 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd360073c20 msgr2=0x7fd36019a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd360073c20 0x7fd36019a1e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd35000ebc0 tx=0x7fd35000c6a0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 shutdown_connections 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fd3440a2e00 0x7fd3440a52c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd36010df50 0x7fd36019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd360074540 0x7fd36019a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 --2- 172.21.15.103:0/1775037140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd360073c20 0x7fd36019a1e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 >> 172.21.15.103:0/1775037140 conn(0x7fd3600fd1c0 msgr2=0x7fd3600fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:42.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.590+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 shutdown_connections 2024-05-23T00:00:42.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.591+0000 7fd365400640 1 -- 172.21.15.103:0/1775037140 wait complete. 2024-05-23T00:00:42.608 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 -- 172.21.15.103:0/896961486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4e4101850 msgr2=0x7fc4e4101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 --2- 172.21.15.103:0/896961486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4e4101850 0x7fc4e4101cd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009890 tx=0x7fc4d402a360 comp rx=0 tx=0).stop 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 -- 172.21.15.103:0/896961486 shutdown_connections 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 --2- 172.21.15.103:0/896961486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4e4102210 0x7fc4e410e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 --2- 172.21.15.103:0/896961486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4e4101850 0x7fc4e4101cd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 --2- 172.21.15.103:0/896961486 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4107860 0x7fc4e4107c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 -- 172.21.15.103:0/896961486 >> 172.21.15.103:0/896961486 conn(0x7fc4e40fd320 msgr2=0x7fc4e40ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 -- 172.21.15.103:0/896961486 shutdown_connections 2024-05-23T00:00:42.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.785+0000 7fc4ec800640 1 -- 172.21.15.103:0/896961486 wait complete. 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.786+0000 7fc4ec800640 1 Processor -- start 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4ec800640 1 -- start start 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4ec800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4101850 0x7fc4e4198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4ec800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4e4102210 0x7fc4e4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4ec800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4e4107860 0x7fc4e419c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4ec800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4111c70 con 0x7fc4e4101850 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4ec800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4111af0 con 0x7fc4e4102210 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4ec800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e4111df0 con 0x7fc4e4107860 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4e9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4101850 0x7fc4e4198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:42.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4e9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4101850 0x7fc4e4198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:39014/0 (socket says 172.21.15.103:39014) 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4e9e00640 1 -- 172.21.15.103:0/3967560175 learned_addr learned my addr 172.21.15.103:0/3967560175 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4ea800640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4e4107860 0x7fc4e419c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.787+0000 7fc4e9400640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4e4102210 0x7fc4e4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4e9e00640 1 -- 172.21.15.103:0/3967560175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4e4107860 msgr2=0x7fc4e419c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4e9e00640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4e4107860 0x7fc4e419c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4e9e00640 1 -- 172.21.15.103:0/3967560175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4e4102210 msgr2=0x7fc4e4198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4e9e00640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4e4102210 0x7fc4e4198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4e9e00640 1 -- 172.21.15.103:0/3967560175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e419d100 con 0x7fc4e4101850 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4e9e00640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4101850 0x7fc4e4198110 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800e9c0 tx=0x7fc4d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4dea00640 1 -- 172.21.15.103:0/3967560175 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d800cde0 con 0x7fc4e4101850 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4dea00640 1 -- 172.21.15.103:0/3967560175 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4d80035f0 con 0x7fc4e4101850 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4dea00640 1 -- 172.21.15.103:0/3967560175 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d8010690 con 0x7fc4e4101850 2024-05-23T00:00:42.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e419d3f0 con 0x7fc4e4101850 2024-05-23T00:00:42.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.788+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e41a4c80 con 0x7fc4e4101850 2024-05-23T00:00:42.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.790+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4e404f630 con 0x7fc4e4101850 2024-05-23T00:00:42.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.792+0000 7fc4dea00640 1 -- 172.21.15.103:0/3967560175 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7fc4d80037a0 con 0x7fc4e4101850 2024-05-23T00:00:42.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.793+0000 7fc4dea00640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4cc0a2ec0 0x7fc4cc0a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:42.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.793+0000 7fc4dea00640 1 -- 172.21.15.103:0/3967560175 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc4d80cfc20 con 0x7fc4e4101850 2024-05-23T00:00:42.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.793+0000 7fc4e9400640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4cc0a2ec0 0x7fc4cc0a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:42.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.794+0000 7fc4e9400640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4cc0a2ec0 0x7fc4cc0a5380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc4d402a950 tx=0x7fc4d40056d0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:42.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:42.795+0000 7fc4dea00640 1 -- 172.21.15.103:0/3967560175 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc4d809c420 con 0x7fc4e4101850 2024-05-23T00:00:43.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.021+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7fc4e4061cb0 con 0x7fc4cc0a2ec0 2024-05-23T00:00:43.043 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:00:42 smithi103 sudo[88488]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:43.043 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:00:42 smithi103 sudo[88488]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:43.043 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:00:42 smithi103 sudo[88488]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:43.043 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:00:42 smithi103 sudo[88488]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:43.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.152+0000 7fc4dea00640 1 -- 172.21.15.103:0/3967560175 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7fc4e4061cb0 con 0x7fc4cc0a2ec0 2024-05-23T00:00:43.155 INFO:teuthology.orchestra.run.smithi103.stdout:Retention added to path / 2024-05-23T00:00:43.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4cc0a2ec0 msgr2=0x7fc4cc0a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4cc0a2ec0 0x7fc4cc0a5380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc4d402a950 tx=0x7fc4d40056d0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4101850 msgr2=0x7fc4e4198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4101850 0x7fc4e4198110 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc4d800e9c0 tx=0x7fc4d800ee80 comp rx=0 tx=0).stop 2024-05-23T00:00:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 shutdown_connections 2024-05-23T00:00:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4cc0a2ec0 0x7fc4cc0a5380 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc4e4107860 0x7fc4e419c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4e4102210 0x7fc4e4198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 --2- 172.21.15.103:0/3967560175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4101850 0x7fc4e4198110 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 >> 172.21.15.103:0/3967560175 conn(0x7fc4e40fd320 msgr2=0x7fc4e40fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 shutdown_connections 2024-05-23T00:00:43.159 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.156+0000 7fc4ec800640 1 -- 172.21.15.103:0/3967560175 wait complete. 2024-05-23T00:00:43.178 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-23T00:00:43.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:00:43 smithi105 sudo[70012]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:43.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:00:43 smithi105 sudo[70012]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:43.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:00:43 smithi105 sudo[70012]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:43.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:00:43 smithi105 sudo[70012]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:43.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:43 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' 2024-05-23T00:00:43.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:43 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:43.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:43 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:43.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:43 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:43.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:43 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:43.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:43 smithi105 ceph-mon[36426]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:43.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:43 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' 2024-05-23T00:00:43.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:43 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:43.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:43 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:43.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:43 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:43.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:43 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:43.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:43 smithi103 ceph-mon[31370]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:43.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:43 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' 2024-05-23T00:00:43.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:43 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:00:43.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:43 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:00:43.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:43 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:00:43.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:43 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:00:43.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:43 smithi174 ceph-mon[36179]: from='client.15513 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:43.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.365+0000 7fcd51400640 1 -- 172.21.15.103:0/2239978937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 msgr2=0x7fcd4c109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.365+0000 7fcd51400640 1 --2- 172.21.15.103:0/2239978937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 0x7fcd4c109e60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c009930 tx=0x7fcd3c02a420 comp rx=0 tx=0).stop 2024-05-23T00:00:43.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.365+0000 7fcd51400640 1 -- 172.21.15.103:0/2239978937 shutdown_connections 2024-05-23T00:00:43.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.365+0000 7fcd51400640 1 --2- 172.21.15.103:0/2239978937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd4c104430 0x7fcd4c110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.365+0000 7fcd51400640 1 --2- 172.21.15.103:0/2239978937 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd4c103a70 0x7fcd4c103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.365+0000 7fcd51400640 1 --2- 172.21.15.103:0/2239978937 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 0x7fcd4c109e60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.365+0000 7fcd51400640 1 -- 172.21.15.103:0/2239978937 >> 172.21.15.103:0/2239978937 conn(0x7fcd4c0ff5a0 msgr2=0x7fcd4c1019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:43.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.366+0000 7fcd51400640 1 -- 172.21.15.103:0/2239978937 shutdown_connections 2024-05-23T00:00:43.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.366+0000 7fcd51400640 1 -- 172.21.15.103:0/2239978937 wait complete. 2024-05-23T00:00:43.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.367+0000 7fcd51400640 1 Processor -- start 2024-05-23T00:00:43.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.367+0000 7fcd51400640 1 -- start start 2024-05-23T00:00:43.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.367+0000 7fcd51400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd4c103a70 0x7fcd4c19a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:43.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.367+0000 7fcd51400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd4c104430 0x7fcd4c19a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:43.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.367+0000 7fcd51400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 0x7fcd4c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.367+0000 7fcd51400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c113e60 con 0x7fcd4c104430 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.367+0000 7fcd51400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c113ce0 con 0x7fcd4c103a70 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.367+0000 7fcd51400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcd4c113fe0 con 0x7fcd4c109a80 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4a800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd4c103a70 0x7fcd4c19a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd49e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd4c104430 0x7fcd4c19a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4a800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd4c103a70 0x7fcd4c19a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49748/0 (socket says 172.21.15.103:49748) 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4a800640 1 -- 172.21.15.103:0/1864748326 learned_addr learned my addr 172.21.15.103:0/1864748326 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4b200640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 0x7fcd4c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4b200640 1 -- 172.21.15.103:0/1864748326 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd4c103a70 msgr2=0x7fcd4c19a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4b200640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd4c103a70 0x7fcd4c19a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4b200640 1 -- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd4c104430 msgr2=0x7fcd4c19a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4b200640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd4c104430 0x7fcd4c19a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4b200640 1 -- 172.21.15.103:0/1864748326 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4c19f310 con 0x7fcd4c109a80 2024-05-23T00:00:43.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.368+0000 7fcd4b200640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 0x7fcd4c19ebe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009fd0 tx=0x7fcd4400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:43.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.369+0000 7fcd43400640 1 -- 172.21.15.103:0/1864748326 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd4400ce20 con 0x7fcd4c109a80 2024-05-23T00:00:43.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.369+0000 7fcd43400640 1 -- 172.21.15.103:0/1864748326 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd44005020 con 0x7fcd4c109a80 2024-05-23T00:00:43.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.369+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd4c19f600 con 0x7fcd4c109a80 2024-05-23T00:00:43.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.369+0000 7fcd43400640 1 -- 172.21.15.103:0/1864748326 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd44010720 con 0x7fcd4c109a80 2024-05-23T00:00:43.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.369+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd4c1a6e90 con 0x7fcd4c109a80 2024-05-23T00:00:43.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.370+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd4c04f630 con 0x7fcd4c109a80 2024-05-23T00:00:43.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.372+0000 7fcd43400640 1 -- 172.21.15.103:0/1864748326 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7fcd44005540 con 0x7fcd4c109a80 2024-05-23T00:00:43.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.373+0000 7fcd43400640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd300a2ec0 0x7fcd300a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:43.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.373+0000 7fcd43400640 1 -- 172.21.15.103:0/1864748326 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fcd440cffe0 con 0x7fcd4c109a80 2024-05-23T00:00:43.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.374+0000 7fcd4a800640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd300a2ec0 0x7fcd300a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:43.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.374+0000 7fcd4a800640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd300a2ec0 0x7fcd300a5380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c000c00 tx=0x7fcd3c009560 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:43.377 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.375+0000 7fcd43400640 1 -- 172.21.15.103:0/1864748326 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fcd4409c710 con 0x7fcd4c109a80 2024-05-23T00:00:43.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.597+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fcd4c061cb0 con 0x7fcd300a2ec0 2024-05-23T00:00:43.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.614+0000 7fcd43400640 1 -- 172.21.15.103:0/1864748326 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fcd4c061cb0 con 0x7fcd300a2ec0 2024-05-23T00:00:43.616 INFO:teuthology.orchestra.run.smithi103.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-23T00:00:00", "created": "2024-05-23T00:00:42", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd300a2ec0 msgr2=0x7fcd300a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd300a2ec0 0x7fcd300a5380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c000c00 tx=0x7fcd3c009560 comp rx=0 tx=0).stop 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 msgr2=0x7fcd4c19ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 0x7fcd4c19ebe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009fd0 tx=0x7fcd4400ef10 comp rx=0 tx=0).stop 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 shutdown_connections 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd300a2ec0 0x7fcd300a5380 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd4c109a80 0x7fcd4c19ebe0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcd4c104430 0x7fcd4c19a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 --2- 172.21.15.103:0/1864748326 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd4c103a70 0x7fcd4c19a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 >> 172.21.15.103:0/1864748326 conn(0x7fcd4c0ff5a0 msgr2=0x7fcd4c0ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:43.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.616+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 shutdown_connections 2024-05-23T00:00:43.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.617+0000 7fcd51400640 1 -- 172.21.15.103:0/1864748326 wait complete. 2024-05-23T00:00:43.637 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-23T00:00:43.808 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:00:43 smithi105 sudo[70015]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:43.808 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:00:43 smithi105 sudo[70015]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:43.808 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:00:43 smithi105 sudo[70015]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:43.808 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:00:43 smithi105 sudo[70015]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:43.832 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:00:43 smithi174 sudo[70446]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:43.832 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:00:43 smithi174 sudo[70446]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:43.832 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:00:43 smithi174 sudo[70446]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:43.832 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:00:43 smithi174 sudo[70446]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:43.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 -- 172.21.15.103:0/3899377289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c112710 msgr2=0x7fa91c114b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 --2- 172.21.15.103:0/3899377289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c112710 0x7fa91c114b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa914009c10 tx=0x7fa91402a6f0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 -- 172.21.15.103:0/3899377289 shutdown_connections 2024-05-23T00:00:43.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 --2- 172.21.15.103:0/3899377289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c112710 0x7fa91c114b00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 --2- 172.21.15.103:0/3899377289 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91c0766c0 0x7fa91c1121d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 --2- 172.21.15.103:0/3899377289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c075da0 0x7fa91c076180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 -- 172.21.15.103:0/3899377289 >> 172.21.15.103:0/3899377289 conn(0x7fa91c0ff5a0 msgr2=0x7fa91c1019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:43.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 -- 172.21.15.103:0/3899377289 shutdown_connections 2024-05-23T00:00:43.840 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.836+0000 7fa922e00640 1 -- 172.21.15.103:0/3899377289 wait complete. 2024-05-23T00:00:43.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.837+0000 7fa922e00640 1 Processor -- start 2024-05-23T00:00:43.841 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa922e00640 1 -- start start 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa922e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c075da0 0x7fa91c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa922e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c0766c0 0x7fa91c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa922e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91c112710 0x7fa91c1a2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa922e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c118230 con 0x7fa91c112710 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa922e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1180b0 con 0x7fa91c075da0 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa922e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c1183b0 con 0x7fa91c0766c0 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa913e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c0766c0 0x7fa91c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa91be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c075da0 0x7fa91c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa913e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c0766c0 0x7fa91c19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:41944/0 (socket says 172.21.15.103:41944) 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.838+0000 7fa913e00640 1 -- 172.21.15.103:0/3849190297 learned_addr learned my addr 172.21.15.103:0/3849190297 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa91be00640 1 -- 172.21.15.103:0/3849190297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c0766c0 msgr2=0x7fa91c19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.842 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa91be00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c0766c0 0x7fa91c19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa91be00640 1 -- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91c112710 msgr2=0x7fa91c1a2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa91be00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91c112710 0x7fa91c1a2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa91be00640 1 -- 172.21.15.103:0/3849190297 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c1a36c0 con 0x7fa91c075da0 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa91be00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c075da0 0x7fa91c19e6d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa90c005fd0 tx=0x7fa90c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa919600640 1 -- 172.21.15.103:0/3849190297 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c00cd00 con 0x7fa91c075da0 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa919600640 1 -- 172.21.15.103:0/3849190297 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c0035f0 con 0x7fa91c075da0 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91c1a39b0 con 0x7fa91c075da0 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa919600640 1 -- 172.21.15.103:0/3849190297 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c010670 con 0x7fa91c075da0 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.839+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c1ab240 con 0x7fa91c075da0 2024-05-23T00:00:43.843 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.841+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa91c04f630 con 0x7fa91c075da0 2024-05-23T00:00:43.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.844+0000 7fa919600640 1 -- 172.21.15.103:0/3849190297 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7fa90c0037a0 con 0x7fa91c075da0 2024-05-23T00:00:43.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.845+0000 7fa919600640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9000a2ec0 0x7fa9000a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:43.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.846+0000 7fa919600640 1 -- 172.21.15.103:0/3849190297 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fa90c0cfb10 con 0x7fa91c075da0 2024-05-23T00:00:43.848 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.846+0000 7fa913e00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9000a2ec0 0x7fa9000a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:43.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.846+0000 7fa913e00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9000a2ec0 0x7fa9000a5380 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0788a0 tx=0x7fa904005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:43.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:43.847+0000 7fa919600640 1 -- 172.21.15.103:0/3849190297 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa90c009510 con 0x7fa91c075da0 2024-05-23T00:00:44.076 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.074+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fa91c02d270 con 0x7fa9000a2ec0 2024-05-23T00:00:44.089 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.087+0000 7fa919600640 1 -- 172.21.15.103:0/3849190297 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fa91c02d270 con 0x7fa9000a2ec0 2024-05-23T00:00:44.089 INFO:teuthology.orchestra.run.smithi103.stdout:/ 1m 6m3h 2024-05-23T00:00:44.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9000a2ec0 msgr2=0x7fa9000a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9000a2ec0 0x7fa9000a5380 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa91c0788a0 tx=0x7fa904005f90 comp rx=0 tx=0).stop 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c075da0 msgr2=0x7fa91c19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c075da0 0x7fa91c19e6d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa90c005fd0 tx=0x7fa90c00ee50 comp rx=0 tx=0).stop 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 shutdown_connections 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9000a2ec0 0x7fa9000a5380 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa91c112710 0x7fa91c1a2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c0766c0 0x7fa91c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 --2- 172.21.15.103:0/3849190297 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa91c075da0 0x7fa91c19e6d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 >> 172.21.15.103:0/3849190297 conn(0x7fa91c0ff5a0 msgr2=0x7fa91c101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:44.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 shutdown_connections 2024-05-23T00:00:44.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:00:44.090+0000 7fa922e00640 1 -- 172.21.15.103:0/3849190297 wait complete. 2024-05-23T00:00:44.112 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-05-23T00:00:44.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:00:43 smithi174 sudo[70449]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:44.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:00:43 smithi174 sudo[70449]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:44.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:00:43 smithi174 sudo[70449]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:44.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:00:43 smithi174 sudo[70449]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:44.189 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-05-23T00:00:44.200 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-23T00:00:44.209 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:00:44.209 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-23T00:00:44.216 INFO:teuthology.run_tasks:Running task workunit... 2024-05-23T00:00:44.228 INFO:tasks.workunit:Pulling workunits from ref 935217f1cb4dc722d5303f5ad364c883e9133376 2024-05-23T00:00:44.228 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-23T00:00:44.229 INFO:tasks.workunit:timeout=3h 2024-05-23T00:00:44.229 INFO:tasks.workunit:cleanup=True 2024-05-23T00:00:44.229 DEBUG:teuthology.orchestra.run.smithi103:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-23T00:00:44.270 INFO:teuthology.orchestra.run.smithi103.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-23T00:00:44.270 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-23T00:00:44.271 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-05-23T00:00:44.271 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-23T00:00:44.271 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-23T00:00:44.271 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-22 23:58:07.253389187 +0000 2024-05-23T00:00:44.271 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-22 23:58:07.253389187 +0000 2024-05-23T00:00:44.271 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-23 00:00:29.982708405 +0000 2024-05-23T00:00:44.271 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-22 23:58:07.253389187 +0000 2024-05-23T00:00:44.271 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-23T00:00:44.271 DEBUG:teuthology.orchestra.run.smithi103:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-23T00:00:44.385 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 935217f1cb4dc722d5303f5ad364c883e9133376 2024-05-23T00:00:44.426 INFO:tasks.workunit.client.0.smithi103.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-23T00:00:44.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 sudo[88645]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:44.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 sudo[88645]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:44.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 sudo[88645]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:44.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 sudo[88645]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:45.214 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:44 smithi174 ceph-mon[36179]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 131 KiB/s wr, 28 op/s 2024-05-23T00:00:45.214 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:44 smithi174 ceph-mon[36179]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:45.214 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:44 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:45.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:44 smithi105 ceph-mon[36426]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 131 KiB/s wr, 28 op/s 2024-05-23T00:00:45.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:44 smithi105 ceph-mon[36426]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:45.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:44 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:45.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 sudo[88669]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:45.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 sudo[88669]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:45.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 sudo[88669]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:45.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 sudo[88669]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:45.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 ceph-mon[31370]: pgmap v8: 129 pgs: 129 active+clean; 1.1 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 131 KiB/s wr, 28 op/s 2024-05-23T00:00:45.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 ceph-mon[31370]: from='client.15519 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:45.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:44 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:45.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 sudo[70452]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:45.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 sudo[70452]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:45.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 sudo[70452]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:45.470 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 sudo[70452]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:45.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 sudo[70456]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:45.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 sudo[70456]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:45.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 sudo[70456]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:45.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 sudo[70456]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:45.968 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 sudo[70019]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-23T00:00:45.968 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 sudo[70019]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:45.968 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 sudo[70019]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:45.969 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 sudo[70019]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:46.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 sudo[70022]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-23T00:00:46.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 sudo[70022]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-23T00:00:46.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 sudo[70022]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-23T00:00:46.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 sudo[70022]: pam_unix(sudo:session): session closed for user root 2024-05-23T00:00:46.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 190 KiB/s wr, 59 op/s 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:46.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:46.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:46.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 190 KiB/s wr, 59 op/s 2024-05-23T00:00:46.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:46.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:46.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:46.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.331 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:46.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:46.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.332 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:46.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='client.25639 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:46.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:00:46.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 190 KiB/s wr, 59 op/s 2024-05-23T00:00:46.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:46.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-23T00:00:46.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-23T00:00:47.270 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:46 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:47.270 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:46 smithi174 ceph-mon[36179]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:47.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:46 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:47.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:46 smithi105 ceph-mon[36426]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:47.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:46 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-23T00:00:47.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:46 smithi103 ceph-mon[31370]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-23T00:00:48.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:47 smithi105 ceph-mon[36426]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-05-23T00:00:48.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:47 smithi103 ceph-mon[31370]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-05-23T00:00:48.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:47 smithi174 ceph-mon[36179]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 174 KiB/s wr, 54 op/s 2024-05-23T00:00:50.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:49 smithi105 ceph-mon[36426]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 197 KiB/s wr, 58 op/s 2024-05-23T00:00:50.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:49 smithi103 ceph-mon[31370]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 197 KiB/s wr, 58 op/s 2024-05-23T00:00:50.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:49 smithi174 ceph-mon[36179]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 197 KiB/s wr, 58 op/s 2024-05-23T00:00:52.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:52 smithi105 ceph-mon[36426]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 99 KiB/s wr, 40 op/s 2024-05-23T00:00:52.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:52 smithi105 ceph-mon[36426]: scrub summary: idle 2024-05-23T00:00:52.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:52 smithi103 ceph-mon[31370]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 99 KiB/s wr, 40 op/s 2024-05-23T00:00:52.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:52 smithi103 ceph-mon[31370]: scrub summary: idle 2024-05-23T00:00:52.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:52 smithi174 ceph-mon[36179]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 99 KiB/s wr, 40 op/s 2024-05-23T00:00:52.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:52 smithi174 ceph-mon[36179]: scrub summary: idle 2024-05-23T00:00:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:54 smithi105 ceph-mon[36426]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 99 KiB/s wr, 39 op/s 2024-05-23T00:00:54.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:54 smithi103 ceph-mon[31370]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 99 KiB/s wr, 39 op/s 2024-05-23T00:00:54.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:54 smithi174 ceph-mon[36179]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 99 KiB/s wr, 39 op/s 2024-05-23T00:00:56.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:56 smithi105 ceph-mon[36426]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 99 KiB/s wr, 41 op/s 2024-05-23T00:00:56.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:56 smithi103 ceph-mon[31370]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 99 KiB/s wr, 41 op/s 2024-05-23T00:00:56.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:56 smithi174 ceph-mon[36179]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 99 KiB/s wr, 41 op/s 2024-05-23T00:00:57.449 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 -- 172.21.15.103:0/2836001543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188107890 msgr2=0x7f8188107c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 --2- 172.21.15.103:0/2836001543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188107890 0x7f8188107c70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8178009900 tx=0x7f817802a3f0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 -- 172.21.15.103:0/2836001543 shutdown_connections 2024-05-23T00:00:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 --2- 172.21.15.103:0/2836001543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188102240 0x7f818810e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 --2- 172.21.15.103:0/2836001543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101880 0x7f8188101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.450 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 --2- 172.21.15.103:0/2836001543 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188107890 0x7f8188107c70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 -- 172.21.15.103:0/2836001543 >> 172.21.15.103:0/2836001543 conn(0x7f8188078d00 msgr2=0x7f8188079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 -- 172.21.15.103:0/2836001543 shutdown_connections 2024-05-23T00:00:57.454 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.447+0000 7f818d400640 1 -- 172.21.15.103:0/2836001543 wait complete. 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.448+0000 7f818d400640 1 Processor -- start 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.448+0000 7f818d400640 1 -- start start 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f818d400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101880 0x7f8188198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f818d400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188102240 0x7f81881985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f818d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188107890 0x7f818819c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f818d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8188111c60 con 0x7f8188107890 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f818d400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8188111ae0 con 0x7f8188102240 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f818d400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8188111de0 con 0x7f8188101880 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f8187200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188107890 0x7f818819c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f8185e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188102240 0x7f81881985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f8187200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188107890 0x7f818819c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53110/0 (socket says 172.21.15.103:53110) 2024-05-23T00:00:57.456 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f8187200640 1 -- 172.21.15.103:0/140986324 learned_addr learned my addr 172.21.15.103:0/140986324 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.449+0000 7f8186800640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101880 0x7f8188198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f8187200640 1 -- 172.21.15.103:0/140986324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101880 msgr2=0x7f8188198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f8187200640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101880 0x7f8188198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f8187200640 1 -- 172.21.15.103:0/140986324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188102240 msgr2=0x7f81881985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f8187200640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188102240 0x7f81881985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f8187200640 1 -- 172.21.15.103:0/140986324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818819d050 con 0x7f8188107890 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f8185e00640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188102240 0x7f81881985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f8187200640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188107890 0x7f818819c920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f818000ea60 tx=0x7f818000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f817f400640 1 -- 172.21.15.103:0/140986324 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f818000ce40 con 0x7f8188107890 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f818d400640 1 -- 172.21.15.103:0/140986324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818819d340 con 0x7f8188107890 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f817f400640 1 -- 172.21.15.103:0/140986324 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8180005020 con 0x7f8188107890 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.450+0000 7f817f400640 1 -- 172.21.15.103:0/140986324 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8180010690 con 0x7f8188107890 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.451+0000 7f818d400640 1 -- 172.21.15.103:0/140986324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81881a4bd0 con 0x7f8188107890 2024-05-23T00:00:57.457 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.453+0000 7f8190213740 1 Processor -- start 2024-05-23T00:00:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.454+0000 7f817f400640 1 -- 172.21.15.103:0/140986324 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 25) ==== 138010+0+0 (secure 0 0 0) 0x7f8180005520 con 0x7f8188107890 2024-05-23T00:00:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.454+0000 7f8190213740 1 -- start start 2024-05-23T00:00:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8190213740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b7fd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.463 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8190213740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fd670 0x555e9b807800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8190213740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b807d40 0x555e9b80a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8190213740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555e9b7612e0 con 0x555e9b7fd670 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8190213740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555e9b708e50 con 0x555e9b7fcd50 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8190213740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555e9b75d2a0 con 0x555e9b807d40 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8186800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fd670 0x555e9b807800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8186800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fd670 0x555e9b807800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53120/0 (socket says 172.21.15.103:53120) 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8185e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b7fd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8186800640 1 -- 172.21.15.103:0/1983427314 learned_addr learned my addr 172.21.15.103:0/1983427314 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8185e00640 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b807d40 0x555e9b80a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8186800640 1 -- 172.21.15.103:0/1983427314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b807d40 msgr2=0x555e9b80a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8186800640 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b807d40 0x555e9b80a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8186800640 1 -- 172.21.15.103:0/1983427314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 msgr2=0x555e9b7fd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8186800640 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b7fd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.455+0000 7f8186800640 1 -- 172.21.15.103:0/1983427314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e9b475210 con 0x555e9b7fd670 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.456+0000 7f8185e00640 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b7fd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.456+0000 7f817f400640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f816c0a2e40 0x7f816c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.456+0000 7f817f400640 1 -- 172.21.15.103:0/140986324 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f81800cfde0 con 0x7f8188107890 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.456+0000 7f8186800640 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fd670 0x555e9b807800 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f817802a910 tx=0x7f8178005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf4d4f907c149eb server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:57.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.456+0000 7f8185e00640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f816c0a2e40 0x7f816c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.456+0000 7f8173400640 1 -- 172.21.15.103:0/1983427314 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178035030 con 0x555e9b7fd670 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.456+0000 7f8173400640 1 -- 172.21.15.103:0/1983427314 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8178040070 con 0x555e9b7fd670 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.456+0000 7f8173400640 1 -- 172.21.15.103:0/1983427314 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81780318a0 con 0x555e9b7fd670 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 -- 172.21.15.103:0/1983427314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fd670 msgr2=0x555e9b807800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8185e00640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f816c0a2e40 0x7f816c0a5300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x555e9b7fa200 tx=0x7f817400bb30 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fd670 0x555e9b807800 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f817802a910 tx=0x7f8178005e80 comp rx=0 tx=0).stop 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 -- 172.21.15.103:0/1983427314 shutdown_connections 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b807d40 0x555e9b80a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fd670 0x555e9b807800 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 --2- 172.21.15.103:0/1983427314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b7fd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 -- 172.21.15.103:0/1983427314 >> 172.21.15.103:0/1983427314 conn(0x555e9b7f9070 msgr2=0x555e9b7f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 -- 172.21.15.103:0/1983427314 shutdown_connections 2024-05-23T00:00:57.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.457+0000 7f8190213740 1 -- 172.21.15.103:0/1983427314 wait complete. 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.458+0000 7f8190213740 1 -- start start 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.458+0000 7f8190213740 1 Processor -- start 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.458+0000 7f8190213740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b804c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.458+0000 7f8190213740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b8051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.458+0000 7f8190213740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8056e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.458+0000 7f8190213740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555e9b80ab70 con 0x555e9b807d40 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.458+0000 7f8190213740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x555e9b71c210 con 0x555e9b7fcd50 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.458+0000 7f8190213740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x555e9b80acf0 con 0x555e9b7fd670 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8190213740 10 client.0 _ll_register_callbacks cb 0x555e9b6ff140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b804c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b804c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:60554/0 (socket says 172.21.15.103:60554) 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 -- 172.21.15.103:0/240018675 learned_addr learned my addr 172.21.15.103:0/240018675 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:57.467 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8187200640 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8056e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8185e00640 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b8051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 -- 172.21.15.103:0/240018675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 msgr2=0x555e9b8051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b8051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 -- 172.21.15.103:0/240018675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 msgr2=0x555e9b8056e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8056e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 -- 172.21.15.103:0/240018675 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e9b541620 con 0x555e9b7fcd50 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8186800640 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b804c60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8178000c00 tx=0x7f8178004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7521a1ae11ada92 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.459+0000 7f8167e00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.460+0000 7f8167e00640 1 -- 172.21.15.103:0/240018675 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8178035070 con 0x555e9b7fcd50 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.460+0000 7f8190213740 1 -- 172.21.15.103:0/240018675 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8178040070 con 0x555e9b7fcd50 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.460+0000 7f8167e00640 1 -- 172.21.15.103:0/240018675 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8178003800 con 0x555e9b7fcd50 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.460+0000 7f8167e00640 1 -- 172.21.15.103:0/240018675 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8178056050 con 0x555e9b7fcd50 2024-05-23T00:00:57.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.460+0000 7f8190213740 10 client.25649 fetch_fsmap learned FSMap version 25 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.460+0000 7f8190213740 1 -- 172.21.15.103:0/240018675 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555e9b816db0 con 0x555e9b7fcd50 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.460+0000 7f8167e00640 1 -- 172.21.15.103:0/240018675 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8178040070 con 0x555e9b7fcd50 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.460+0000 7f8167e00640 1 -- 172.21.15.103:0/240018675 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f8178004910 con 0x555e9b7fcd50 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.461+0000 7f8190213740 10 client.25649 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.461+0000 7f8190213740 10 client.25649 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.461+0000 7f8190213740 20 client.25649 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.461+0000 7f8190213740 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x555e9b818100 0x555e9b81a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.461+0000 7f8190213740 4 client.25649 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.461+0000 7f8190213740 1 -- 172.21.15.103:0/240018675 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555e9b774ec0 con 0x555e9b818100 2024-05-23T00:00:57.469 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.461+0000 7f8185e00640 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x555e9b818100 0x555e9b81a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.462+0000 7f8185e00640 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x555e9b818100 0x555e9b81a4f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=901cb7d45a61404b server_cookie=c98e333d075665de in_seq=0 out_seq=0 2024-05-23T00:00:57.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.462+0000 7f8167e00640 10 client.25649 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.462+0000 7f8167e00640 1 -- 172.21.15.103:0/240018675 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555e9b774ec0 con 0x555e9b818100 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.462+0000 7f8167e00640 10 client.25649 handle_command_reply: tid=0 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.462+0000 7f8190213740 1 client.25649 shutdown 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 -- 172.21.15.103:0/240018675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 msgr2=0x555e9b804c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b804c60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8178000c00 tx=0x7f8178004530 comp rx=0 tx=0).stop 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 -- 172.21.15.103:0/240018675 shutdown_connections 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x555e9b818100 0x555e9b81a4f0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8056e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8167e00640 0 client.25649 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b8051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 --2- 172.21.15.103:0/240018675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b804c60 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 -- 172.21.15.103:0/240018675 >> 172.21.15.103:0/240018675 conn(0x555e9b7f9070 msgr2=0x555e9b7fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 -- 172.21.15.103:0/240018675 shutdown_connections 2024-05-23T00:00:57.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 1 -- 172.21.15.103:0/240018675 wait complete. 2024-05-23T00:00:57.472 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.463+0000 7f8190213740 20 client.25649 trim_cache size 0 max 16384 2024-05-23T00:00:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.479+0000 7f8190213740 1 Processor -- start 2024-05-23T00:00:57.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.479+0000 7f8190213740 1 -- start start 2024-05-23T00:00:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.479+0000 7f8190213740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fcd50 0x555e9b813a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.479+0000 7f8190213740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b81fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.479+0000 7f8190213740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b807d40 0x555e9b81ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.482 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.479+0000 7f8190213740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81881b1390 con 0x555e9b7fcd50 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.479+0000 7f8190213740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f81881aec60 con 0x555e9b807d40 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.479+0000 7f8190213740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81881acab0 con 0x555e9b7fd670 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b81fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b81fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:42182/0 (socket says 172.21.15.103:42182) 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 -- 172.21.15.103:0/3222502466 learned_addr learned my addr 172.21.15.103:0/3222502466 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8186800640 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fcd50 0x555e9b813a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8187200640 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b807d40 0x555e9b81ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 -- 172.21.15.103:0/3222502466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b807d40 msgr2=0x555e9b81ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b807d40 0x555e9b81ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 -- 172.21.15.103:0/3222502466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fcd50 msgr2=0x555e9b813a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fcd50 0x555e9b813a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 -- 172.21.15.103:0/3222502466 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555e9b80a4a0 con 0x555e9b7fd670 2024-05-23T00:00:57.483 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.480+0000 7f8185e00640 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b81fa90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f81880fe840 tx=0x7f8174014070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=599b909f81facf1e server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.481+0000 7f8172a00640 1 -- 172.21.15.103:0/3222502466 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817402bb50 con 0x555e9b7fd670 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.481+0000 7f8172a00640 1 -- 172.21.15.103:0/3222502466 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f817402bcf0 con 0x555e9b7fd670 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.481+0000 7f8172a00640 1 -- 172.21.15.103:0/3222502466 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81740227f0 con 0x555e9b7fd670 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.481+0000 7f8190213740 1 -- 172.21.15.103:0/3222502466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 msgr2=0x555e9b81fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b81fa90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f81880fe840 tx=0x7f8174014070 comp rx=0 tx=0).stop 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 -- 172.21.15.103:0/3222502466 shutdown_connections 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b807d40 0x555e9b81ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b81fa90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 --2- 172.21.15.103:0/3222502466 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b7fcd50 0x555e9b813a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.486 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 -- 172.21.15.103:0/3222502466 >> 172.21.15.103:0/3222502466 conn(0x555e9b7f9070 msgr2=0x555e9b7fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 -- 172.21.15.103:0/3222502466 shutdown_connections 2024-05-23T00:00:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 -- 172.21.15.103:0/3222502466 wait complete. 2024-05-23T00:00:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 -- start start 2024-05-23T00:00:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.482+0000 7f8190213740 1 Processor -- start 2024-05-23T00:00:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8190213740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b801a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8190213740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b801f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.487 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8190213740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8024a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8190213740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81881ad120 con 0x555e9b807d40 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8190213740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f81881ae590 con 0x555e9b7fcd50 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8190213740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81881ad7f0 con 0x555e9b7fd670 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8190213740 10 client.0 _ll_register_callbacks cb 0x7f8178005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8187200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8185e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b801f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8187200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8024a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:53144/0 (socket says 172.21.15.103:53144) 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8187200640 1 -- 172.21.15.103:0/1208146760 learned_addr learned my addr 172.21.15.103:0/1208146760 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:00:57.488 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8186800640 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b801a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8185e00640 1 -- 172.21.15.103:0/1208146760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 msgr2=0x555e9b801a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8185e00640 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b801a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8185e00640 1 -- 172.21.15.103:0/1208146760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 msgr2=0x555e9b8024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8185e00640 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8185e00640 1 -- 172.21.15.103:0/1208146760 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555e9b885c10 con 0x555e9b7fd670 2024-05-23T00:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.483+0000 7f8187200640 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.484+0000 7f8185e00640 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b801f60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x555e9b7f4910 tx=0x7f817400bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ebb8a3799f15fd6 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:00:57.489 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.484+0000 7f8167e00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:00:57.490 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.484+0000 7f8167e00640 1 -- 172.21.15.103:0/1208146760 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817402b340 con 0x555e9b7fd670 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.484+0000 7f8190213740 1 -- 172.21.15.103:0/1208146760 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f817402bcf0 con 0x555e9b7fd670 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.484+0000 7f8167e00640 1 -- 172.21.15.103:0/1208146760 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f8174030040 con 0x555e9b7fd670 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.484+0000 7f8167e00640 1 -- 172.21.15.103:0/1208146760 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8174042050 con 0x555e9b7fd670 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.484+0000 7f8190213740 10 client.25666 fetch_fsmap learned FSMap version 25 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.484+0000 7f8190213740 1 -- 172.21.15.103:0/1208146760 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x555e9b886f20 con 0x555e9b7fd670 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8167e00640 1 -- 172.21.15.103:0/1208146760 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f817402bcf0 con 0x555e9b7fd670 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8167e00640 1 -- 172.21.15.103:0/1208146760 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f81740227f0 con 0x555e9b7fd670 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8190213740 10 client.25666 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8190213740 10 client.25666 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8190213740 20 client.25666 populate_metadata read hostname 'smithi103' 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8190213740 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x555e9b887610 0x555e9b889a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8190213740 4 client.25666 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:00:57.491 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8190213740 1 -- 172.21.15.103:0/1208146760 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555e9b76ea10 con 0x555e9b887610 2024-05-23T00:00:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.485+0000 7f8186800640 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x555e9b887610 0x555e9b889a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:00:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.486+0000 7f8186800640 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x555e9b887610 0x555e9b889a30 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86cb5b310674c4f9 server_cookie=2073aadd91df611d in_seq=0 out_seq=0 2024-05-23T00:00:57.492 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.486+0000 7f8167e00640 10 client.25666 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:57.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8167e00640 1 -- 172.21.15.103:0/1208146760 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555e9b76ea10 con 0x555e9b887610 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8167e00640 10 client.25666 handle_command_reply: tid=0 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 client.25666 shutdown 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 -- 172.21.15.103:0/1208146760 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 msgr2=0x555e9b801f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b801f60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x555e9b7f4910 tx=0x7f817400bef0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 -- 172.21.15.103:0/1208146760 shutdown_connections 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x555e9b887610 0x555e9b889a30 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555e9b807d40 0x555e9b8024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x555e9b7fd670 0x555e9b801f60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 --2- 172.21.15.103:0/1208146760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x555e9b7fcd50 0x555e9b801a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 -- 172.21.15.103:0/1208146760 >> 172.21.15.103:0/1208146760 conn(0x555e9b7f9070 msgr2=0x555e9b7f7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8167e00640 0 client.25666 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:00:57.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 -- 172.21.15.103:0/1208146760 shutdown_connections 2024-05-23T00:00:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.500+0000 7f8190213740 1 -- 172.21.15.103:0/1208146760 wait complete. 2024-05-23T00:00:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f8190213740 20 client.25666 trim_cache size 0 max 16384 2024-05-23T00:00:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:00:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:00:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:00:57.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 -- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f816c0a2e40 msgr2=0x7f816c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f816c0a2e40 0x7f816c0a5300 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x555e9b7fa200 tx=0x7f817400bb30 comp rx=0 tx=0).stop 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 -- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188107890 msgr2=0x7f818819c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188107890 0x7f818819c920 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f818000ea60 tx=0x7f818000ef20 comp rx=0 tx=0).stop 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 -- 172.21.15.103:0/140986324 shutdown_connections 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f816c0a2e40 0x7f816c0a5300 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8188107890 0x7f818819c920 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8188102240 0x7f81881985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 --2- 172.21.15.103:0/140986324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8188101880 0x7f8188198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:00:57.505 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 -- 172.21.15.103:0/140986324 >> 172.21.15.103:0/140986324 conn(0x7f8188078d00 msgr2=0x7f81880ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:00:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 -- 172.21.15.103:0/140986324 shutdown_connections 2024-05-23T00:00:57.506 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:00:57.501+0000 7f818d400640 1 -- 172.21.15.103:0/140986324 wait complete. 2024-05-23T00:00:58.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:00:58 smithi105 ceph-mon[36426]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-05-23T00:00:58.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:00:58 smithi103 ceph-mon[31370]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-05-23T00:00:58.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:00:58 smithi174 ceph-mon[36179]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 23 KiB/s wr, 8 op/s 2024-05-23T00:01:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:00 smithi105 ceph-mon[36426]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-05-23T00:01:00.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:00 smithi103 ceph-mon[31370]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-05-23T00:01:00.330 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:01:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:01:00.177+0000 7fd806400700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:01:00.330 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:01:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:01:00.177+0000 7fd806400700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:01:00.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:01:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:01:00.177+0000 7fd806400700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:01:00.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:01:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:01:00.177+0000 7fd806400700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:01:00.331 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:01:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:01:00.177+0000 7fd806400700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-23T00:01:00.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:00 smithi174 ceph-mon[36179]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 24 KiB/s wr, 14 op/s 2024-05-23T00:01:02.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:02 smithi105 ceph-mon[36426]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-23T00:01:02.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:02 smithi105 ceph-mon[36426]: mgrmap e26: x(active, since 28s), standbys: z, y 2024-05-23T00:01:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:02 smithi103 ceph-mon[31370]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-23T00:01:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:02 smithi103 ceph-mon[31370]: mgrmap e26: x(active, since 28s), standbys: z, y 2024-05-23T00:01:02.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:02 smithi174 ceph-mon[36179]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 2.0 KiB/s wr, 11 op/s 2024-05-23T00:01:02.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:02 smithi174 ceph-mon[36179]: mgrmap e26: x(active, since 28s), standbys: z, y 2024-05-23T00:01:04.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:04 smithi105 ceph-mon[36426]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-05-23T00:01:04.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:04 smithi103 ceph-mon[31370]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-05-23T00:01:04.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:04 smithi174 ceph-mon[36179]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.8 KiB/s wr, 9 op/s 2024-05-23T00:01:06.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:06 smithi105 ceph-mon[36426]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-05-23T00:01:06.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:06 smithi103 ceph-mon[31370]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-05-23T00:01:06.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:06 smithi174 ceph-mon[36179]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-05-23T00:01:08.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:08 smithi105 ceph-mon[36426]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-05-23T00:01:08.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:08 smithi103 ceph-mon[31370]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-05-23T00:01:08.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:08 smithi174 ceph-mon[36179]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 25 op/s 2024-05-23T00:01:10.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:10 smithi103 ceph-mon[31370]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-05-23T00:01:10.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:10 smithi174 ceph-mon[36179]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-05-23T00:01:10.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:10 smithi105 ceph-mon[36426]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 27 op/s 2024-05-23T00:01:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.641+0000 7fc383e00640 1 -- 172.21.15.103:0/2828529440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc384075da0 msgr2=0x7fc384076180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.641+0000 7fc383e00640 1 --2- 172.21.15.103:0/2828529440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc384075da0 0x7fc384076180 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc370009890 tx=0x7fc37002a360 comp rx=0 tx=0).stop 2024-05-23T00:01:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.641+0000 7fc383e00640 1 -- 172.21.15.103:0/2828529440 shutdown_connections 2024-05-23T00:01:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.641+0000 7fc383e00640 1 --2- 172.21.15.103:0/2828529440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc384112540 0x7fc384114930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.641+0000 7fc383e00640 1 --2- 172.21.15.103:0/2828529440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc3840766c0 0x7fc384112000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.641+0000 7fc383e00640 1 --2- 172.21.15.103:0/2828529440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc384075da0 0x7fc384076180 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.645 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.641+0000 7fc383e00640 1 -- 172.21.15.103:0/2828529440 >> 172.21.15.103:0/2828529440 conn(0x7fc3840ff350 msgr2=0x7fc384101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.642+0000 7fc383e00640 1 -- 172.21.15.103:0/2828529440 shutdown_connections 2024-05-23T00:01:10.646 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.642+0000 7fc383e00640 1 -- 172.21.15.103:0/2828529440 wait complete. 2024-05-23T00:01:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.643+0000 7fc383e00640 1 Processor -- start 2024-05-23T00:01:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.643+0000 7fc383e00640 1 -- start start 2024-05-23T00:01:10.647 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.643+0000 7fc383e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc384075da0 0x7fc38419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.643+0000 7fc383e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3840766c0 0x7fc38419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.643+0000 7fc383e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc384112540 0x7fc3841a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.643+0000 7fc383e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841180a0 con 0x7fc384075da0 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.643+0000 7fc383e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc384117f20 con 0x7fc384112540 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.643+0000 7fc383e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc384118220 con 0x7fc3840766c0 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc384075da0 0x7fc38419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc384075da0 0x7fc38419e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41234/0 (socket says 172.21.15.103:41234) 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382a00640 1 -- 172.21.15.103:0/3350288500 learned_addr learned my addr 172.21.15.103:0/3350288500 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc383400640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc384112540 0x7fc3841a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382000640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3840766c0 0x7fc38419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382a00640 1 -- 172.21.15.103:0/3350288500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3840766c0 msgr2=0x7fc38419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382a00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3840766c0 0x7fc38419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382a00640 1 -- 172.21.15.103:0/3350288500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc384112540 msgr2=0x7fc3841a2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.648 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382a00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc384112540 0x7fc3841a2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.644+0000 7fc382a00640 1 -- 172.21.15.103:0/3350288500 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3841a3650 con 0x7fc384075da0 2024-05-23T00:01:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.645+0000 7fc382a00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc384075da0 0x7fc38419e660 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc370009860 tx=0x7fc370033d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:10.649 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.645+0000 7fc388a00640 1 -- 172.21.15.103:0/3350288500 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37003f020 con 0x7fc384075da0 2024-05-23T00:01:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.645+0000 7fc388a00640 1 -- 172.21.15.103:0/3350288500 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc370032770 con 0x7fc384075da0 2024-05-23T00:01:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.645+0000 7fc383e00640 1 -- 172.21.15.103:0/3350288500 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3841a38e0 con 0x7fc384075da0 2024-05-23T00:01:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.645+0000 7fc388a00640 1 -- 172.21.15.103:0/3350288500 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc370037040 con 0x7fc384075da0 2024-05-23T00:01:10.650 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.645+0000 7fc383e00640 1 -- 172.21.15.103:0/3350288500 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3841ab1d0 con 0x7fc384075da0 2024-05-23T00:01:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.649+0000 7fc388a00640 1 -- 172.21.15.103:0/3350288500 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7fc370042020 con 0x7fc384075da0 2024-05-23T00:01:10.652 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.649+0000 7fc38cbee740 1 Processor -- start 2024-05-23T00:01:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.649+0000 7fc38cbee740 1 -- start start 2024-05-23T00:01:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38c2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc38cbee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5566f38192e0 con 0x5566f38bff20 2024-05-23T00:01:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc38cbee740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5566f37c0e50 con 0x5566f38b5850 2024-05-23T00:01:10.653 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc38cbee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5566f38152a0 con 0x5566f38b4f30 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc382000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38c2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc383400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc382000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38c2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41236/0 (socket says 172.21.15.103:41236) 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc382000640 1 -- 172.21.15.103:0/906931688 learned_addr learned my addr 172.21.15.103:0/906931688 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.650+0000 7fc382a00640 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc383400640 1 -- 172.21.15.103:0/906931688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 msgr2=0x5566f38bf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc383400640 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc383400640 1 -- 172.21.15.103:0/906931688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 msgr2=0x5566f38c2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:10.656 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc383400640 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38c2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc383400640 1 -- 172.21.15.103:0/906931688 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566f352d210 con 0x5566f38b4f30 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc382000640 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38c2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc388a00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc3680a2e10 0x7fc3680a52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc383400640 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38b5310 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc37c005fd0 tx=0x7fc37c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=635247e7d099016 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc388a00640 1 -- 172.21.15.103:0/3350288500 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc3700ef920 con 0x7fc384075da0 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc382a00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc3680a2e10 0x7fc3680a52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc36fe00640 1 -- 172.21.15.103:0/906931688 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c004d20 con 0x5566f38b4f30 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc36fe00640 1 -- 172.21.15.103:0/906931688 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c004050 con 0x5566f38b4f30 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.651+0000 7fc36fe00640 1 -- 172.21.15.103:0/906931688 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c005280 con 0x5566f38b4f30 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 -- 172.21.15.103:0/906931688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 msgr2=0x5566f38b5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38b5310 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc37c005fd0 tx=0x7fc37c011810 comp rx=0 tx=0).stop 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc382a00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc3680a2e10 0x7fc3680a52d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc3700ecd20 tx=0x7fc3700f5480 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:10.657 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 -- 172.21.15.103:0/906931688 shutdown_connections 2024-05-23T00:01:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38c2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 --2- 172.21.15.103:0/906931688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38b5310 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 -- 172.21.15.103:0/906931688 >> 172.21.15.103:0/906931688 conn(0x5566f38b1250 msgr2=0x5566f38b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 -- 172.21.15.103:0/906931688 shutdown_connections 2024-05-23T00:01:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.652+0000 7fc38cbee740 1 -- 172.21.15.103:0/906931688 wait complete. 2024-05-23T00:01:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.653+0000 7fc38cbee740 1 -- start start 2024-05-23T00:01:10.658 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.653+0000 7fc38cbee740 1 Processor -- start 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.653+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38bce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.653+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.653+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38bd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc38cbee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5566f38c2db0 con 0x5566f38bff20 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc38cbee740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5566f37d4210 con 0x5566f38b5850 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc38cbee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5566f38c2f30 con 0x5566f38b4f30 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc38cbee740 10 client.0 _ll_register_callbacks cb 0x5566f37b7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc383400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38bd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.662 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc382000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc383400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38bd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41248/0 (socket says 172.21.15.103:41248) 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc383400640 1 -- 172.21.15.103:0/3215333125 learned_addr learned my addr 172.21.15.103:0/3215333125 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc383400640 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38bce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc382000640 1 -- 172.21.15.103:0/3215333125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 msgr2=0x5566f38bce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc382000640 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38bce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc382000640 1 -- 172.21.15.103:0/3215333125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 msgr2=0x5566f38bd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc382000640 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38bd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.654+0000 7fc382000640 1 -- 172.21.15.103:0/3215333125 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566f35f9620 con 0x5566f38b5850 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.655+0000 7fc382000640 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bd3b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc3740116c0 tx=0x7fc374011b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e2b43071bd00055 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.655+0000 7fc36cc00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.655+0000 7fc36cc00640 1 -- 172.21.15.103:0/3215333125 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37401e020 con 0x5566f38b5850 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.655+0000 7fc38cbee740 1 -- 172.21.15.103:0/3215333125 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc37c004050 con 0x5566f38b5850 2024-05-23T00:01:10.663 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.655+0000 7fc36cc00640 1 -- 172.21.15.103:0/3215333125 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc37400fb90 con 0x5566f38b5850 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.655+0000 7fc36cc00640 1 -- 172.21.15.103:0/3215333125 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc374026050 con 0x5566f38b5850 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.655+0000 7fc38cbee740 10 client.25673 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.655+0000 7fc38cbee740 1 -- 172.21.15.103:0/3215333125 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5566f38cf0e0 con 0x5566f38b5850 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc36cc00640 1 -- 172.21.15.103:0/3215333125 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c004050 con 0x5566f38b5850 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc36cc00640 1 -- 172.21.15.103:0/3215333125 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc374019070 con 0x5566f38b5850 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc38cbee740 10 client.25673 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc38cbee740 10 client.25673 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc38cbee740 20 client.25673 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5566f38d05a0 0x5566f38d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc38cbee740 4 client.25673 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:10.664 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc38cbee740 1 -- 172.21.15.103:0/3215333125 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566f382cec0 con 0x5566f38d05a0 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.656+0000 7fc383400640 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5566f38d05a0 0x5566f38d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.657+0000 7fc383400640 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5566f38d05a0 0x5566f38d2990 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4605deaf3ed2d24 server_cookie=af8fbb02cc415924 in_seq=0 out_seq=0 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.657+0000 7fc36cc00640 10 client.25673 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.657+0000 7fc36cc00640 1 -- 172.21.15.103:0/3215333125 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566f382cec0 con 0x5566f38d05a0 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.657+0000 7fc36cc00640 10 client.25673 handle_command_reply: tid=0 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.657+0000 7fc38cbee740 1 client.25673 shutdown 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 -- 172.21.15.103:0/3215333125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 msgr2=0x5566f38bd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bd3b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc3740116c0 tx=0x7fc374011b80 comp rx=0 tx=0).stop 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 -- 172.21.15.103:0/3215333125 shutdown_connections 2024-05-23T00:01:10.665 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5566f38d05a0 0x5566f38d2990 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38bff20 0x5566f38bd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38b5850 0x5566f38bd3b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc36cc00640 0 client.25673 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3215333125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38bce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 -- 172.21.15.103:0/3215333125 >> 172.21.15.103:0/3215333125 conn(0x5566f38b1250 msgr2=0x5566f38b7050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 -- 172.21.15.103:0/3215333125 shutdown_connections 2024-05-23T00:01:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 1 -- 172.21.15.103:0/3215333125 wait complete. 2024-05-23T00:01:10.666 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.658+0000 7fc38cbee740 20 client.25673 trim_cache size 0 max 16384 2024-05-23T00:01:10.678 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.675+0000 7fc38cbee740 1 Processor -- start 2024-05-23T00:01:10.678 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.675+0000 7fc38cbee740 1 -- start start 2024-05-23T00:01:10.678 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.675+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.678 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.675+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b5850 0x5566f38bd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.675+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 0x5566f38b0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.675+0000 7fc38cbee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841b78f0 con 0x5566f38b5850 2024-05-23T00:01:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.675+0000 7fc38cbee740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841b51c0 con 0x5566f38bff20 2024-05-23T00:01:10.679 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.675+0000 7fc38cbee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841b3010 con 0x5566f38b4f30 2024-05-23T00:01:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:54992/0 (socket says 172.21.15.103:54992) 2024-05-23T00:01:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 -- 172.21.15.103:0/3796289943 learned_addr learned my addr 172.21.15.103:0/3796289943 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc382000640 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b5850 0x5566f38bd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 0x5566f38b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 -- 172.21.15.103:0/3796289943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 msgr2=0x5566f38b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 0x5566f38b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.680 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 -- 172.21.15.103:0/3796289943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b5850 msgr2=0x5566f38bd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:10.681 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b5850 0x5566f38bd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.681 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 -- 172.21.15.103:0/3796289943 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566f38d2d00 con 0x5566f38b4f30 2024-05-23T00:01:10.681 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.676+0000 7fc383400640 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38cbca0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc37c005fd0 tx=0x7fc37c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94dbe7320ee1ceb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:10.681 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.677+0000 7fc36f400640 1 -- 172.21.15.103:0/3796289943 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c02f070 con 0x5566f38b4f30 2024-05-23T00:01:10.681 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.677+0000 7fc36f400640 1 -- 172.21.15.103:0/3796289943 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c004520 con 0x5566f38b4f30 2024-05-23T00:01:10.681 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.677+0000 7fc36f400640 1 -- 172.21.15.103:0/3796289943 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c028a30 con 0x5566f38b4f30 2024-05-23T00:01:10.682 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.677+0000 7fc38cbee740 1 -- 172.21.15.103:0/3796289943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 msgr2=0x5566f38cbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.677+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38cbca0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc37c005fd0 tx=0x7fc37c004a40 comp rx=0 tx=0).stop 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 -- 172.21.15.103:0/3796289943 shutdown_connections 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 0x5566f38b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b5850 0x5566f38bd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 --2- 172.21.15.103:0/3796289943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b4f30 0x5566f38cbca0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 -- 172.21.15.103:0/3796289943 >> 172.21.15.103:0/3796289943 conn(0x5566f38b1250 msgr2=0x5566f38afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 -- 172.21.15.103:0/3796289943 shutdown_connections 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 -- 172.21.15.103:0/3796289943 wait complete. 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 -- start start 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.678+0000 7fc38cbee740 1 Processor -- start 2024-05-23T00:01:10.683 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b4f30 0x5566f38c86f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b5850 0x5566f38b9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc38cbee740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 0x5566f38ba080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc38cbee740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841b3680 con 0x5566f38b4f30 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc38cbee740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841b4af0 con 0x5566f38bff20 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc38cbee740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc3841b3d50 con 0x5566f38b5850 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc38cbee740 10 client.0 _ll_register_callbacks cb 0x7fc374009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc383400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b4f30 0x5566f38c86f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc382000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b5850 0x5566f38b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc383400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b4f30 0x5566f38c86f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41268/0 (socket says 172.21.15.103:41268) 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc383400640 1 -- 172.21.15.103:0/2618147537 learned_addr learned my addr 172.21.15.103:0/2618147537 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:10.684 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.679+0000 7fc383400640 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 0x5566f38ba080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc383400640 1 -- 172.21.15.103:0/2618147537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b5850 msgr2=0x5566f38b9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc383400640 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b5850 0x5566f38b9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc383400640 1 -- 172.21.15.103:0/2618147537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 msgr2=0x5566f38ba080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc383400640 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 0x5566f38ba080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc383400640 1 -- 172.21.15.103:0/2618147537 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566f3935b10 con 0x5566f38b4f30 2024-05-23T00:01:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc383400640 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b4f30 0x5566f38c86f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009540 tx=0x7fc37c009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aae76543cfb78c81 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:10.686 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc36cc00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc36cc00640 1 -- 172.21.15.103:0/2618147537 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c02f070 con 0x5566f38b4f30 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc36cc00640 1 -- 172.21.15.103:0/2618147537 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc37c028450 con 0x5566f38b4f30 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.680+0000 7fc38cbee740 1 -- 172.21.15.103:0/2618147537 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc37c004520 con 0x5566f38b4f30 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc36cc00640 1 -- 172.21.15.103:0/2618147537 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc37c004760 con 0x5566f38b4f30 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc38cbee740 10 client.15573 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc38cbee740 1 -- 172.21.15.103:0/2618147537 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5566f3936c80 con 0x5566f38b4f30 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc36cc00640 1 -- 172.21.15.103:0/2618147537 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc37c004520 con 0x5566f38b4f30 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc36cc00640 1 -- 172.21.15.103:0/2618147537 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc37c03a690 con 0x5566f38b4f30 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc38cbee740 10 client.15573 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc38cbee740 10 client.15573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc38cbee740 20 client.15573 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc38cbee740 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5566f39370b0 0x5566f39394d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:10.687 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc38cbee740 4 client.15573 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:01:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.681+0000 7fc38cbee740 1 -- 172.21.15.103:0/2618147537 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5566f3826a10 con 0x5566f39370b0 2024-05-23T00:01:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.682+0000 7fc382000640 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5566f39370b0 0x5566f39394d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.682+0000 7fc382000640 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5566f39370b0 0x5566f39394d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6bc936b51f66ecd server_cookie=eb8e869a8617bbeb in_seq=0 out_seq=0 2024-05-23T00:01:10.688 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.682+0000 7fc36cc00640 10 client.15573 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:10.693 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.690+0000 7fc36cc00640 1 -- 172.21.15.103:0/2618147537 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5566f3826a10 con 0x5566f39370b0 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.690+0000 7fc36cc00640 10 client.15573 handle_command_reply: tid=0 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.690+0000 7fc38cbee740 1 client.15573 shutdown 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.690+0000 7fc38cbee740 1 -- 172.21.15.103:0/2618147537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b4f30 msgr2=0x5566f38c86f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.690+0000 7fc38cbee740 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b4f30 0x5566f38c86f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc37c009540 tx=0x7fc37c009230 comp rx=0 tx=0).stop 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 1 -- 172.21.15.103:0/2618147537 shutdown_connections 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5566f39370b0 0x5566f39394d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5566f38bff20 0x5566f38ba080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5566f38b5850 0x5566f38b9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 1 --2- 172.21.15.103:0/2618147537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5566f38b4f30 0x5566f38c86f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 1 -- 172.21.15.103:0/2618147537 >> 172.21.15.103:0/2618147537 conn(0x5566f38b1250 msgr2=0x5566f38b03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:10.694 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc36cc00640 0 client.15573 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 1 -- 172.21.15.103:0/2618147537 shutdown_connections 2024-05-23T00:01:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 1 -- 172.21.15.103:0/2618147537 wait complete. 2024-05-23T00:01:10.695 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.691+0000 7fc38cbee740 20 client.15573 trim_cache size 0 max 16384 2024-05-23T00:01:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:01:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:01:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:01:10.696 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 -- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc3680a2e10 msgr2=0x7fc3680a52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc3680a2e10 0x7fc3680a52d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc3700ecd20 tx=0x7fc3700f5480 comp rx=0 tx=0).stop 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 -- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc384075da0 msgr2=0x7fc38419e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc384075da0 0x7fc38419e660 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc370009860 tx=0x7fc370033d10 comp rx=0 tx=0).stop 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 -- 172.21.15.103:0/3350288500 shutdown_connections 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc3680a2e10 0x7fc3680a52d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc384112540 0x7fc3841a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3840766c0 0x7fc38419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.697 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 --2- 172.21.15.103:0/3350288500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc384075da0 0x7fc38419e660 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:10.698 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 -- 172.21.15.103:0/3350288500 >> 172.21.15.103:0/3350288500 conn(0x7fc3840ff350 msgr2=0x7fc3841131b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:10.698 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 -- 172.21.15.103:0/3350288500 shutdown_connections 2024-05-23T00:01:10.698 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:10.692+0000 7fc383e00640 1 -- 172.21.15.103:0/3350288500 wait complete. 2024-05-23T00:01:11.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-23T00:01:11.294 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-23T00:01:11.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 -- 172.21.15.103:0/4154332386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0108890 msgr2=0x7f9fe010c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/4154332386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0108890 0x7f9fe010c8c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009900 tx=0x7f9fc802a410 comp rx=0 tx=0).stop 2024-05-23T00:01:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 -- 172.21.15.103:0/4154332386 shutdown_connections 2024-05-23T00:01:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/4154332386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe010cfa0 0x7f9fe0114b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/4154332386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0108890 0x7f9fe010c8c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/4154332386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0107ee0 0x7f9fe01082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 -- 172.21.15.103:0/4154332386 >> 172.21.15.103:0/4154332386 conn(0x7f9fe0077480 msgr2=0x7f9fe00798a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 -- 172.21.15.103:0/4154332386 shutdown_connections 2024-05-23T00:01:11.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.515+0000 7f9fe4c00640 1 -- 172.21.15.103:0/4154332386 wait complete. 2024-05-23T00:01:11.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.516+0000 7f9fe4c00640 1 Processor -- start 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.516+0000 7f9fe4c00640 1 -- start start 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fe4c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0107ee0 0x7f9fe019e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fe4c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe010cfa0 0x7f9fe019eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fe4c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe01a2df0 0x7f9fe01a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fe4c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0118050 con 0x7f9fe0107ee0 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fe4c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0117ed0 con 0x7f9fe01a2df0 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fe4c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01181d0 con 0x7f9fe010cfa0 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0107ee0 0x7f9fe019e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0107ee0 0x7f9fe019e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41288/0 (socket says 172.21.15.103:41288) 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdf400640 1 -- 172.21.15.103:0/403552080 learned_addr learned my addr 172.21.15.103:0/403552080 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe010cfa0 0x7f9fe019eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe01a2df0 0x7f9fe01a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.522 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdf400640 1 -- 172.21.15.103:0/403552080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe010cfa0 msgr2=0x7f9fe019eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdf400640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe010cfa0 0x7f9fe019eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdf400640 1 -- 172.21.15.103:0/403552080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe01a2df0 msgr2=0x7f9fe01a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdf400640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe01a2df0 0x7f9fe01a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdf400640 1 -- 172.21.15.103:0/403552080 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe01a38e0 con 0x7f9fe0107ee0 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.517+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe01a2df0 0x7f9fe01a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.518+0000 7f9fdf400640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0107ee0 0x7f9fe019e5d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000e9f0 tx=0x7f9fd000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.518+0000 7f9fdcc00640 1 -- 172.21.15.103:0/403552080 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd000cd80 con 0x7f9fe0107ee0 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.518+0000 7f9fdcc00640 1 -- 172.21.15.103:0/403552080 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd0005500 con 0x7f9fe0107ee0 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.518+0000 7f9fdcc00640 1 -- 172.21.15.103:0/403552080 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0010690 con 0x7f9fe0107ee0 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.518+0000 7f9fe4c00640 1 -- 172.21.15.103:0/403552080 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe01a3bd0 con 0x7f9fe0107ee0 2024-05-23T00:01:11.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.518+0000 7f9fe4c00640 1 -- 172.21.15.103:0/403552080 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe01ab390 con 0x7f9fe0107ee0 2024-05-23T00:01:11.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fe904f740 1 Processor -- start 2024-05-23T00:01:11.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fdcc00640 1 -- 172.21.15.103:0/403552080 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f9fd00046c0 con 0x7f9fe0107ee0 2024-05-23T00:01:11.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fe904f740 1 -- start start 2024-05-23T00:01:11.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3123f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3130f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fe904f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5590d30e7320 con 0x5590d312eb50 2024-05-23T00:01:11.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fe904f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5590d3088cb0 con 0x5590d3124480 2024-05-23T00:01:11.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fe904f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5590d302ffc0 con 0x5590d3123b60 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.522+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fd7e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3130f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3123f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3123f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:55042/0 (socket says 172.21.15.103:55042) 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:41964/0 (socket says 172.21.15.103:41964) 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 -- 172.21.15.103:0/580966508 learned_addr learned my addr 172.21.15.103:0/580966508 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 -- 172.21.15.103:0/580966508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 msgr2=0x5590d312e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/580966508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 -- 172.21.15.103:0/580966508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 msgr2=0x5590d3130f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/580966508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3130f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.528 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 -- 172.21.15.103:0/580966508 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590d2e667d0 con 0x5590d3123b60 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/580966508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3130f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/580966508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3123f40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009f80 tx=0x7f9fd8011870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b425022102d4fe9 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fdcc00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2ea0 0x7f9fc40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.523+0000 7f9fcfe00640 1 -- 172.21.15.103:0/580966508 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8012710 con 0x5590d3123b60 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.524+0000 7f9fcfe00640 1 -- 172.21.15.103:0/580966508 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8012d90 con 0x5590d3123b60 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.524+0000 7f9fcfe00640 1 -- 172.21.15.103:0/580966508 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd801a3f0 con 0x5590d3123b60 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.524+0000 7f9fdcc00640 1 -- 172.21.15.103:0/403552080 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9fd00cfee0 con 0x7f9fe0107ee0 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.524+0000 7f9fdf400640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2ea0 0x7f9fc40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.524+0000 7f9fe904f740 1 -- 172.21.15.103:0/580966508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 msgr2=0x5590d3123f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.524+0000 7f9fe904f740 1 --2- 172.21.15.103:0/580966508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3123f40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009f80 tx=0x7f9fd8011870 comp rx=0 tx=0).stop 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.524+0000 7f9fdf400640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2ea0 0x7f9fc40a5360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9fd00cd170 tx=0x7f9fd00d3970 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 -- 172.21.15.103:0/580966508 shutdown_connections 2024-05-23T00:01:11.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 --2- 172.21.15.103:0/580966508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3130f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 --2- 172.21.15.103:0/580966508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 --2- 172.21.15.103:0/580966508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3123f40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 -- 172.21.15.103:0/580966508 >> 172.21.15.103:0/580966508 conn(0x5590d311fe80 msgr2=0x5590d3120260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:11.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 -- 172.21.15.103:0/580966508 shutdown_connections 2024-05-23T00:01:11.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 -- 172.21.15.103:0/580966508 wait complete. 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 -- start start 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.525+0000 7f9fe904f740 1 Processor -- start 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d3123b60 0x5590d312ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d312eb50 0x5590d312c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fe904f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5590d3043410 con 0x5590d3123b60 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fe904f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5590d3084c70 con 0x5590d3124480 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fe904f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5590d3131940 con 0x5590d312eb50 2024-05-23T00:01:11.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fe904f740 10 client.0 _ll_register_callbacks cb 0x5590d307f0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d3123b60 0x5590d312ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d3123b60 0x5590d312ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41306/0 (socket says 172.21.15.103:41306) 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fdfe00640 1 -- 172.21.15.103:0/924750342 learned_addr learned my addr 172.21.15.103:0/924750342 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.526+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d312eb50 0x5590d312c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fd7e00640 1 -- 172.21.15.103:0/924750342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d312eb50 msgr2=0x5590d312c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d312eb50 0x5590d312c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fd7e00640 1 -- 172.21.15.103:0/924750342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d3123b60 msgr2=0x5590d312ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d3123b60 0x5590d312ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fd7e00640 1 -- 172.21.15.103:0/924750342 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590d2d9c3f0 con 0x5590d3124480 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312bf90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8006010 tx=0x7f9fc8031850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24bfe8abd389ad11 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fccc00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fccc00640 1 -- 172.21.15.103:0/924750342 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8035040 con 0x5590d3124480 2024-05-23T00:01:11.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fe904f740 1 -- 172.21.15.103:0/924750342 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fd8012d90 con 0x5590d3124480 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.527+0000 7f9fccc00640 1 -- 172.21.15.103:0/924750342 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9fc8004240 con 0x5590d3124480 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fccc00640 1 -- 172.21.15.103:0/924750342 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9fc8004440 con 0x5590d3124480 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fe904f740 10 client.25691 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fe904f740 1 -- 172.21.15.103:0/924750342 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5590d2d5c0f0 con 0x5590d3124480 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fccc00640 1 -- 172.21.15.103:0/924750342 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8012d90 con 0x5590d3124480 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fccc00640 1 -- 172.21.15.103:0/924750342 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9fc8004c20 con 0x5590d3124480 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fe904f740 10 client.25691 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fe904f740 10 client.25691 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fe904f740 20 client.25691 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fe904f740 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5590d313ed00 0x5590d31410f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fe904f740 4 client.25691 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.528+0000 7f9fe904f740 1 -- 172.21.15.103:0/924750342 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590d3047e00 con 0x5590d313ed00 2024-05-23T00:01:11.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.529+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5590d313ed00 0x5590d31410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.529+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5590d313ed00 0x5590d31410f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbb03578ec503448 server_cookie=c2bbf303f7b2a365 in_seq=0 out_seq=0 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.529+0000 7f9fccc00640 10 client.25691 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fccc00640 1 -- 172.21.15.103:0/924750342 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5590d3047e00 con 0x5590d313ed00 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fccc00640 10 client.25691 handle_command_reply: tid=0 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 client.25691 shutdown 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 -- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 msgr2=0x5590d312bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312bf90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8006010 tx=0x7f9fc8031850 comp rx=0 tx=0).stop 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 -- 172.21.15.103:0/924750342 shutdown_connections 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5590d313ed00 0x5590d31410f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d312eb50 0x5590d312c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d312bf90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 --2- 172.21.15.103:0/924750342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d3123b60 0x5590d312ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fccc00640 0 client.25691 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:11.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 -- 172.21.15.103:0/924750342 >> 172.21.15.103:0/924750342 conn(0x5590d311fe80 msgr2=0x5590d3125c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:11.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 -- 172.21.15.103:0/924750342 shutdown_connections 2024-05-23T00:01:11.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.530+0000 7f9fe904f740 1 -- 172.21.15.103:0/924750342 wait complete. 2024-05-23T00:01:11.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.531+0000 7f9fe904f740 20 client.25691 trim_cache size 0 max 16384 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.549+0000 7f9fe904f740 1 Processor -- start 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fe904f740 1 -- start start 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d311f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31263d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d312c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fe904f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b7800 con 0x5590d312eb50 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fe904f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b50d0 con 0x5590d3124480 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fe904f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b2f20 con 0x5590d3123b60 2024-05-23T00:01:11.554 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d312c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fd7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31263d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d311f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fd7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31263d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:41972/0 (socket says 172.21.15.103:41972) 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.550+0000 7f9fd7e00640 1 -- 172.21.15.103:0/2013833309 learned_addr learned my addr 172.21.15.103:0/2013833309 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fd7e00640 1 -- 172.21.15.103:0/2013833309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 msgr2=0x5590d311f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/2013833309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d311f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fd7e00640 1 -- 172.21.15.103:0/2013833309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 msgr2=0x5590d312c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/2013833309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d312c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fd7e00640 1 -- 172.21.15.103:0/2013833309 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590d3141460 con 0x5590d3124480 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/2013833309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31263d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9fc80096c0 tx=0x7f9fc8009230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c279e77298bb60c server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fcf400640 1 -- 172.21.15.103:0/2013833309 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8035030 con 0x5590d3124480 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fcf400640 1 -- 172.21.15.103:0/2013833309 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc80054e0 con 0x5590d3124480 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.551+0000 7f9fcf400640 1 -- 172.21.15.103:0/2013833309 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc80041c0 con 0x5590d3124480 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 -- 172.21.15.103:0/2013833309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 msgr2=0x5590d31263d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 --2- 172.21.15.103:0/2013833309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31263d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9fc80096c0 tx=0x7f9fc8009230 comp rx=0 tx=0).stop 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 -- 172.21.15.103:0/2013833309 shutdown_connections 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 --2- 172.21.15.103:0/2013833309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d312c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 --2- 172.21.15.103:0/2013833309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31263d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 --2- 172.21.15.103:0/2013833309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d311f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 -- 172.21.15.103:0/2013833309 >> 172.21.15.103:0/2013833309 conn(0x5590d311fe80 msgr2=0x5590d312ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 -- 172.21.15.103:0/2013833309 shutdown_connections 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 -- 172.21.15.103:0/2013833309 wait complete. 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.552+0000 7f9fe904f740 1 -- start start 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fe904f740 1 Processor -- start 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3136b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31370d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fe904f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3133b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fe904f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b3590 con 0x5590d312eb50 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fe904f740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b4a00 con 0x5590d3124480 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fe904f740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b3c60 con 0x5590d3123b60 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fe904f740 10 client.0 _ll_register_callbacks cb 0x7f9fc8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:11.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fd7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fd7e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31370d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:41976/0 (socket says 172.21.15.103:41976) 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fd7e00640 1 -- 172.21.15.103:0/3732287255 learned_addr learned my addr 172.21.15.103:0/3732287255 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3133b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.553+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3136b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fd7e00640 1 -- 172.21.15.103:0/3732287255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 msgr2=0x5590d3136b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3136b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fd7e00640 1 -- 172.21.15.103:0/3732287255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 msgr2=0x5590d3133b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3133b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fd7e00640 1 -- 172.21.15.103:0/3732287255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590d31341a0 con 0x5590d3124480 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fd7e00640 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31370d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9fc802a930 tx=0x7f9fc8005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6de54e781b71cfa8 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fccc00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fccc00640 1 -- 172.21.15.103:0/3732287255 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8035070 con 0x5590d3124480 2024-05-23T00:01:11.562 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fe904f740 1 -- 172.21.15.103:0/3732287255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fc80054e0 con 0x5590d3124480 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fccc00640 1 -- 172.21.15.103:0/3732287255 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f9fc8003e40 con 0x5590d3124480 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fccc00640 1 -- 172.21.15.103:0/3732287255 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9fc8069050 con 0x5590d3124480 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fe904f740 10 client.25697 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.554+0000 7f9fe904f740 1 -- 172.21.15.103:0/3732287255 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5590d31a5b70 con 0x5590d3124480 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fccc00640 1 -- 172.21.15.103:0/3732287255 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc80054e0 con 0x5590d3124480 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fccc00640 1 -- 172.21.15.103:0/3732287255 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f9fc8004ba0 con 0x5590d3124480 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fe904f740 10 client.25697 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fe904f740 10 client.25697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fe904f740 20 client.25697 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fe904f740 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5590d31a62a0 0x5590d31a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fe904f740 4 client.25697 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fe904f740 1 -- 172.21.15.103:0/3732287255 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5590d30963e0 con 0x5590d31a62a0 2024-05-23T00:01:11.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.555+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5590d31a62a0 0x5590d31a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5590d31a62a0 0x5590d31a86c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd4a89f65c27d41c server_cookie=dba51a44fa316c9 in_seq=0 out_seq=0 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fccc00640 10 client.25697 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fccc00640 1 -- 172.21.15.103:0/3732287255 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5590d30963e0 con 0x5590d31a62a0 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fccc00640 10 client.25697 handle_command_reply: tid=0 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 client.25697 shutdown 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 -- 172.21.15.103:0/3732287255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 msgr2=0x5590d31370d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31370d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f9fc802a930 tx=0x7f9fc8005a20 comp rx=0 tx=0).stop 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 -- 172.21.15.103:0/3732287255 shutdown_connections 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5590d31a62a0 0x5590d31a86c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5590d312eb50 0x5590d3133b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5590d3124480 0x5590d31370d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 --2- 172.21.15.103:0/3732287255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5590d3123b60 0x5590d3136b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fccc00640 0 client.25697 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:11.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.556+0000 7f9fe904f740 1 -- 172.21.15.103:0/3732287255 >> 172.21.15.103:0/3732287255 conn(0x5590d311fe80 msgr2=0x5590d311ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:11.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe904f740 1 -- 172.21.15.103:0/3732287255 shutdown_connections 2024-05-23T00:01:11.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe904f740 1 -- 172.21.15.103:0/3732287255 wait complete. 2024-05-23T00:01:11.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe904f740 20 client.25697 trim_cache size 0 max 16384 2024-05-23T00:01:11.565 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:01:11.565 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "no active scrubs running", 2024-05-23T00:01:11.565 INFO:teuthology.orchestra.run.smithi103.stdout: "scrubs": {} 2024-05-23T00:01:11.565 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:01:11.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 -- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2ea0 msgr2=0x7f9fc40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2ea0 0x7f9fc40a5360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9fd00cd170 tx=0x7f9fd00d3970 comp rx=0 tx=0).stop 2024-05-23T00:01:11.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 -- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0107ee0 msgr2=0x7f9fe019e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0107ee0 0x7f9fe019e5d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000e9f0 tx=0x7f9fd000edf0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 -- 172.21.15.103:0/403552080 shutdown_connections 2024-05-23T00:01:11.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2ea0 0x7f9fc40a5360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe01a2df0 0x7f9fe01a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe010cfa0 0x7f9fe019eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 --2- 172.21.15.103:0/403552080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe0107ee0 0x7f9fe019e5d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.557+0000 7f9fe4c00640 1 -- 172.21.15.103:0/403552080 >> 172.21.15.103:0/403552080 conn(0x7f9fe0077480 msgr2=0x7f9fe0079900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:11.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.558+0000 7f9fe4c00640 1 -- 172.21.15.103:0/403552080 shutdown_connections 2024-05-23T00:01:11.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.558+0000 7f9fe4c00640 1 -- 172.21.15.103:0/403552080 wait complete. 2024-05-23T00:01:11.580 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-23T00:01:11.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:01:11.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 -- 172.21.15.103:0/289319343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11101076a0 msgr2=0x7f1110107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 --2- 172.21.15.103:0/289319343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11101076a0 0x7f1110107a80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1108009be0 tx=0x7f110802a6c0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 -- 172.21.15.103:0/289319343 shutdown_connections 2024-05-23T00:01:11.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 --2- 172.21.15.103:0/289319343 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1110102050 0x7f111010e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 --2- 172.21.15.103:0/289319343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1110101690 0x7f1110101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 --2- 172.21.15.103:0/289319343 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11101076a0 0x7f1110107a80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 -- 172.21.15.103:0/289319343 >> 172.21.15.103:0/289319343 conn(0x7f11100fd1a0 msgr2=0x7f11100ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:11.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 -- 172.21.15.103:0/289319343 shutdown_connections 2024-05-23T00:01:11.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.786+0000 7f1116e00640 1 -- 172.21.15.103:0/289319343 wait complete. 2024-05-23T00:01:11.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.787+0000 7f1116e00640 1 Processor -- start 2024-05-23T00:01:11.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.787+0000 7f1116e00640 1 -- start start 2024-05-23T00:01:11.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1116e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1110101690 0x7f1110073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1116e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1110102050 0x7f11100740b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1116e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11101076a0 0x7f11100745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1116e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1110111ab0 con 0x7f1110102050 2024-05-23T00:01:11.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1116e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1110111930 con 0x7f1110101690 2024-05-23T00:01:11.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1116e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1110111c30 con 0x7f11101076a0 2024-05-23T00:01:11.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1115000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1110102050 0x7f11100740b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1115a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1110101690 0x7f1110073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1115a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1110101690 0x7f1110073b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:42000/0 (socket says 172.21.15.103:42000) 2024-05-23T00:01:11.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1115a00640 1 -- 172.21.15.103:0/3498605230 learned_addr learned my addr 172.21.15.103:0/3498605230 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:11.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1116400640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11101076a0 0x7f11100745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1115a00640 1 -- 172.21.15.103:0/3498605230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11101076a0 msgr2=0x7f11100745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1115a00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11101076a0 0x7f11100745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1115a00640 1 -- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1110102050 msgr2=0x7f11100740b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:11.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.788+0000 7f1115a00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1110102050 0x7f11100740b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:11.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.789+0000 7f1115a00640 1 -- 172.21.15.103:0/3498605230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11101a6450 con 0x7f1110101690 2024-05-23T00:01:11.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.789+0000 7f1115a00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1110101690 0x7f1110073b70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f110802acb0 tx=0x7f1108030c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:11.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.789+0000 7f110e000640 1 -- 172.21.15.103:0/3498605230 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1108041020 con 0x7f1110101690 2024-05-23T00:01:11.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.789+0000 7f110e000640 1 -- 172.21.15.103:0/3498605230 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1108003e50 con 0x7f1110101690 2024-05-23T00:01:11.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.789+0000 7f110e000640 1 -- 172.21.15.103:0/3498605230 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1108036040 con 0x7f1110101690 2024-05-23T00:01:11.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.789+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11101a66e0 con 0x7f1110101690 2024-05-23T00:01:11.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.789+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11101a6b80 con 0x7f1110101690 2024-05-23T00:01:11.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.790+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11100685b0 con 0x7f1110101690 2024-05-23T00:01:11.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.793+0000 7f110e000640 1 -- 172.21.15.103:0/3498605230 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f1108044020 con 0x7f1110101690 2024-05-23T00:01:11.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.794+0000 7f110e000640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f10f40a2ea0 0x7f10f40a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:11.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.794+0000 7f110e000640 1 -- 172.21.15.103:0/3498605230 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f11080f0820 con 0x7f1110101690 2024-05-23T00:01:11.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.794+0000 7f1115000640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f10f40a2ea0 0x7f10f40a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:11.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.795+0000 7f1115000640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f10f40a2ea0 0x7f10f40a5360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1110075600 tx=0x7f10fc009260 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:11.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:11.797+0000 7f110e000640 1 -- 172.21.15.103:0/3498605230 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f110804b020 con 0x7f1110101690 2024-05-23T00:01:12.042 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.039+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11100750a0 con 0x7f1110101690 2024-05-23T00:01:12.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.040+0000 7f110e000640 1 -- 172.21.15.103:0/3498605230 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11896 (secure 0 0 0) 0x7f11100750a0 con 0x7f1110101690 2024-05-23T00:01:12.044 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:01:12.044 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-23T00:00:31.011025+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24559,24563]},"id":1}]} 2024-05-23T00:01:12.044 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 25 2024-05-23T00:01:12.046 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f10f40a2ea0 msgr2=0x7f10f40a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f10f40a2ea0 0x7f10f40a5360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1110075600 tx=0x7f10fc009260 comp rx=0 tx=0).stop 2024-05-23T00:01:12.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1110101690 msgr2=0x7f1110073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1110101690 0x7f1110073b70 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f110802acb0 tx=0x7f1108030c50 comp rx=0 tx=0).stop 2024-05-23T00:01:12.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 shutdown_connections 2024-05-23T00:01:12.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f10f40a2ea0 0x7f10f40a5360 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11101076a0 0x7f11100745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1110102050 0x7f11100740b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 --2- 172.21.15.103:0/3498605230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1110101690 0x7f1110073b70 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 >> 172.21.15.103:0/3498605230 conn(0x7f11100fd1a0 msgr2=0x7f11100fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 shutdown_connections 2024-05-23T00:01:12.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.043+0000 7f1116e00640 1 -- 172.21.15.103:0/3498605230 wait complete. 2024-05-23T00:01:12.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-23T00:01:12.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 -- 172.21.15.103:0/1322144408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad601126d0 msgr2=0x7fad60114ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 --2- 172.21.15.103:0/1322144408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad601126d0 0x7fad60114ac0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fad58009900 tx=0x7fad5802a3f0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 -- 172.21.15.103:0/1322144408 shutdown_connections 2024-05-23T00:01:12.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 --2- 172.21.15.103:0/1322144408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad601126d0 0x7fad60114ac0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 --2- 172.21.15.103:0/1322144408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad600766c0 0x7fad60112190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 --2- 172.21.15.103:0/1322144408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad60075da0 0x7fad60076180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 -- 172.21.15.103:0/1322144408 >> 172.21.15.103:0/1322144408 conn(0x7fad600ff520 msgr2=0x7fad60101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 -- 172.21.15.103:0/1322144408 shutdown_connections 2024-05-23T00:01:12.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.261+0000 7fad66c00640 1 -- 172.21.15.103:0/1322144408 wait complete. 2024-05-23T00:01:12.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.263+0000 7fad66c00640 1 Processor -- start 2024-05-23T00:01:12.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.263+0000 7fad66c00640 1 -- start start 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.263+0000 7fad66c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad60075da0 0x7fad6019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.263+0000 7fad66c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad600766c0 0x7fad6019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.263+0000 7fad66c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad601126d0 0x7fad601a3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.263+0000 7fad66c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad60118260 con 0x7fad60075da0 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.263+0000 7fad66c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad601180e0 con 0x7fad601126d0 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.263+0000 7fad66c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fad601183e0 con 0x7fad600766c0 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad60075da0 0x7fad6019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad60075da0 0x7fad6019e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41332/0 (socket says 172.21.15.103:41332) 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 -- 172.21.15.103:0/2687289329 learned_addr learned my addr 172.21.15.103:0/2687289329 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad57400640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad600766c0 0x7fad6019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad64c00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad601126d0 0x7fad601a3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 -- 172.21.15.103:0/2687289329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad600766c0 msgr2=0x7fad6019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad600766c0 0x7fad6019eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 -- 172.21.15.103:0/2687289329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad601126d0 msgr2=0x7fad601a3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad601126d0 0x7fad601a3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 -- 172.21.15.103:0/2687289329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad601a3750 con 0x7fad60075da0 2024-05-23T00:01:12.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.264+0000 7fad5fe00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad60075da0 0x7fad6019e760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad5000eb90 tx=0x7fad5000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.265+0000 7fad5d600640 1 -- 172.21.15.103:0/2687289329 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50004d20 con 0x7fad60075da0 2024-05-23T00:01:12.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.265+0000 7fad5d600640 1 -- 172.21.15.103:0/2687289329 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad5000ee90 con 0x7fad60075da0 2024-05-23T00:01:12.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.265+0000 7fad5d600640 1 -- 172.21.15.103:0/2687289329 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50010830 con 0x7fad60075da0 2024-05-23T00:01:12.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.265+0000 7fad66c00640 1 -- 172.21.15.103:0/2687289329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad601a3a40 con 0x7fad60075da0 2024-05-23T00:01:12.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.265+0000 7fad66c00640 1 -- 172.21.15.103:0/2687289329 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad601ab2d0 con 0x7fad60075da0 2024-05-23T00:01:12.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.268+0000 7fad69a04740 1 Processor -- start 2024-05-23T00:01:12.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.268+0000 7fad5d600640 1 -- 172.21.15.103:0/2687289329 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7fad50005020 con 0x7fad60075da0 2024-05-23T00:01:12.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.268+0000 7fad69a04740 1 -- start start 2024-05-23T00:01:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad69a04740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc88b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad69a04740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc895990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad69a04740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc8983b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad69a04740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5654bc7efb20 con 0x5654bc88b8f0 2024-05-23T00:01:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad69a04740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5654bc7971b0 con 0x5654bc88af40 2024-05-23T00:01:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad69a04740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5654bc7ebae0 con 0x5654bc895f60 2024-05-23T00:01:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad5fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc895990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad64c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc88b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad5fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc895990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41342/0 (socket says 172.21.15.103:41342) 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad5fe00640 1 -- 172.21.15.103:0/3613272164 learned_addr learned my addr 172.21.15.103:0/3613272164 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad57400640 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc8983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad64c00640 1 -- 172.21.15.103:0/3613272164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 msgr2=0x5654bc8983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad64c00640 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc8983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad64c00640 1 -- 172.21.15.103:0/3613272164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 msgr2=0x5654bc895990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad64c00640 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc895990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad64c00640 1 -- 172.21.15.103:0/3613272164 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654bc853610 con 0x5654bc88af40 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.269+0000 7fad5fe00640 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc895990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.270+0000 7fad64c00640 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc88b320 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad58009230 tx=0x7fad58005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14b04e64b314af12 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.270+0000 7fad5d600640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fad440a2ea0 0x7fad440a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.270+0000 7fad55200640 1 -- 172.21.15.103:0/3613272164 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58040070 con 0x5654bc88af40 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.270+0000 7fad55200640 1 -- 172.21.15.103:0/3613272164 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad58004380 con 0x5654bc88af40 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.270+0000 7fad55200640 1 -- 172.21.15.103:0/3613272164 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad58035040 con 0x5654bc88af40 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.270+0000 7fad5d600640 1 -- 172.21.15.103:0/2687289329 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fad500d0030 con 0x7fad60075da0 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.270+0000 7fad5fe00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fad440a2ea0 0x7fad440a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 -- 172.21.15.103:0/3613272164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 msgr2=0x5654bc88b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad5fe00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fad440a2ea0 0x7fad440a5360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fad500cda40 tx=0x7fad500d3b40 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc88b320 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad58009230 tx=0x7fad58005e60 comp rx=0 tx=0).stop 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 -- 172.21.15.103:0/3613272164 shutdown_connections 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc8983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc895990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 --2- 172.21.15.103:0/3613272164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc88b320 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 -- 172.21.15.103:0/3613272164 >> 172.21.15.103:0/3613272164 conn(0x5654bc887200 msgr2=0x5654bc8875e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 -- 172.21.15.103:0/3613272164 shutdown_connections 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.271+0000 7fad69a04740 1 -- 172.21.15.103:0/3613272164 wait complete. 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.272+0000 7fad69a04740 1 -- start start 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.272+0000 7fad69a04740 1 Processor -- start 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.272+0000 7fad69a04740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc892ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.272+0000 7fad69a04740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc893430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.272+0000 7fad69a04740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc893970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.272+0000 7fad69a04740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5654bc898e70 con 0x5654bc88b8f0 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.272+0000 7fad69a04740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5654bc7aa600 con 0x5654bc88af40 2024-05-23T00:01:12.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.272+0000 7fad69a04740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5654bc898ff0 con 0x5654bc895f60 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad69a04740 10 client.0 _ll_register_callbacks cb 0x5654bc7e5160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad57400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc893430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad64c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc892ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad57400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc893430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41358/0 (socket says 172.21.15.103:41358) 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad57400640 1 -- 172.21.15.103:0/169845406 learned_addr learned my addr 172.21.15.103:0/169845406 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad64c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc892ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:42046/0 (socket says 172.21.15.103:42046) 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad64c00640 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc893970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad57400640 1 -- 172.21.15.103:0/169845406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 msgr2=0x5654bc893970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad57400640 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc893970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad57400640 1 -- 172.21.15.103:0/169845406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 msgr2=0x5654bc892ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad57400640 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc892ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.273+0000 7fad57400640 1 -- 172.21.15.103:0/169845406 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654bc7b7a90 con 0x5654bc88b8f0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad57400640 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc893430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fad480126e0 tx=0x7fad48012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1757e5b0bd9d25 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad4d600640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad4d600640 1 -- 172.21.15.103:0/169845406 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480120d0 con 0x5654bc88b8f0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad4d600640 1 -- 172.21.15.103:0/169845406 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fad4800f920 con 0x5654bc88b8f0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad69a04740 1 -- 172.21.15.103:0/169845406 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad58004380 con 0x5654bc88b8f0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad4d600640 1 -- 172.21.15.103:0/169845406 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fad48012270 con 0x5654bc88b8f0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad69a04740 10 client.15609 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad69a04740 1 -- 172.21.15.103:0/169845406 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5654bc79f0d0 con 0x5654bc88b8f0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.274+0000 7fad4d600640 1 -- 172.21.15.103:0/169845406 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad58004380 con 0x5654bc88b8f0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.275+0000 7fad4d600640 1 -- 172.21.15.103:0/169845406 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fad4801c420 con 0x5654bc88b8f0 2024-05-23T00:01:12.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.275+0000 7fad69a04740 10 client.15609 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.275+0000 7fad69a04740 10 client.15609 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-23T00:01:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.275+0000 7fad69a04740 20 client.15609 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.275+0000 7fad69a04740 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x5654bc8a6160 0x5654bc8a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.275+0000 7fad69a04740 4 client.15609 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.275+0000 7fad69a04740 1 -- 172.21.15.103:0/169845406 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654bc7fd250 con 0x5654bc8a6160 2024-05-23T00:01:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.275+0000 7fad64c00640 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x5654bc8a6160 0x5654bc8a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.276+0000 7fad64c00640 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x5654bc8a6160 0x5654bc8a8590 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19ef6a14827d83ae server_cookie=ef6b50d71f672ade in_seq=0 out_seq=0 2024-05-23T00:01:12.283 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.276+0000 7fad4d600640 10 client.15609 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-23T00:01:12.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.276+0000 7fad4d600640 1 -- 172.21.15.103:0/169845406 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5654bc7fd250 con 0x5654bc8a6160 2024-05-23T00:01:12.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.276+0000 7fad4d600640 10 client.15609 handle_command_reply: tid=0 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.276+0000 7fad69a04740 1 client.15609 shutdown 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 -- 172.21.15.103:0/169845406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 msgr2=0x5654bc893430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc893430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fad480126e0 tx=0x7fad48012ba0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 -- 172.21.15.103:0/169845406 shutdown_connections 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x5654bc8a6160 0x5654bc8a8590 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc893970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc893430 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad4d600640 0 client.15609 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 --2- 172.21.15.103:0/169845406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc892ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 -- 172.21.15.103:0/169845406 >> 172.21.15.103:0/169845406 conn(0x5654bc887200 msgr2=0x5654bc88d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 -- 172.21.15.103:0/169845406 shutdown_connections 2024-05-23T00:01:12.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 1 -- 172.21.15.103:0/169845406 wait complete. 2024-05-23T00:01:12.286 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.277+0000 7fad69a04740 20 client.15609 trim_cache size 0 max 16384 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad69a04740 1 Processor -- start 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad69a04740 1 -- start start 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad69a04740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88af40 0x5654bc88d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad69a04740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88b8f0 0x5654bc893740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad69a04740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc8ade60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad69a04740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad601b7a80 con 0x5654bc88af40 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad69a04740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad601b5350 con 0x5654bc88b8f0 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad69a04740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fad601b31a0 con 0x5654bc895f60 2024-05-23T00:01:12.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad64c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88af40 0x5654bc88d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad64c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88af40 0x5654bc88d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41366/0 (socket says 172.21.15.103:41366) 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad64c00640 1 -- 172.21.15.103:0/4178895000 learned_addr learned my addr 172.21.15.103:0/4178895000 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.292+0000 7fad57400640 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88b8f0 0x5654bc893740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad64c00640 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc8ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad57400640 1 -- 172.21.15.103:0/4178895000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 msgr2=0x5654bc8ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad57400640 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc8ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad57400640 1 -- 172.21.15.103:0/4178895000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88af40 msgr2=0x5654bc88d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad57400640 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88af40 0x5654bc88d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad57400640 1 -- 172.21.15.103:0/4178895000 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654bc5033f0 con 0x5654bc88b8f0 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad64c00640 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88af40 0x5654bc88d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:12.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad57400640 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88b8f0 0x5654bc893740 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad480126b0 tx=0x7fad4800f810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a4e8d722d178bc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad4fe00640 1 -- 172.21.15.103:0/4178895000 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4801c420 con 0x5654bc88b8f0 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad4fe00640 1 -- 172.21.15.103:0/4178895000 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad4801c5c0 con 0x5654bc88b8f0 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.293+0000 7fad4fe00640 1 -- 172.21.15.103:0/4178895000 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad48005460 con 0x5654bc88b8f0 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 -- 172.21.15.103:0/4178895000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88b8f0 msgr2=0x5654bc893740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88b8f0 0x5654bc893740 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad480126b0 tx=0x7fad4800f810 comp rx=0 tx=0).stop 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 -- 172.21.15.103:0/4178895000 shutdown_connections 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc8ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88b8f0 0x5654bc893740 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 --2- 172.21.15.103:0/4178895000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88af40 0x5654bc88d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 -- 172.21.15.103:0/4178895000 >> 172.21.15.103:0/4178895000 conn(0x5654bc887200 msgr2=0x5654bc885c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 -- 172.21.15.103:0/4178895000 shutdown_connections 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 -- 172.21.15.103:0/4178895000 wait complete. 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.294+0000 7fad69a04740 1 -- start start 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad69a04740 1 Processor -- start 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad69a04740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc89a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad69a04740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc89a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad69a04740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc90b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad69a04740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fad601b3810 con 0x5654bc88b8f0 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad69a04740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fad601b4c80 con 0x5654bc88af40 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad69a04740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fad601b3ee0 con 0x5654bc895f60 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad69a04740 10 client.0 _ll_register_callbacks cb 0x7fad48009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad64c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc89a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad57400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc89a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad64c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc89a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:42068/0 (socket says 172.21.15.103:42068) 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.295+0000 7fad64c00640 1 -- 172.21.15.103:0/3875194865 learned_addr learned my addr 172.21.15.103:0/3875194865 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad64c00640 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc90b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad57400640 1 -- 172.21.15.103:0/3875194865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 msgr2=0x5654bc90b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad57400640 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc90b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad57400640 1 -- 172.21.15.103:0/3875194865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 msgr2=0x5654bc89a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad57400640 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc89a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad57400640 1 -- 172.21.15.103:0/3875194865 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654bc5cd7d0 con 0x5654bc88b8f0 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad64c00640 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc89a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad57400640 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc89a860 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x5654bc894080 tx=0x7fad4800fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d0743b7bbf6405 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad4d600640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:12.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad4d600640 1 -- 172.21.15.103:0/3875194865 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad4801c420 con 0x5654bc88b8f0 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad4d600640 1 -- 172.21.15.103:0/3875194865 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fad48005320 con 0x5654bc88b8f0 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.296+0000 7fad69a04740 1 -- 172.21.15.103:0/3875194865 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad4801c5c0 con 0x5654bc88b8f0 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad4d600640 1 -- 172.21.15.103:0/3875194865 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fad48005ab0 con 0x5654bc88b8f0 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad69a04740 10 client.15615 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad69a04740 1 -- 172.21.15.103:0/3875194865 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5654bc90cca0 con 0x5654bc88b8f0 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad4d600640 1 -- 172.21.15.103:0/3875194865 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad4801c5c0 con 0x5654bc88b8f0 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad4d600640 1 -- 172.21.15.103:0/3875194865 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fad48003ad0 con 0x5654bc88b8f0 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad69a04740 10 client.15615 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad69a04740 10 client.15615 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad69a04740 20 client.15615 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad69a04740 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x5654bc90d350 0x5654bc90f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad69a04740 4 client.15615 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:01:12.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.297+0000 7fad69a04740 1 -- 172.21.15.103:0/3875194865 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654bc7f75a0 con 0x5654bc90d350 2024-05-23T00:01:12.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.298+0000 7fad64c00640 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x5654bc90d350 0x5654bc90f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.298+0000 7fad64c00640 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x5654bc90d350 0x5654bc90f740 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cfafbfd577fb19bd server_cookie=970f30bdf057b76a in_seq=0 out_seq=0 2024-05-23T00:01:12.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.298+0000 7fad4d600640 10 client.15615 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-23T00:01:12.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.298+0000 7fad4d600640 1 -- 172.21.15.103:0/3875194865 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654bc7f75a0 con 0x5654bc90d350 2024-05-23T00:01:12.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.298+0000 7fad4d600640 10 client.15615 handle_command_reply: tid=0 2024-05-23T00:01:12.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.298+0000 7fad69a04740 1 client.15615 shutdown 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.298+0000 7fad69a04740 1 -- 172.21.15.103:0/3875194865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 msgr2=0x5654bc89a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.298+0000 7fad69a04740 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc89a860 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x5654bc894080 tx=0x7fad4800fb50 comp rx=0 tx=0).stop 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 1 -- 172.21.15.103:0/3875194865 shutdown_connections 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x5654bc90d350 0x5654bc90f740 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5654bc895f60 0x5654bc90b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5654bc88b8f0 0x5654bc89a860 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 1 --2- 172.21.15.103:0/3875194865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5654bc88af40 0x5654bc89a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 1 -- 172.21.15.103:0/3875194865 >> 172.21.15.103:0/3875194865 conn(0x5654bc887200 msgr2=0x5654bc8935c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad4d600640 0 client.15615 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 1 -- 172.21.15.103:0/3875194865 shutdown_connections 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 1 -- 172.21.15.103:0/3875194865 wait complete. 2024-05-23T00:01:12.307 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad69a04740 20 client.15615 trim_cache size 0 max 16384 2024-05-23T00:01:12.308 INFO:teuthology.orchestra.run.smithi103.stdout:[] 2024-05-23T00:01:12.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad66c00640 1 -- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fad440a2ea0 msgr2=0x7fad440a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad66c00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fad440a2ea0 0x7fad440a5360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fad500cda40 tx=0x7fad500d3b40 comp rx=0 tx=0).stop 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad66c00640 1 -- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad60075da0 msgr2=0x7fad6019e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.299+0000 7fad66c00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad60075da0 0x7fad6019e760 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad5000eb90 tx=0x7fad5000c6a0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.300+0000 7fad66c00640 1 -- 172.21.15.103:0/2687289329 shutdown_connections 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.300+0000 7fad66c00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fad440a2ea0 0x7fad440a5360 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.300+0000 7fad66c00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad601126d0 0x7fad601a3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.300+0000 7fad66c00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad600766c0 0x7fad6019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.300+0000 7fad66c00640 1 --2- 172.21.15.103:0/2687289329 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fad60075da0 0x7fad6019e760 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.300+0000 7fad66c00640 1 -- 172.21.15.103:0/2687289329 >> 172.21.15.103:0/2687289329 conn(0x7fad600ff520 msgr2=0x7fad60101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.300+0000 7fad66c00640 1 -- 172.21.15.103:0/2687289329 shutdown_connections 2024-05-23T00:01:12.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.300+0000 7fad66c00640 1 -- 172.21.15.103:0/2687289329 wait complete. 2024-05-23T00:01:12.317 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:12 smithi103 ceph-mon[31370]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-05-23T00:01:12.317 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:12 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3498605230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:01:12.320 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-23T00:01:12.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:12 smithi174 ceph-mon[36179]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-05-23T00:01:12.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:12 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3498605230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:01:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 -- 172.21.15.103:0/145047058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c10e0d0 msgr2=0x7fc69c1104c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/145047058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c10e0d0 0x7fc69c1104c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc694009ca0 tx=0x7fc69402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 -- 172.21.15.103:0/145047058 shutdown_connections 2024-05-23T00:01:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/145047058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c10e0d0 0x7fc69c1104c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/145047058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc69c100ed0 0x7fc69c10db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/145047058 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc69c100520 0x7fc69c100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 -- 172.21.15.103:0/145047058 >> 172.21.15.103:0/145047058 conn(0x7fc69c0fc160 msgr2=0x7fc69c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 -- 172.21.15.103:0/145047058 shutdown_connections 2024-05-23T00:01:12.527 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.522+0000 7fc6a1e00640 1 -- 172.21.15.103:0/145047058 wait complete. 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.524+0000 7fc6a1e00640 1 Processor -- start 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.524+0000 7fc6a1e00640 1 -- start start 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.524+0000 7fc6a1e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c100520 0x7fc69c19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.524+0000 7fc6a1e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc69c100ed0 0x7fc69c19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.524+0000 7fc6a1e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc69c10e0d0 0x7fc69c19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.524+0000 7fc6a1e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c113d80 con 0x7fc69c100ed0 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.524+0000 7fc6a1e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c113c00 con 0x7fc69c100520 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.524+0000 7fc6a1e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c113f00 con 0x7fc69c10e0d0 2024-05-23T00:01:12.529 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c100520 0x7fc69c19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc69be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc69c100ed0 0x7fc69c19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c100520 0x7fc69c19a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:42092/0 (socket says 172.21.15.103:42092) 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 -- 172.21.15.103:0/2658426607 learned_addr learned my addr 172.21.15.103:0/2658426607 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a1400640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc69c10e0d0 0x7fc69c19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 -- 172.21.15.103:0/2658426607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc69c10e0d0 msgr2=0x7fc69c19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc69c10e0d0 0x7fc69c19eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 -- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc69c100ed0 msgr2=0x7fc69c19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc69c100ed0 0x7fc69c19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 -- 172.21.15.103:0/2658426607 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c19f270 con 0x7fc69c100520 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.525+0000 7fc6a0a00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c100520 0x7fc69c19a280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc69000e9f0 tx=0x7fc69000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.526+0000 7fc699600640 1 -- 172.21.15.103:0/2658426607 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69000cd80 con 0x7fc69c100520 2024-05-23T00:01:12.530 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.526+0000 7fc6a1e00640 1 -- 172.21.15.103:0/2658426607 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c19f560 con 0x7fc69c100520 2024-05-23T00:01:12.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.526+0000 7fc699600640 1 -- 172.21.15.103:0/2658426607 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc690005500 con 0x7fc69c100520 2024-05-23T00:01:12.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.526+0000 7fc699600640 1 -- 172.21.15.103:0/2658426607 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690010690 con 0x7fc69c100520 2024-05-23T00:01:12.531 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.526+0000 7fc6a1e00640 1 -- 172.21.15.103:0/2658426607 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c1a6df0 con 0x7fc69c100520 2024-05-23T00:01:12.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc6a637b740 1 Processor -- start 2024-05-23T00:01:12.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc6a637b740 1 -- start start 2024-05-23T00:01:12.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc699600640 1 -- 172.21.15.103:0/2658426607 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7fc6900046c0 con 0x7fc69c100520 2024-05-23T00:01:12.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a58484f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848ac0 0x5596a5852b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5853130 0x5596a5855580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc6a637b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5596a57acb20 con 0x5596a5848ac0 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc6a637b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5596a57541b0 con 0x5596a5848110 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.529+0000 7fc6a637b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5596a57a8ae0 con 0x5596a5853130 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a58484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a58484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:42102/0 (socket says 172.21.15.103:42102) 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a1400640 1 -- 172.21.15.103:0/2419774236 learned_addr learned my addr 172.21.15.103:0/2419774236 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a0a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848ac0 0x5596a5852b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc69be00640 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5853130 0x5596a5855580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a1400640 1 -- 172.21.15.103:0/2419774236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5853130 msgr2=0x5596a5855580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a1400640 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5853130 0x5596a5855580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a1400640 1 -- 172.21.15.103:0/2419774236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848ac0 msgr2=0x5596a5852b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a1400640 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848ac0 0x5596a5852b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc6a1400640 1 -- 172.21.15.103:0/2419774236 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596a5810610 con 0x5596a5848110 2024-05-23T00:01:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.530+0000 7fc69be00640 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5853130 0x5596a5855580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:01:12.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.531+0000 7fc6a1400640 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a58484f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc69c102700 tx=0x7fc694005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433ccf260d959e8d server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.531+0000 7fc68dc00640 1 -- 172.21.15.103:0/2419774236 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694031db0 con 0x5596a5848110 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.531+0000 7fc68dc00640 1 -- 172.21.15.103:0/2419774236 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc694036040 con 0x5596a5848110 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.531+0000 7fc68dc00640 1 -- 172.21.15.103:0/2419774236 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694004330 con 0x5596a5848110 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.531+0000 7fc699600640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc6840a2ea0 0x7fc6840a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.531+0000 7fc699600640 1 -- 172.21.15.103:0/2658426607 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc6900cfe30 con 0x7fc69c100520 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.531+0000 7fc6a0a00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc6840a2ea0 0x7fc6840a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 -- 172.21.15.103:0/2419774236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 msgr2=0x5596a58484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a58484f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc69c102700 tx=0x7fc694005f10 comp rx=0 tx=0).stop 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 -- 172.21.15.103:0/2419774236 shutdown_connections 2024-05-23T00:01:12.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5853130 0x5596a5855580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848ac0 0x5596a5852b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 --2- 172.21.15.103:0/2419774236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a58484f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 -- 172.21.15.103:0/2419774236 >> 172.21.15.103:0/2419774236 conn(0x5596a58443d0 msgr2=0x5596a58447b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 -- 172.21.15.103:0/2419774236 shutdown_connections 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 -- 172.21.15.103:0/2419774236 wait complete. 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a0a00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc6840a2ea0 0x7fc6840a5360 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc6900cd0c0 tx=0x7fc6900d38c0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.532+0000 7fc6a637b740 1 -- start start 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a637b740 1 Processor -- start 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a5850020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a5850560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a5850aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a637b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5596a5855ec0 con 0x5596a5853130 2024-05-23T00:01:12.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a637b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5596a5767600 con 0x5596a5848110 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a637b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5596a5856040 con 0x5596a5848ac0 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a637b740 10 client.0 _ll_register_callbacks cb 0x5596a57a2160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a5850020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a5850aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc69be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a5850560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a5850aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41406/0 (socket says 172.21.15.103:41406) 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.533+0000 7fc6a1400640 1 -- 172.21.15.103:0/1451628407 learned_addr learned my addr 172.21.15.103:0/1451628407 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.534+0000 7fc6a1400640 1 -- 172.21.15.103:0/1451628407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 msgr2=0x5596a5850560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.534+0000 7fc6a1400640 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a5850560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.534+0000 7fc6a1400640 1 -- 172.21.15.103:0/1451628407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 msgr2=0x5596a5850020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.534+0000 7fc6a1400640 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a5850020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.534+0000 7fc6a1400640 1 -- 172.21.15.103:0/1451628407 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596a5774a90 con 0x5596a5853130 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.534+0000 7fc6a1400640 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a5850aa0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc69402ade0 tx=0x7fc69402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39728ebe3a29be8 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.534+0000 7fc682000640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.534+0000 7fc682000640 1 -- 172.21.15.103:0/1451628407 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6940040b0 con 0x5596a5853130 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc682000640 1 -- 172.21.15.103:0/1451628407 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc694004250 con 0x5596a5853130 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 1 -- 172.21.15.103:0/1451628407 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc694036040 con 0x5596a5853130 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc682000640 1 -- 172.21.15.103:0/1451628407 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc694004b40 con 0x5596a5853130 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 10 client.15627 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 1 -- 172.21.15.103:0/1451628407 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5596a575c0d0 con 0x5596a5853130 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc682000640 1 -- 172.21.15.103:0/1451628407 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc694036040 con 0x5596a5853130 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc682000640 1 -- 172.21.15.103:0/1451628407 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc694043070 con 0x5596a5853130 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 10 client.15627 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 10 client.15627 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 20 client.15627 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:12.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5596a5862ea0 0x5596a5865290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 4 client.15627 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.535+0000 7fc6a637b740 1 -- 172.21.15.103:0/1451628407 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596a57ba250 con 0x5596a5862ea0 2024-05-23T00:01:12.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.536+0000 7fc69be00640 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5596a5862ea0 0x5596a5865290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.536+0000 7fc69be00640 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5596a5862ea0 0x5596a5865290 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b20e5a4a505cfcde server_cookie=bb65cc5cc303774c in_seq=0 out_seq=0 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.536+0000 7fc682000640 10 client.15627 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc682000640 1 -- 172.21.15.103:0/1451628407 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596a57ba250 con 0x5596a5862ea0 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc682000640 10 client.15627 handle_command_reply: tid=0 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 client.15627 shutdown 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 -- 172.21.15.103:0/1451628407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 msgr2=0x5596a5850aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a5850aa0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fc69402ade0 tx=0x7fc69402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 -- 172.21.15.103:0/1451628407 shutdown_connections 2024-05-23T00:01:12.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a5850aa0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5596a5862ea0 0x5596a5865290 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a5850560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 --2- 172.21.15.103:0/1451628407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848110 0x5596a5850020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc6a637b740 1 -- 172.21.15.103:0/1451628407 >> 172.21.15.103:0/1451628407 conn(0x5596a58443d0 msgr2=0x5596a584a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.537+0000 7fc682000640 0 client.15627 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-23T00:01:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.538+0000 7fc6a637b740 1 -- 172.21.15.103:0/1451628407 shutdown_connections 2024-05-23T00:01:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.538+0000 7fc6a637b740 1 -- 172.21.15.103:0/1451628407 wait complete. 2024-05-23T00:01:12.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.538+0000 7fc6a637b740 20 client.15627 trim_cache size 0 max 16384 2024-05-23T00:01:12.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.552+0000 7fc6a637b740 1 Processor -- start 2024-05-23T00:01:12.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.552+0000 7fc6a637b740 1 -- start start 2024-05-23T00:01:12.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848110 0x5596a5843930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a585ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5853130 0x5596a586a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a637b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1b3400 con 0x5596a5848110 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a637b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1b0cd0 con 0x5596a5853130 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a637b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1aeb20 con 0x5596a5848ac0 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848110 0x5596a5843930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848110 0x5596a5843930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41418/0 (socket says 172.21.15.103:41418) 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 -- 172.21.15.103:0/3914679519 learned_addr learned my addr 172.21.15.103:0/3914679519 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc69be00640 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a585ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5853130 0x5596a586a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 -- 172.21.15.103:0/3914679519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 msgr2=0x5596a585ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a585ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 -- 172.21.15.103:0/3914679519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5853130 msgr2=0x5596a586a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5853130 0x5596a586a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc6a1400640 1 -- 172.21.15.103:0/3914679519 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596a54c03f0 con 0x5596a5848110 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.553+0000 7fc69be00640 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a585ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.554+0000 7fc6a1400640 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848110 0x5596a5843930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc69402b0b0 tx=0x7fc69402b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd42328635008fc4 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.554+0000 7fc68d200640 1 -- 172.21.15.103:0/3914679519 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc69403f040 con 0x5596a5848110 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.554+0000 7fc68d200640 1 -- 172.21.15.103:0/3914679519 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc694036070 con 0x5596a5848110 2024-05-23T00:01:12.559 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.554+0000 7fc68d200640 1 -- 172.21.15.103:0/3914679519 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694030aa0 con 0x5596a5848110 2024-05-23T00:01:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:12 smithi105 ceph-mon[36426]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-05-23T00:01:12.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:12 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3498605230' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:01:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.554+0000 7fc6a637b740 1 -- 172.21.15.103:0/3914679519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848110 msgr2=0x5596a5843930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.554+0000 7fc6a637b740 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848110 0x5596a5843930 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc69402b0b0 tx=0x7fc69402b040 comp rx=0 tx=0).stop 2024-05-23T00:01:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 -- 172.21.15.103:0/3914679519 shutdown_connections 2024-05-23T00:01:12.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5853130 0x5596a586a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848ac0 0x5596a585ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 --2- 172.21.15.103:0/3914679519 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5848110 0x5596a5843930 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 -- 172.21.15.103:0/3914679519 >> 172.21.15.103:0/3914679519 conn(0x5596a58443d0 msgr2=0x5596a584a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 -- 172.21.15.103:0/3914679519 shutdown_connections 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 -- 172.21.15.103:0/3914679519 wait complete. 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 -- start start 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.555+0000 7fc6a637b740 1 Processor -- start 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848110 0x5596a585b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848ac0 0x5596a584cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a637b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a584d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a637b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1af190 con 0x5596a5853130 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a637b740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1b0600 con 0x5596a5848ac0 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a637b740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1af860 con 0x5596a5848110 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a637b740 10 client.0 _ll_register_callbacks cb 0x7fc6900d3790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a584d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848110 0x5596a585b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc69be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848ac0 0x5596a584cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc69be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848ac0 0x5596a584cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:42128/0 (socket says 172.21.15.103:42128) 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc6a1400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848110 0x5596a585b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:55182/0 (socket says 172.21.15.103:55182) 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.556+0000 7fc69be00640 1 -- 172.21.15.103:0/476465233 learned_addr learned my addr 172.21.15.103:0/476465233 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc69be00640 1 -- 172.21.15.103:0/476465233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848110 msgr2=0x5596a585b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.566 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc69be00640 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848110 0x5596a585b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc69be00640 1 -- 172.21.15.103:0/476465233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 msgr2=0x5596a584d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc69be00640 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a584d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc69be00640 1 -- 172.21.15.103:0/476465233 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596a558a7d0 con 0x5596a5848ac0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc69be00640 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848ac0 0x5596a584cac0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc688009d00 tx=0x7fc6880128e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eef11310ee74fd25 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc682000640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc682000640 1 -- 172.21.15.103:0/476465233 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68801e880 con 0x5596a5848ac0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc6a637b740 1 -- 172.21.15.103:0/476465233 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc694036070 con 0x5596a5848ac0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc682000640 1 -- 172.21.15.103:0/476465233 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7fc688010430 con 0x5596a5848ac0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc682000640 1 -- 172.21.15.103:0/476465233 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc68802d050 con 0x5596a5848ac0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc6a637b740 10 client.25739 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.557+0000 7fc6a637b740 1 -- 172.21.15.103:0/476465233 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5596a58d19d0 con 0x5596a5848ac0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc682000640 1 -- 172.21.15.103:0/476465233 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc694036070 con 0x5596a5848ac0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc682000640 1 -- 172.21.15.103:0/476465233 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc68800f870 con 0x5596a5848ac0 2024-05-23T00:01:12.567 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc6a637b740 10 client.25739 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc6a637b740 10 client.25739 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc6a637b740 20 client.25739 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc6a637b740 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5596a58d2130 0x5596a58d4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc6a637b740 4 client.25739 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc6a637b740 1 -- 172.21.15.103:0/476465233 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596a57b45a0 con 0x5596a58d2130 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.558+0000 7fc6a1400640 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5596a58d2130 0x5596a58d4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a1400640 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5596a58d2130 0x5596a58d4550 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6fa30c62b4b19e2 server_cookie=7532fe74f8f52d48 in_seq=0 out_seq=0 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc682000640 10 client.25739 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc682000640 1 -- 172.21.15.103:0/476465233 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596a57b45a0 con 0x5596a58d2130 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc682000640 10 client.25739 handle_command_reply: tid=0 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 client.25739 shutdown 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 -- 172.21.15.103:0/476465233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848ac0 msgr2=0x5596a584cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.568 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848ac0 0x5596a584cac0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc688009d00 tx=0x7fc6880128e0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 -- 172.21.15.103:0/476465233 shutdown_connections 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596a5853130 0x5596a584d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5596a5848ac0 0x5596a584cac0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x5596a58d2130 0x5596a58d4550 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 --2- 172.21.15.103:0/476465233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5596a5848110 0x5596a585b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc682000640 0 client.25739 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 -- 172.21.15.103:0/476465233 >> 172.21.15.103:0/476465233 conn(0x5596a58443d0 msgr2=0x5596a584ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 -- 172.21.15.103:0/476465233 shutdown_connections 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.559+0000 7fc6a637b740 1 -- 172.21.15.103:0/476465233 wait complete. 2024-05-23T00:01:12.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a637b740 20 client.25739 trim_cache size 0 max 16384 2024-05-23T00:01:12.570 INFO:teuthology.orchestra.run.smithi103.stdout:[] 2024-05-23T00:01:12.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 -- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc6840a2ea0 msgr2=0x7fc6840a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc6840a2ea0 0x7fc6840a5360 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc6900cd0c0 tx=0x7fc6900d38c0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 -- 172.21.15.103:0/2658426607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c100520 msgr2=0x7fc69c19a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c100520 0x7fc69c19a280 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc69000e9f0 tx=0x7fc69000edf0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 -- 172.21.15.103:0/2658426607 shutdown_connections 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc6840a2ea0 0x7fc6840a5360 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc69c10e0d0 0x7fc69c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc69c100ed0 0x7fc69c19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 --2- 172.21.15.103:0/2658426607 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc69c100520 0x7fc69c19a280 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 -- 172.21.15.103:0/2658426607 >> 172.21.15.103:0/2658426607 conn(0x7fc69c0fc160 msgr2=0x7fc69c0fe550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 -- 172.21.15.103:0/2658426607 shutdown_connections 2024-05-23T00:01:12.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.560+0000 7fc6a1e00640 1 -- 172.21.15.103:0/2658426607 wait complete. 2024-05-23T00:01:12.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-23T00:01:12.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 -- 172.21.15.103:0/1916425704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08680745c0 msgr2=0x7f086810da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 --2- 172.21.15.103:0/1916425704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08680745c0 0x7f086810da10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0854009900 tx=0x7f085402a480 comp rx=0 tx=0).stop 2024-05-23T00:01:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 -- 172.21.15.103:0/1916425704 shutdown_connections 2024-05-23T00:01:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 --2- 172.21.15.103:0/1916425704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f086810df50 0x7f08681103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 --2- 172.21.15.103:0/1916425704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08680745c0 0x7f086810da10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 --2- 172.21.15.103:0/1916425704 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0868073ca0 0x7f0868074080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 -- 172.21.15.103:0/1916425704 >> 172.21.15.103:0/1916425704 conn(0x7f08680fd240 msgr2=0x7f08680ff660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 -- 172.21.15.103:0/1916425704 shutdown_connections 2024-05-23T00:01:12.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.772+0000 7f086ea00640 1 -- 172.21.15.103:0/1916425704 wait complete. 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.773+0000 7f086ea00640 1 Processor -- start 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.773+0000 7f086ea00640 1 -- start start 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0868073ca0 0x7f086819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086ea00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08680745c0 0x7f086819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086810df50 0x7f086819ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0868113d40 con 0x7f0868073ca0 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086ea00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0868113bc0 con 0x7f086810df50 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086ea00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0868113ec0 con 0x7f08680745c0 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0868073ca0 0x7f086819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0868073ca0 0x7f086819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41452/0 (socket says 172.21.15.103:41452) 2024-05-23T00:01:12.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086d600640 1 -- 172.21.15.103:0/3674117309 learned_addr learned my addr 172.21.15.103:0/3674117309 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086e000640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086810df50 0x7f086819ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086d600640 1 -- 172.21.15.103:0/3674117309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08680745c0 msgr2=0x7f086819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086d600640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08680745c0 0x7f086819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086d600640 1 -- 172.21.15.103:0/3674117309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086810df50 msgr2=0x7f086819ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086d600640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086810df50 0x7f086819ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.774+0000 7f086d600640 1 -- 172.21.15.103:0/3674117309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086819f200 con 0x7f0868073ca0 2024-05-23T00:01:12.779 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.775+0000 7f086d600640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0868073ca0 0x7f086819a260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f085c00e9f0 tx=0x7f085c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.775+0000 7f0866000640 1 -- 172.21.15.103:0/3674117309 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c00cd80 con 0x7f0868073ca0 2024-05-23T00:01:12.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.775+0000 7f0866000640 1 -- 172.21.15.103:0/3674117309 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f085c005500 con 0x7f0868073ca0 2024-05-23T00:01:12.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.775+0000 7f086ea00640 1 -- 172.21.15.103:0/3674117309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086819f4f0 con 0x7f0868073ca0 2024-05-23T00:01:12.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.775+0000 7f0866000640 1 -- 172.21.15.103:0/3674117309 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c010690 con 0x7f0868073ca0 2024-05-23T00:01:12.780 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.775+0000 7f086ea00640 1 -- 172.21.15.103:0/3674117309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08681a6d80 con 0x7f0868073ca0 2024-05-23T00:01:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0866000640 1 -- 172.21.15.103:0/3674117309 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f085c0046c0 con 0x7f0868073ca0 2024-05-23T00:01:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0872fda740 1 Processor -- start 2024-05-23T00:01:12.782 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0872fda740 1 -- start start 2024-05-23T00:01:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0872fda740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.783 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0872fda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 0x563fa42cdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0872fda740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0872fda740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563fa4227b20 con 0x563fa42c3110 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0872fda740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563fa41cf1b0 con 0x563fa42ce130 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.779+0000 7f0872fda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563fa4223ae0 con 0x563fa42c3ac0 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086cc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086e000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086cc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42d0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:42144/0 (socket says 172.21.15.103:42144) 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086d600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 0x563fa42cdb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086cc00640 1 -- 172.21.15.103:0/2376130080 learned_addr learned my addr 172.21.15.103:0/2376130080 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086cc00640 1 -- 172.21.15.103:0/2376130080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 msgr2=0x563fa42cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086cc00640 1 --2- 172.21.15.103:0/2376130080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 0x563fa42cdb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086cc00640 1 -- 172.21.15.103:0/2376130080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 msgr2=0x563fa42c34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.784 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086cc00640 1 --2- 172.21.15.103:0/2376130080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42c34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.780+0000 7f086cc00640 1 -- 172.21.15.103:0/2376130080 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fa428b610 con 0x563fa42ce130 2024-05-23T00:01:12.785 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.781+0000 7f086cc00640 1 --2- 172.21.15.103:0/2376130080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42d0580 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f08540314d0 tx=0x7f0854031500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4009564049d6f6c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.781+0000 7f0866000640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f08500a2e20 0x7f08500a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.781+0000 7f085a600640 1 -- 172.21.15.103:0/2376130080 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085403e030 con 0x563fa42ce130 2024-05-23T00:01:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.781+0000 7f085a600640 1 -- 172.21.15.103:0/2376130080 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08540035f0 con 0x563fa42ce130 2024-05-23T00:01:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.781+0000 7f0866000640 1 -- 172.21.15.103:0/3674117309 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f085c0cff20 con 0x7f0868073ca0 2024-05-23T00:01:12.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.781+0000 7f085a600640 1 -- 172.21.15.103:0/2376130080 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854035040 con 0x563fa42ce130 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.781+0000 7f086e000640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f08500a2e20 0x7f08500a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 -- 172.21.15.103:0/2376130080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 msgr2=0x563fa42d0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 --2- 172.21.15.103:0/2376130080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42d0580 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f08540314d0 tx=0x7f0854031500 comp rx=0 tx=0).stop 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f086e000640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f08500a2e20 0x7f08500a52e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0860005fd0 tx=0x7f0860009500 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 -- 172.21.15.103:0/2376130080 shutdown_connections 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 --2- 172.21.15.103:0/2376130080 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42d0580 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 --2- 172.21.15.103:0/2376130080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 0x563fa42cdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 --2- 172.21.15.103:0/2376130080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42c34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 -- 172.21.15.103:0/2376130080 >> 172.21.15.103:0/2376130080 conn(0x563fa42bf3d0 msgr2=0x563fa42bf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 -- 172.21.15.103:0/2376130080 shutdown_connections 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.782+0000 7f0872fda740 1 -- 172.21.15.103:0/2376130080 wait complete. 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 1 -- start start 2024-05-23T00:01:12.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 1 Processor -- start 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42c3ac0 0x563fa42cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42ce130 0x563fa42cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563fa42d1040 con 0x563fa42c3110 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x563fa41e2600 con 0x563fa42c3ac0 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x563fa42d11c0 con 0x563fa42ce130 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.783+0000 7f0872fda740 10 client.0 _ll_register_callbacks cb 0x563fa421d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086cc00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42c3ac0 0x563fa42cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41472/0 (socket says 172.21.15.103:41472) 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086d600640 1 -- 172.21.15.103:0/2500743504 learned_addr learned my addr 172.21.15.103:0/2500743504 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086cc00640 1 -- 172.21.15.103:0/2500743504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42ce130 msgr2=0x563fa42cb9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-23T00:01:12.789 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086cc00640 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42ce130 0x563fa42cb9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086cc00640 1 -- 172.21.15.103:0/2500743504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 msgr2=0x563fa42caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086cc00640 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086cc00640 1 -- 172.21.15.103:0/2500743504 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fa41efa90 con 0x563fa42c3ac0 2024-05-23T00:01:12.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086d600640 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:12.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f086cc00640 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42c3ac0 0x563fa42cb490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0854005c40 tx=0x7f08540043b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8651104e4ec8f5dd server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f084ea00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:12.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.784+0000 7f084ea00640 1 -- 172.21.15.103:0/2500743504 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0854047070 con 0x563fa42c3ac0 2024-05-23T00:01:12.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f084ea00640 1 -- 172.21.15.103:0/2500743504 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f08540039a0 con 0x563fa42c3ac0 2024-05-23T00:01:12.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f0872fda740 1 -- 172.21.15.103:0/2500743504 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f08540035f0 con 0x563fa42c3ac0 2024-05-23T00:01:12.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f084ea00640 1 -- 172.21.15.103:0/2500743504 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0854055050 con 0x563fa42c3ac0 2024-05-23T00:01:12.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f0872fda740 10 client.25751 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f0872fda740 1 -- 172.21.15.103:0/2500743504 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563fa41d70d0 con 0x563fa42c3ac0 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f084ea00640 1 -- 172.21.15.103:0/2500743504 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08540035f0 con 0x563fa42c3ac0 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f084ea00640 1 -- 172.21.15.103:0/2500743504 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f085404c460 con 0x563fa42c3ac0 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f0872fda740 10 client.25751 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.785+0000 7f0872fda740 10 client.25751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.786+0000 7f0872fda740 20 client.25751 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.786+0000 7f0872fda740 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x563fa42ddff0 0x563fa42e03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.786+0000 7f0872fda740 4 client.25751 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.786+0000 7f0872fda740 1 -- 172.21.15.103:0/2500743504 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563fa4235250 con 0x563fa42ddff0 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.786+0000 7f086d600640 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x563fa42ddff0 0x563fa42e03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.786+0000 7f086d600640 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x563fa42ddff0 0x563fa42e03e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88cd516d9c06ea05 server_cookie=a4dfb926246cad4b in_seq=0 out_seq=0 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.786+0000 7f084ea00640 10 client.25751 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f084ea00640 1 -- 172.21.15.103:0/2500743504 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563fa4235250 con 0x563fa42ddff0 2024-05-23T00:01:12.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f084ea00640 10 client.25751 handle_command_reply: tid=0 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 client.25751 shutdown 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 -- 172.21.15.103:0/2500743504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42c3ac0 msgr2=0x563fa42cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42c3ac0 0x563fa42cb490 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0854005c40 tx=0x7f08540043b0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 -- 172.21.15.103:0/2500743504 shutdown_connections 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x563fa42ddff0 0x563fa42e03e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42ce130 0x563fa42cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42c3ac0 0x563fa42cb490 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f084ea00640 0 client.25751 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 --2- 172.21.15.103:0/2500743504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.787+0000 7f0872fda740 1 -- 172.21.15.103:0/2500743504 >> 172.21.15.103:0/2500743504 conn(0x563fa42bf3d0 msgr2=0x563fa42be6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.788+0000 7f0872fda740 1 -- 172.21.15.103:0/2500743504 shutdown_connections 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.788+0000 7f0872fda740 1 -- 172.21.15.103:0/2500743504 wait complete. 2024-05-23T00:01:12.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.788+0000 7f0872fda740 20 client.25751 trim_cache size 0 max 16384 2024-05-23T00:01:12.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f0872fda740 1 Processor -- start 2024-05-23T00:01:12.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f0872fda740 1 -- start start 2024-05-23T00:01:12.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f0872fda740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42bee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f0872fda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 0x563fa42c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f0872fda740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42c8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f0872fda740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08681b3480 con 0x563fa42c3110 2024-05-23T00:01:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f0872fda740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08681b0d50 con 0x563fa42ce130 2024-05-23T00:01:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f0872fda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f08681aeba0 con 0x563fa42c3ac0 2024-05-23T00:01:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f086d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42bee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f086d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42bee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41488/0 (socket says 172.21.15.103:41488) 2024-05-23T00:01:12.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f086d600640 1 -- 172.21.15.103:0/1049221554 learned_addr learned my addr 172.21.15.103:0/1049221554 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.803+0000 7f086e000640 1 --2- 172.21.15.103:0/1049221554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f086d600640 1 -- 172.21.15.103:0/1049221554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 msgr2=0x563fa42c8230 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-23T00:01:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f086d600640 1 --2- 172.21.15.103:0/1049221554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 0x563fa42c8230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f086d600640 1 -- 172.21.15.103:0/1049221554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 msgr2=0x563fa42c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f086d600640 1 --2- 172.21.15.103:0/1049221554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f086d600640 1 -- 172.21.15.103:0/1049221554 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563fa3f3b3f0 con 0x563fa42c3110 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f086d600640 1 --2- 172.21.15.103:0/1049221554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42bee70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f085c098880 tx=0x7f085c0d3a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5c17873eab8d8c server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f0859c00640 1 -- 172.21.15.103:0/1049221554 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c098150 con 0x563fa42c3110 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f0859c00640 1 -- 172.21.15.103:0/1049221554 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f085c09c5b0 con 0x563fa42c3110 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.804+0000 7f0859c00640 1 -- 172.21.15.103:0/1049221554 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c09b810 con 0x563fa42c3110 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 -- 172.21.15.103:0/1049221554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 msgr2=0x563fa42bee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 --2- 172.21.15.103:0/1049221554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42bee70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f085c098880 tx=0x7f085c0d3a30 comp rx=0 tx=0).stop 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 -- 172.21.15.103:0/1049221554 shutdown_connections 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 --2- 172.21.15.103:0/1049221554 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 --2- 172.21.15.103:0/1049221554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3ac0 0x563fa42c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 --2- 172.21.15.103:0/1049221554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3110 0x563fa42bee70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 -- 172.21.15.103:0/1049221554 >> 172.21.15.103:0/1049221554 conn(0x563fa42bf3d0 msgr2=0x563fa42c9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 -- 172.21.15.103:0/1049221554 shutdown_connections 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 -- 172.21.15.103:0/1049221554 wait complete. 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.805+0000 7f0872fda740 1 -- start start 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f0872fda740 1 Processor -- start 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f0872fda740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3110 0x563fa42e38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f0872fda740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3ac0 0x563fa42e0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f0872fda740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42e1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f0872fda740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08681af210 con 0x563fa42c3ac0 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f0872fda740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f08681b0680 con 0x563fa42ce130 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f0872fda740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f08681af8e0 con 0x563fa42c3110 2024-05-23T00:01:12.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f0872fda740 10 client.0 _ll_register_callbacks cb 0x7f0860009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f086cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3ac0 0x563fa42e0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f086e000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42e1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f086cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3ac0 0x563fa42e0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41502/0 (socket says 172.21.15.103:41502) 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.806+0000 7f086cc00640 1 -- 172.21.15.103:0/3834694865 learned_addr learned my addr 172.21.15.103:0/3834694865 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f086d600640 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3110 0x563fa42e38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f086e000640 1 -- 172.21.15.103:0/3834694865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3110 msgr2=0x563fa42e38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f086e000640 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3110 0x563fa42e38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f086e000640 1 -- 172.21.15.103:0/3834694865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3ac0 msgr2=0x563fa42e0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f086e000640 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3ac0 0x563fa42e0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f086e000640 1 -- 172.21.15.103:0/3834694865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563fa40057d0 con 0x563fa42ce130 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f086cc00640 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3ac0 0x563fa42e0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f086e000640 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42e1270 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x563fa42bb060 tx=0x7f0860004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e1fe521c77a6b72 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:12.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f084ea00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f084ea00640 1 -- 172.21.15.103:0/3834694865 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0860004b20 con 0x563fa42ce130 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f0872fda740 1 -- 172.21.15.103:0/3834694865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f085c09c5b0 con 0x563fa42ce130 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.807+0000 7f084ea00640 1 -- 172.21.15.103:0/3834694865 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f0860017030 con 0x563fa42ce130 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f084ea00640 1 -- 172.21.15.103:0/3834694865 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f0860004cc0 con 0x563fa42ce130 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f0872fda740 10 client.25757 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f0872fda740 1 -- 172.21.15.103:0/3834694865 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563fa4344160 con 0x563fa42ce130 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f084ea00640 1 -- 172.21.15.103:0/3834694865 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f085c09c5b0 con 0x563fa42ce130 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f084ea00640 1 -- 172.21.15.103:0/3834694865 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f0860005470 con 0x563fa42ce130 2024-05-23T00:01:12.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f0872fda740 10 client.25757 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f0872fda740 10 client.25757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f0872fda740 20 client.25757 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f0872fda740 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x563fa4344900 0x563fa4346d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f0872fda740 4 client.25757 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.808+0000 7f0872fda740 1 -- 172.21.15.103:0/3834694865 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563fa422f5a0 con 0x563fa4344900 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f086d600640 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x563fa4344900 0x563fa4346d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f086d600640 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x563fa4344900 0x563fa4346d20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e5453884b2055e7 server_cookie=aba68df3d8cba1cb in_seq=0 out_seq=0 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f084ea00640 10 client.25757 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f084ea00640 1 -- 172.21.15.103:0/3834694865 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563fa422f5a0 con 0x563fa4344900 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f084ea00640 10 client.25757 handle_command_reply: tid=0 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 client.25757 shutdown 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 -- 172.21.15.103:0/3834694865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 msgr2=0x563fa42e1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42e1270 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x563fa42bb060 tx=0x7f0860004ab0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 -- 172.21.15.103:0/3834694865 shutdown_connections 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x563fa4344900 0x563fa4346d20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563fa42ce130 0x563fa42e1270 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f084ea00640 0 client.25757 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563fa42c3ac0 0x563fa42e0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 --2- 172.21.15.103:0/3834694865 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x563fa42c3110 0x563fa42e38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.809+0000 7f0872fda740 1 -- 172.21.15.103:0/3834694865 >> 172.21.15.103:0/3834694865 conn(0x563fa42bf3d0 msgr2=0x563fa42d9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f0872fda740 1 -- 172.21.15.103:0/3834694865 shutdown_connections 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f0872fda740 1 -- 172.21.15.103:0/3834694865 wait complete. 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f0872fda740 20 client.25757 trim_cache size 0 max 16384 2024-05-23T00:01:12.819 INFO:teuthology.orchestra.run.smithi103.stdout:[] 2024-05-23T00:01:12.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 -- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f08500a2e20 msgr2=0x7f08500a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f08500a2e20 0x7f08500a52e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0860005fd0 tx=0x7f0860009500 comp rx=0 tx=0).stop 2024-05-23T00:01:12.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 -- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0868073ca0 msgr2=0x7f086819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:12.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0868073ca0 0x7f086819a260 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f085c00e9f0 tx=0x7f085c00edf0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 -- 172.21.15.103:0/3674117309 shutdown_connections 2024-05-23T00:01:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f08500a2e20 0x7f08500a52e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f086810df50 0x7f086819ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f08680745c0 0x7f086819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 --2- 172.21.15.103:0/3674117309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0868073ca0 0x7f086819a260 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.810+0000 7f086ea00640 1 -- 172.21.15.103:0/3674117309 >> 172.21.15.103:0/3674117309 conn(0x7f08680fd240 msgr2=0x7f08680fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.811+0000 7f086ea00640 1 -- 172.21.15.103:0/3674117309 shutdown_connections 2024-05-23T00:01:12.821 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:12.811+0000 7f086ea00640 1 -- 172.21.15.103:0/3674117309 wait complete. 2024-05-23T00:01:13.833 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-23T00:01:13.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:01:13.834 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.046+0000 7f24c9000640 1 -- 172.21.15.103:0/584537398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c410de30 msgr2=0x7f24c410e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.046+0000 7f24c9000640 1 --2- 172.21.15.103:0/584537398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c410de30 0x7f24c410e210 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009900 tx=0x7f24b402a480 comp rx=0 tx=0).stop 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.046+0000 7f24c9000640 1 -- 172.21.15.103:0/584537398 shutdown_connections 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.046+0000 7f24c9000640 1 --2- 172.21.15.103:0/584537398 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24c41087b0 0x7f24c4114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.046+0000 7f24c9000640 1 --2- 172.21.15.103:0/584537398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24c4107e10 0x7f24c4108270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.046+0000 7f24c9000640 1 --2- 172.21.15.103:0/584537398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c410de30 0x7f24c410e210 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.046+0000 7f24c9000640 1 -- 172.21.15.103:0/584537398 >> 172.21.15.103:0/584537398 conn(0x7f24c4077690 msgr2=0x7f24c4106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.046+0000 7f24c9000640 1 -- 172.21.15.103:0/584537398 shutdown_connections 2024-05-23T00:01:14.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.047+0000 7f24c9000640 1 -- 172.21.15.103:0/584537398 wait complete. 2024-05-23T00:01:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.048+0000 7f24c9000640 1 Processor -- start 2024-05-23T00:01:14.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.048+0000 7f24c9000640 1 -- start start 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c9000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24c4107e10 0x7f24c419e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c9000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c41087b0 0x7f24c419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24c410de30 0x7f24c41a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c9000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24c4118110 con 0x7f24c41087b0 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c9000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f24c4117f90 con 0x7f24c410de30 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c9000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24c4118290 con 0x7f24c4107e10 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c41087b0 0x7f24c419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c41087b0 0x7f24c419ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48712/0 (socket says 172.21.15.103:48712) 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c2a00640 1 -- 172.21.15.103:0/510706673 learned_addr learned my addr 172.21.15.103:0/510706673 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c3e00640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24c410de30 0x7f24c41a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.053 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c3400640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24c4107e10 0x7f24c419e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c2a00640 1 -- 172.21.15.103:0/510706673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24c4107e10 msgr2=0x7f24c419e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c2a00640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24c4107e10 0x7f24c419e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c2a00640 1 -- 172.21.15.103:0/510706673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24c410de30 msgr2=0x7f24c41a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c2a00640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24c410de30 0x7f24c41a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.049+0000 7f24c2a00640 1 -- 172.21.15.103:0/510706673 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c41a3530 con 0x7f24c41087b0 2024-05-23T00:01:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.050+0000 7f24c3e00640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24c410de30 0x7f24c41a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:14.054 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.050+0000 7f24c2a00640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c41087b0 0x7f24c419ead0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f24b000e9f0 tx=0x7f24b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.050+0000 7f24bbe00640 1 -- 172.21.15.103:0/510706673 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b000cd80 con 0x7f24c41087b0 2024-05-23T00:01:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.050+0000 7f24bbe00640 1 -- 172.21.15.103:0/510706673 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b0010910 con 0x7f24c41087b0 2024-05-23T00:01:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.050+0000 7f24c9000640 1 -- 172.21.15.103:0/510706673 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c41a3820 con 0x7f24c41087b0 2024-05-23T00:01:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.050+0000 7f24bbe00640 1 -- 172.21.15.103:0/510706673 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0018990 con 0x7f24c41087b0 2024-05-23T00:01:14.055 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.050+0000 7f24c9000640 1 -- 172.21.15.103:0/510706673 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c41ab0b0 con 0x7f24c41087b0 2024-05-23T00:01:14.056 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.053+0000 7f24cd3ab740 1 Processor -- start 2024-05-23T00:01:14.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24cd3ab740 1 -- start start 2024-05-23T00:01:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396150750 0x564396150b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x56439615b200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x56439615db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5643960b47c0 con 0x564396150750 2024-05-23T00:01:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56439605c270 con 0x56439615b740 2024-05-23T00:01:14.058 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5643960b0780 con 0x564396151070 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24bbe00640 1 -- 172.21.15.103:0/510706673 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f24b0010ac0 con 0x7f24c41087b0 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24c3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396150750 0x564396150b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24c3400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x56439615b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24c3400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x56439615b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34452/0 (socket says 172.21.15.103:34452) 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.054+0000 7f24c3400640 1 -- 172.21.15.103:0/1466269579 learned_addr learned my addr 172.21.15.103:0/1466269579 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c2a00640 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x56439615db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c3e00640 1 -- 172.21.15.103:0/1466269579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 msgr2=0x56439615b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c3e00640 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x56439615b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c3e00640 1 -- 172.21.15.103:0/1466269579 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 msgr2=0x56439615db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c3e00640 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x56439615db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c3e00640 1 -- 172.21.15.103:0/1466269579 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564395e907e0 con 0x564396150750 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c3400640 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x56439615b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:14.059 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c2a00640 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x56439615db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:01:14.060 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24c3e00640 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396150750 0x564396150b30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f24bc011760 tx=0x7f24bc011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b4fc33ab61bb1 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.063 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.055+0000 7f24b8c00640 1 -- 172.21.15.103:0/1466269579 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc004d20 con 0x564396150750 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.056+0000 7f24b8c00640 1 -- 172.21.15.103:0/1466269579 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24bc00f940 con 0x564396150750 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.056+0000 7f24b8c00640 1 -- 172.21.15.103:0/1466269579 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc004450 con 0x564396150750 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.056+0000 7f24bbe00640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f24a80a2ea0 0x7f24a80a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 -- 172.21.15.103:0/1466269579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396150750 msgr2=0x564396150b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396150750 0x564396150b30 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f24bc011760 tx=0x7f24bc011c20 comp rx=0 tx=0).stop 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24bbe00640 1 -- 172.21.15.103:0/510706673 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f24b00cfdd0 con 0x7f24c41087b0 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24c3400640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f24a80a2ea0 0x7f24a80a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 -- 172.21.15.103:0/1466269579 shutdown_connections 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x56439615db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x56439615b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/1466269579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396150750 0x564396150b30 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.064 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 -- 172.21.15.103:0/1466269579 >> 172.21.15.103:0/1466269579 conn(0x56439614ca70 msgr2=0x56439614ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 -- 172.21.15.103:0/1466269579 shutdown_connections 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24cd3ab740 1 -- 172.21.15.103:0/1466269579 wait complete. 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.057+0000 7f24c3400640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f24a80a2ea0 0x7f24a80a5360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f24b40096c0 tx=0x7f24b40058b0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 1 -- start start 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 1 Processor -- start 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396158660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x564396158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x5643961590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56439615e530 con 0x564396151070 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56439606f620 con 0x56439615b740 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56439615e6b0 con 0x564396150750 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.058+0000 7f24cd3ab740 10 client.0 _ll_register_callbacks cb 0x56439605d8b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:14.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x564396158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c3400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c3400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34464/0 (socket says 172.21.15.103:34464) 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c3400640 1 -- 172.21.15.103:0/2045277438 learned_addr learned my addr 172.21.15.103:0/2045277438 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c2a00640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x564396158ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48730/0 (socket says 172.21.15.103:48730) 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c3e00640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x5643961590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c2a00640 1 -- 172.21.15.103:0/2045277438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 msgr2=0x564396158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c2a00640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c2a00640 1 -- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 msgr2=0x5643961590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c2a00640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x5643961590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c2a00640 1 -- 172.21.15.103:0/2045277438 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564395ef0f70 con 0x564396151070 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c3400640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c3e00640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x5643961590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:14.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24c2a00640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x564396158ba0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f24b00cd000 tx=0x7f24b009b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834d04c6ddb7471a server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.059+0000 7f24ad600640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24ad600640 1 -- 172.21.15.103:0/2045277438 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b009bd00 con 0x564396151070 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24ad600640 1 -- 172.21.15.103:0/2045277438 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f24b0018b30 con 0x564396151070 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2045277438 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24bc00f940 con 0x564396151070 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24ad600640 1 -- 172.21.15.103:0/2045277438 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f24b00a00b0 con 0x564396151070 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 10 client.15663 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2045277438 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564395d88170 con 0x564396151070 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24ad600640 1 -- 172.21.15.103:0/2045277438 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24bc00f940 con 0x564396151070 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24ad600640 1 -- 172.21.15.103:0/2045277438 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f24b009b270 con 0x564396151070 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 10 client.15663 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 10 client.15663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 20 client.15663 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:14.067 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56439616b8e0 0x56439616dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 4 client.15663 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:14.068 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.060+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2045277438 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643960c81e0 con 0x56439616b8e0 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.061+0000 7f24c3400640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56439616b8e0 0x56439616dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.061+0000 7f24c3400640 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56439616b8e0 0x56439616dcd0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9db9bd27237e5140 server_cookie=658c319997e99347 in_seq=0 out_seq=0 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.061+0000 7f24ad600640 10 client.15663 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.062+0000 7f24ad600640 1 -- 172.21.15.103:0/2045277438 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5643960c81e0 con 0x56439616b8e0 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.062+0000 7f24ad600640 10 client.15663 handle_command_reply: tid=0 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.062+0000 7f24cd3ab740 1 client.15663 shutdown 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2045277438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 msgr2=0x564396158ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x564396158ba0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f24b00cd000 tx=0x7f24b009b930 comp rx=0 tx=0).stop 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2045277438 shutdown_connections 2024-05-23T00:01:14.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56439616b8e0 0x56439616dcd0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x5643961590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x564396158ba0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2045277438 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24ad600640 0 client.15663 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2045277438 >> 172.21.15.103:0/2045277438 conn(0x56439614ca70 msgr2=0x564396152770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2045277438 shutdown_connections 2024-05-23T00:01:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2045277438 wait complete. 2024-05-23T00:01:14.070 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.063+0000 7f24cd3ab740 20 client.15663 trim_cache size 0 max 16384 2024-05-23T00:01:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.082+0000 7f24cd3ab740 1 Processor -- start 2024-05-23T00:01:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24cd3ab740 1 -- start start 2024-05-23T00:01:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396167420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x5643961735a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x564396173ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24c41b7840 con 0x564396151070 2024-05-23T00:01:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f24c41b5110 con 0x56439615b740 2024-05-23T00:01:14.086 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24c41b2f60 con 0x564396150750 2024-05-23T00:01:14.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24c3400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396167420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.087 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24c2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x5643961735a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24c3400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396167420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34474/0 (socket says 172.21.15.103:34474) 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24c2a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x5643961735a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48744/0 (socket says 172.21.15.103:48744) 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24c3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x564396173ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.083+0000 7f24c3400640 1 -- 172.21.15.103:0/2018092584 learned_addr learned my addr 172.21.15.103:0/2018092584 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24c3e00640 1 -- 172.21.15.103:0/2018092584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 msgr2=0x564396167420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24c3e00640 1 --2- 172.21.15.103:0/2018092584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396167420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24c3e00640 1 -- 172.21.15.103:0/2018092584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 msgr2=0x5643961735a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24c3e00640 1 --2- 172.21.15.103:0/2018092584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x5643961735a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24c3e00640 1 -- 172.21.15.103:0/2018092584 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56439615dea0 con 0x56439615b740 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24c3400640 1 --2- 172.21.15.103:0/2018092584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396167420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24c3e00640 1 --2- 172.21.15.103:0/2018092584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x564396173ae0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f24bc004130 tx=0x7f24bc011460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57a2920f05bd45d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.088 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24afe00640 1 -- 172.21.15.103:0/2018092584 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc028030 con 0x56439615b740 2024-05-23T00:01:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24afe00640 1 -- 172.21.15.103:0/2018092584 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24bc01f040 con 0x56439615b740 2024-05-23T00:01:14.089 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.084+0000 7f24afe00640 1 -- 172.21.15.103:0/2018092584 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc005760 con 0x56439615b740 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2018092584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 msgr2=0x564396173ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2018092584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x564396173ae0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f24bc004130 tx=0x7f24bc011460 comp rx=0 tx=0).stop 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2018092584 shutdown_connections 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2018092584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56439615b740 0x564396173ae0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2018092584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564396151070 0x5643961735a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/2018092584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396150750 0x564396167420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2018092584 >> 172.21.15.103:0/2018092584 conn(0x56439614ca70 msgr2=0x56439614bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2018092584 shutdown_connections 2024-05-23T00:01:14.092 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.085+0000 7f24cd3ab740 1 -- 172.21.15.103:0/2018092584 wait complete. 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.086+0000 7f24cd3ab740 1 -- start start 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.086+0000 7f24cd3ab740 1 Processor -- start 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.086+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564396150750 0x564396161300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.086+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x564396161840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.086+0000 7f24cd3ab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56439615b740 0x564396161d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.086+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24c41b35d0 con 0x56439615b740 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.086+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f24c41b4a40 con 0x564396150750 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.086+0000 7f24cd3ab740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24c41b3ca0 con 0x564396151070 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24cd3ab740 10 client.0 _ll_register_callbacks cb 0x7f24b000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564396150750 0x564396161300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564396150750 0x564396161300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:38240/0 (socket says 172.21.15.103:38240) 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 -- 172.21.15.103:0/3380681822 learned_addr learned my addr 172.21.15.103:0/3380681822 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.093 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3e00640 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56439615b740 0x564396161d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c2a00640 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x564396161840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 -- 172.21.15.103:0/3380681822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 msgr2=0x564396161840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x564396161840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 -- 172.21.15.103:0/3380681822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56439615b740 msgr2=0x564396161d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56439615b740 0x564396161d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 -- 172.21.15.103:0/3380681822 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643961d0e20 con 0x564396150750 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24c3400640 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564396150750 0x564396161300 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009850 tx=0x7f24b403e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67c8cbe440624e9b server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.087+0000 7f24ad600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.088+0000 7f24ad600640 1 -- 172.21.15.103:0/3380681822 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4035040 con 0x564396150750 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.088+0000 7f24ad600640 1 -- 172.21.15.103:0/3380681822 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f24b4005240 con 0x564396150750 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.088+0000 7f24cd3ab740 1 -- 172.21.15.103:0/3380681822 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24bc01f040 con 0x564396150750 2024-05-23T00:01:14.094 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.088+0000 7f24ad600640 1 -- 172.21.15.103:0/3380681822 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f24b4061050 con 0x564396150750 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.088+0000 7f24cd3ab740 10 client.25772 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.088+0000 7f24cd3ab740 1 -- 172.21.15.103:0/3380681822 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5643961d2140 con 0x564396150750 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.088+0000 7f24ad600640 1 -- 172.21.15.103:0/3380681822 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24bc01f040 con 0x564396150750 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24ad600640 1 -- 172.21.15.103:0/3380681822 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f24b4004750 con 0x564396150750 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24cd3ab740 10 client.25772 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24cd3ab740 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24cd3ab740 20 client.25772 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5643961d28c0 0x5643961d4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24cd3ab740 4 client.25772 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24cd3ab740 1 -- 172.21.15.103:0/3380681822 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5643960c1ef0 con 0x5643961d28c0 2024-05-23T00:01:14.095 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24c2a00640 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5643961d28c0 0x5643961d4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24c2a00640 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5643961d28c0 0x5643961d4ce0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6497184465465f9 server_cookie=8869b9fc88be4ff5 in_seq=0 out_seq=0 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.089+0000 7f24ad600640 10 client.25772 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24ad600640 1 -- 172.21.15.103:0/3380681822 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5643960c1ef0 con 0x5643961d28c0 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24ad600640 10 client.25772 handle_command_reply: tid=0 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 client.25772 shutdown 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 -- 172.21.15.103:0/3380681822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564396150750 msgr2=0x564396161300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564396150750 0x564396161300 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f24b4009850 tx=0x7f24b403e070 comp rx=0 tx=0).stop 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 -- 172.21.15.103:0/3380681822 shutdown_connections 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5643961d28c0 0x5643961d4ce0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56439615b740 0x564396161d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564396151070 0x564396161840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.097 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 --2- 172.21.15.103:0/3380681822 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564396150750 0x564396161300 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 -- 172.21.15.103:0/3380681822 >> 172.21.15.103:0/3380681822 conn(0x56439614ca70 msgr2=0x56439614b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24ad600640 0 client.25772 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 -- 172.21.15.103:0/3380681822 shutdown_connections 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 1 -- 172.21.15.103:0/3380681822 wait complete. 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.091+0000 7f24cd3ab740 20 client.25772 trim_cache size 0 max 16384 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stdout: "return_code": 0, 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_tag": "7de8af36-27d4-4ecb-8338-ac63c1b54732", 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": "asynchronous" 2024-05-23T00:01:14.098 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:01:14.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 -- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f24a80a2ea0 msgr2=0x7f24a80a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f24a80a2ea0 0x7f24a80a5360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f24b40096c0 tx=0x7f24b40058b0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 -- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c41087b0 msgr2=0x7f24c419ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c41087b0 0x7f24c419ead0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f24b000e9f0 tx=0x7f24b000eeb0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.099 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 -- 172.21.15.103:0/510706673 shutdown_connections 2024-05-23T00:01:14.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f24a80a2ea0 0x7f24a80a5360 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24c410de30 0x7f24c41a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c41087b0 0x7f24c419ead0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 --2- 172.21.15.103:0/510706673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24c4107e10 0x7f24c419e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.092+0000 7f24c9000640 1 -- 172.21.15.103:0/510706673 >> 172.21.15.103:0/510706673 conn(0x7f24c4077690 msgr2=0x7f24c4078610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.093+0000 7f24c9000640 1 -- 172.21.15.103:0/510706673 shutdown_connections 2024-05-23T00:01:14.100 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.093+0000 7f24c9000640 1 -- 172.21.15.103:0/510706673 wait complete. 2024-05-23T00:01:14.114 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-23T00:01:14.114 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-23T00:01:14.181 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:14 smithi103 ceph-mon[31370]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-05-23T00:01:14.317 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 -- 172.21.15.103:0/3713074140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 msgr2=0x7f2f98110520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 --2- 172.21.15.103:0/3713074140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 0x7f2f98110520 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2f90009ca0 tx=0x7f2f9002a950 comp rx=0 tx=0).stop 2024-05-23T00:01:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 -- 172.21.15.103:0/3713074140 shutdown_connections 2024-05-23T00:01:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 --2- 172.21.15.103:0/3713074140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 0x7f2f98110520 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 --2- 172.21.15.103:0/3713074140 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f98074460 0x7f2f9810dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 --2- 172.21.15.103:0/3713074140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f98073b40 0x7f2f98073f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 -- 172.21.15.103:0/3713074140 >> 172.21.15.103:0/3713074140 conn(0x7f2f98078cc0 msgr2=0x7f2f980790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 -- 172.21.15.103:0/3713074140 shutdown_connections 2024-05-23T00:01:14.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.314+0000 7f2f9f600640 1 -- 172.21.15.103:0/3713074140 wait complete. 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.315+0000 7f2f9f600640 1 Processor -- start 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.315+0000 7f2f9f600640 1 -- start start 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9f600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f98073b40 0x7f2f9819a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9f600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f98074460 0x7f2f9819a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9f600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 0x7f2f9819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9f600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98113e80 con 0x7f2f9810e130 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9f600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98113d00 con 0x7f2f98074460 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9f600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f98114000 con 0x7f2f98073b40 2024-05-23T00:01:14.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 0x7f2f9819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9cc00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f98073b40 0x7f2f9819a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 0x7f2f9819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48762/0 (socket says 172.21.15.103:48762) 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9cc00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f98073b40 0x7f2f9819a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34490/0 (socket says 172.21.15.103:34490) 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f9d600640 1 -- 172.21.15.103:0/1764119266 learned_addr learned my addr 172.21.15.103:0/1764119266 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.316+0000 7f2f97e00640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f98074460 0x7f2f9819a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9cc00640 1 -- 172.21.15.103:0/1764119266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f98074460 msgr2=0x7f2f9819a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f98074460 0x7f2f9819a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9cc00640 1 -- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 msgr2=0x7f2f9819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 0x7f2f9819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9cc00640 1 -- 172.21.15.103:0/1764119266 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9819f2c0 con 0x7f2f98073b40 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9d600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 0x7f2f9819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:14.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f98073b40 0x7f2f9819a320 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00e9f0 tx=0x7f2f8c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f95600640 1 -- 172.21.15.103:0/1764119266 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c00ccc0 con 0x7f2f98073b40 2024-05-23T00:01:14.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9f600640 1 -- 172.21.15.103:0/1764119266 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9819f5b0 con 0x7f2f98073b40 2024-05-23T00:01:14.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f95600640 1 -- 172.21.15.103:0/1764119266 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f8c010910 con 0x7f2f98073b40 2024-05-23T00:01:14.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f95600640 1 -- 172.21.15.103:0/1764119266 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c018960 con 0x7f2f98073b40 2024-05-23T00:01:14.323 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.317+0000 7f2f9f600640 1 -- 172.21.15.103:0/1764119266 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f981a6e40 con 0x7f2f98073b40 2024-05-23T00:01:14.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2fa2406740 1 Processor -- start 2024-05-23T00:01:14.326 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2fa2406740 1 -- start start 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2f95600640 1 -- 172.21.15.103:0/1764119266 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f2f8c010ac0 con 0x7f2f98073b40 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e529a0 0x55bba8e52d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e5d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e5fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2fa2406740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bba8e16320 con 0x55bba8e5d990 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2fa2406740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bba8db7cb0 con 0x55bba8e529a0 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.322+0000 7f2fa2406740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bba8d5efc0 con 0x55bba8e532c0 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f9d600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e529a0 0x55bba8e52d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f9d600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e529a0 0x55bba8e52d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:38272/0 (socket says 172.21.15.103:38272) 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f9d600640 1 -- 172.21.15.103:0/2217923792 learned_addr learned my addr 172.21.15.103:0/2217923792 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f97e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e5fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.327 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e5d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f97e00640 1 -- 172.21.15.103:0/2217923792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 msgr2=0x55bba8e5d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f97e00640 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e5d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f97e00640 1 -- 172.21.15.103:0/2217923792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e529a0 msgr2=0x55bba8e52d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f97e00640 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e529a0 0x55bba8e52d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f97e00640 1 -- 172.21.15.103:0/2217923792 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bba8b957d0 con 0x55bba8e5d990 2024-05-23T00:01:14.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.323+0000 7f2f9d600640 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e529a0 0x55bba8e52d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:14.328 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.324+0000 7f2f97e00640 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e5fd80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009bf0 tx=0x7f2f84011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bce4ec8fbfef5c server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.324+0000 7f2f95600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2f800a2e20 0x7f2f800a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.332 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.324+0000 7f2f89c00640 1 -- 172.21.15.103:0/2217923792 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84004050 con 0x55bba8e5d990 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.324+0000 7f2f89c00640 1 -- 172.21.15.103:0/2217923792 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f84005500 con 0x55bba8e5d990 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.324+0000 7f2f89c00640 1 -- 172.21.15.103:0/2217923792 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84004d90 con 0x55bba8e5d990 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.324+0000 7f2f95600640 1 -- 172.21.15.103:0/1764119266 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2f8c0cfdc0 con 0x7f2f98073b40 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.324+0000 7f2f9d600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2f800a2e20 0x7f2f800a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 -- 172.21.15.103:0/2217923792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 msgr2=0x55bba8e5fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e5fd80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2f84009bf0 tx=0x7f2f84011810 comp rx=0 tx=0).stop 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 -- 172.21.15.103:0/2217923792 shutdown_connections 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e5fd80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e5d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 --2- 172.21.15.103:0/2217923792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e529a0 0x55bba8e52d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.333 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 -- 172.21.15.103:0/2217923792 >> 172.21.15.103:0/2217923792 conn(0x55bba8e4ecc0 msgr2=0x55bba8e4f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 -- 172.21.15.103:0/2217923792 shutdown_connections 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2f9d600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2f800a2e20 0x7f2f800a52e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f90003b10 tx=0x7f2f90005d80 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 -- 172.21.15.103:0/2217923792 wait complete. 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.325+0000 7f2fa2406740 1 -- start start 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2fa2406740 1 Processor -- start 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e5a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 0x55bba8e5ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e5d990 0x55bba8e5b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2fa2406740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bba8d72410 con 0x55bba8e529a0 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2fa2406740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bba8db3c70 con 0x55bba8e532c0 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2fa2406740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bba8e60780 con 0x55bba8e5d990 2024-05-23T00:01:14.334 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2fa2406740 10 client.0 _ll_register_callbacks cb 0x55bba8dae0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2f9cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e5a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.326+0000 7f2f97e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 0x55bba8e5ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e5a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48784/0 (socket says 172.21.15.103:48784) 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9cc00640 1 -- 172.21.15.103:0/1985473036 learned_addr learned my addr 172.21.15.103:0/1985473036 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9d600640 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e5d990 0x55bba8e5b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9cc00640 1 -- 172.21.15.103:0/1985473036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e5d990 msgr2=0x55bba8e5b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e5d990 0x55bba8e5b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9cc00640 1 -- 172.21.15.103:0/1985473036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 msgr2=0x55bba8e5ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 0x55bba8e5ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9cc00640 1 -- 172.21.15.103:0/1985473036 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bba8acb3f0 con 0x55bba8e529a0 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e5a8a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c09bd90 tx=0x7f2f8c0d3a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9eceb71fafb378 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.335 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f7e000640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.327+0000 7f2f7e000640 1 -- 172.21.15.103:0/1985473036 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c09b940 con 0x55bba8e529a0 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2f7e000640 1 -- 172.21.15.103:0/1985473036 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2f8c09b250 con 0x55bba8e529a0 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 1 -- 172.21.15.103:0/1985473036 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f84005500 con 0x55bba8e529a0 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2f7e000640 1 -- 172.21.15.103:0/1985473036 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2f8c09b760 con 0x55bba8e529a0 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 10 client.15681 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 1 -- 172.21.15.103:0/1985473036 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bba8a8b0f0 con 0x55bba8e529a0 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2f7e000640 1 -- 172.21.15.103:0/1985473036 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f84005500 con 0x55bba8e529a0 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2f7e000640 1 -- 172.21.15.103:0/1985473036 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2f8c018b90 con 0x55bba8e529a0 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 10 client.15681 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 10 client.15681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:14.336 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 20 client.15681 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bba8e6db80 0x55bba8e70010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 4 client.15681 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.328+0000 7f2fa2406740 1 -- 172.21.15.103:0/1985473036 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bba8d76e00 con 0x55bba8e6db80 2024-05-23T00:01:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.329+0000 7f2f97e00640 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bba8e6db80 0x55bba8e70010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.329+0000 7f2f97e00640 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bba8e6db80 0x55bba8e70010 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e76f6d4ceb27beca server_cookie=c7a9b4d78e9bea16 in_seq=0 out_seq=0 2024-05-23T00:01:14.337 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.329+0000 7f2f7e000640 10 client.15681 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2f7e000640 1 -- 172.21.15.103:0/1985473036 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bba8d76e00 con 0x55bba8e6db80 2024-05-23T00:01:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2f7e000640 10 client.15681 handle_command_reply: tid=0 2024-05-23T00:01:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 client.15681 shutdown 2024-05-23T00:01:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 -- 172.21.15.103:0/1985473036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 msgr2=0x55bba8e5a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e5a8a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c09bd90 tx=0x7f2f8c0d3a80 comp rx=0 tx=0).stop 2024-05-23T00:01:14.338 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 -- 172.21.15.103:0/1985473036 shutdown_connections 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bba8e6db80 0x55bba8e70010 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e5d990 0x55bba8e5b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 0x55bba8e5ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2f7e000640 0 client.15681 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 --2- 172.21.15.103:0/1985473036 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e5a8a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.330+0000 7f2fa2406740 1 -- 172.21.15.103:0/1985473036 >> 172.21.15.103:0/1985473036 conn(0x55bba8e4ecc0 msgr2=0x55bba8e54ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.331+0000 7f2fa2406740 1 -- 172.21.15.103:0/1985473036 shutdown_connections 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.331+0000 7f2fa2406740 1 -- 172.21.15.103:0/1985473036 wait complete. 2024-05-23T00:01:14.339 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.331+0000 7f2fa2406740 20 client.15681 trim_cache size 0 max 16384 2024-05-23T00:01:14.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.349+0000 7f2fa2406740 1 Processor -- start 2024-05-23T00:01:14.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.349+0000 7f2fa2406740 1 -- start start 2024-05-23T00:01:14.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e4e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e694c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e5d990 0x55bba8e75840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2fa2406740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f981b3760 con 0x55bba8e529a0 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2fa2406740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f981b1030 con 0x55bba8e5d990 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2fa2406740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f981aee80 con 0x55bba8e532c0 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f9cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e4e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f97e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f9cc00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e4e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48798/0 (socket says 172.21.15.103:48798) 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f9cc00640 1 -- 172.21.15.103:0/3264771265 learned_addr learned my addr 172.21.15.103:0/3264771265 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f9d600640 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e5d990 0x55bba8e75840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f97e00640 1 -- 172.21.15.103:0/3264771265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e5d990 msgr2=0x55bba8e75840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f97e00640 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e5d990 0x55bba8e75840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f97e00640 1 -- 172.21.15.103:0/3264771265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 msgr2=0x55bba8e4e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.350+0000 7f2f97e00640 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e4e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.355 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.351+0000 7f2f97e00640 1 -- 172.21.15.103:0/3264771265 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bba8e55620 con 0x55bba8e532c0 2024-05-23T00:01:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.351+0000 7f2f9d600640 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e5d990 0x55bba8e75840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.351+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e4e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.351+0000 7f2f97e00640 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e694c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2f84004650 tx=0x7f2f84005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea4fdb3318443839 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.351+0000 7f2f89200640 1 -- 172.21.15.103:0/3264771265 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f840097a0 con 0x55bba8e532c0 2024-05-23T00:01:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.351+0000 7f2f89200640 1 -- 172.21.15.103:0/3264771265 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f8402e400 con 0x55bba8e532c0 2024-05-23T00:01:14.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.351+0000 7f2f89200640 1 -- 172.21.15.103:0/3264771265 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84003850 con 0x55bba8e532c0 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 -- 172.21.15.103:0/3264771265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 msgr2=0x55bba8e694c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e694c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2f84004650 tx=0x7f2f84005f90 comp rx=0 tx=0).stop 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 -- 172.21.15.103:0/3264771265 shutdown_connections 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e5d990 0x55bba8e75840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e532c0 0x55bba8e694c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 --2- 172.21.15.103:0/3264771265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e529a0 0x55bba8e4e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 -- 172.21.15.103:0/3264771265 >> 172.21.15.103:0/3264771265 conn(0x55bba8e4ecc0 msgr2=0x55bba8e5b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 -- 172.21.15.103:0/3264771265 shutdown_connections 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.352+0000 7f2fa2406740 1 -- 172.21.15.103:0/3264771265 wait complete. 2024-05-23T00:01:14.359 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 1 -- start start 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 1 Processor -- start 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e529a0 0x55bba8e61660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 0x55bba8e61ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2f981af4f0 con 0x55bba8e5d990 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f2f981b0960 con 0x55bba8e532c0 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f981afbc0 con 0x55bba8e529a0 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.353+0000 7f2fa2406740 10 client.0 _ll_register_callbacks cb 0x7f2f8c00ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e529a0 0x55bba8e61660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9d600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f97e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 0x55bba8e61ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e529a0 0x55bba8e61660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34518/0 (socket says 172.21.15.103:34518) 2024-05-23T00:01:14.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 -- 172.21.15.103:0/535435060 learned_addr learned my addr 172.21.15.103:0/535435060 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 -- 172.21.15.103:0/535435060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 msgr2=0x55bba8e61ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 0x55bba8e61ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 -- 172.21.15.103:0/535435060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 msgr2=0x55bba8e620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e620e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 -- 172.21.15.103:0/535435060 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bba8ed3350 con 0x55bba8e529a0 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f9cc00640 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e529a0 0x55bba8e61660 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c09a0b0 tx=0x7f2f8c0d3bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22b3699a56e6fb68 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f7e000640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.354+0000 7f2f7e000640 1 -- 172.21.15.103:0/535435060 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c0b9030 con 0x55bba8e529a0 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 1 -- 172.21.15.103:0/535435060 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f8402e400 con 0x55bba8e529a0 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2f7e000640 1 -- 172.21.15.103:0/535435060 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7f2f8c09b250 con 0x55bba8e529a0 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2f7e000640 1 -- 172.21.15.103:0/535435060 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2f8c083050 con 0x55bba8e529a0 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 10 client.25792 fetch_fsmap learned FSMap version 25 2024-05-23T00:01:14.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 1 -- 172.21.15.103:0/535435060 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55bba8ed4640 con 0x55bba8e529a0 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2f7e000640 1 -- 172.21.15.103:0/535435060 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f8402e400 con 0x55bba8e529a0 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2f7e000640 1 -- 172.21.15.103:0/535435060 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2f8c097b50 con 0x55bba8e529a0 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 10 client.25792 fetch_fsmap finished waiting for FSMap version 25 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 10 client.25792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 20 client.25792 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bba8ed4d60 0x55bba8ed7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 4 client.25792 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.355+0000 7f2fa2406740 1 -- 172.21.15.103:0/535435060 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bba8dc53e0 con 0x55bba8ed4d60 2024-05-23T00:01:14.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.356+0000 7f2f97e00640 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bba8ed4d60 0x55bba8ed7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:14.363 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.356+0000 7f2f97e00640 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bba8ed4d60 0x55bba8ed7150 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13b265547b6bddc8 server_cookie=cf0bacbed2c91114 in_seq=0 out_seq=0 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.356+0000 7f2f7e000640 10 client.25792 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.356+0000 7f2f7e000640 1 -- 172.21.15.103:0/535435060 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55bba8dc53e0 con 0x55bba8ed4d60 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.356+0000 7f2f7e000640 10 client.25792 handle_command_reply: tid=0 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.356+0000 7f2fa2406740 1 client.25792 shutdown 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 -- 172.21.15.103:0/535435060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e529a0 msgr2=0x55bba8e61660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e529a0 0x55bba8e61660 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c09a0b0 tx=0x7f2f8c0d3bf0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 -- 172.21.15.103:0/535435060 shutdown_connections 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bba8ed4d60 0x55bba8ed7150 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bba8e5d990 0x55bba8e620e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bba8e532c0 0x55bba8e61ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 --2- 172.21.15.103:0/535435060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bba8e529a0 0x55bba8e61660 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2f7e000640 0 client.25792 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 -- 172.21.15.103:0/535435060 >> 172.21.15.103:0/535435060 conn(0x55bba8e4ecc0 msgr2=0x55bba8e540f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 -- 172.21.15.103:0/535435060 shutdown_connections 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 1 -- 172.21.15.103:0/535435060 wait complete. 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.357+0000 7f2fa2406740 20 client.25792 trim_cache size 0 max 16384 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stdout: "scrubs": { 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stdout: "7de8af36-27d4-4ecb-8338-ac63c1b54732": { 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stdout: "path": "/", 2024-05-23T00:01:14.365 INFO:teuthology.orchestra.run.smithi103.stdout: "tag": "7de8af36-27d4-4ecb-8338-ac63c1b54732", 2024-05-23T00:01:14.366 INFO:teuthology.orchestra.run.smithi103.stdout: "options": "recursive,force" 2024-05-23T00:01:14.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:01:14.366 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:01:14.366 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 -- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2f800a2e20 msgr2=0x7f2f800a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2f800a2e20 0x7f2f800a52e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2f90003b10 tx=0x7f2f90005d80 comp rx=0 tx=0).stop 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 -- 172.21.15.103:0/1764119266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f98073b40 msgr2=0x7f2f9819a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f98073b40 0x7f2f9819a320 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c00e9f0 tx=0x7f2f8c00eeb0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 -- 172.21.15.103:0/1764119266 shutdown_connections 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f2f800a2e20 0x7f2f800a52e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2f9810e130 0x7f2f9819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2f98074460 0x7f2f9819a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 --2- 172.21.15.103:0/1764119266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f98073b40 0x7f2f9819a320 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 -- 172.21.15.103:0/1764119266 >> 172.21.15.103:0/1764119266 conn(0x7f2f98078cc0 msgr2=0x7f2f980fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 -- 172.21.15.103:0/1764119266 shutdown_connections 2024-05-23T00:01:14.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:14.358+0000 7f2f9f600640 1 -- 172.21.15.103:0/1764119266 wait complete. 2024-05-23T00:01:14.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:14 smithi174 ceph-mon[36179]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-05-23T00:01:14.379 INFO:tasks.cephfs.filesystem:scrub status for tag:7de8af36-27d4-4ecb-8338-ac63c1b54732 - {'path': '/', 'tag': '7de8af36-27d4-4ecb-8338-ac63c1b54732', 'options': 'recursive,force'} 2024-05-23T00:01:14.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:14 smithi105 ceph-mon[36426]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-05-23T00:01:15.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:15 smithi103 ceph-mon[31370]: scrub queued for path: / 2024-05-23T00:01:15.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:15 smithi103 ceph-mon[31370]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:15.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:15 smithi103 ceph-mon[31370]: scrub summary: active paths [/] 2024-05-23T00:01:15.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:15 smithi103 ceph-mon[31370]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:15.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:15 smithi174 ceph-mon[36179]: scrub queued for path: / 2024-05-23T00:01:15.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:15 smithi174 ceph-mon[36179]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:15.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:15 smithi174 ceph-mon[36179]: scrub summary: active paths [/] 2024-05-23T00:01:15.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:15 smithi174 ceph-mon[36179]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:15.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:15 smithi105 ceph-mon[36426]: scrub queued for path: / 2024-05-23T00:01:15.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:15 smithi105 ceph-mon[36426]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:15.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:15 smithi105 ceph-mon[36426]: scrub summary: active paths [/] 2024-05-23T00:01:15.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:15 smithi105 ceph-mon[36426]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:16.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:16 smithi174 ceph-mon[36179]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:01:16.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:16 smithi105 ceph-mon[36426]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:01:16.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:16 smithi103 ceph-mon[31370]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:01:17.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:17 smithi174 ceph-mon[36179]: scrub summary: idle 2024-05-23T00:01:17.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:17 smithi105 ceph-mon[36426]: scrub summary: idle 2024-05-23T00:01:17.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:17 smithi103 ceph-mon[31370]: scrub summary: idle 2024-05-23T00:01:18.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:18 smithi174 ceph-mon[36179]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-23T00:01:18.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:18 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:18.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:18 smithi105 ceph-mon[36426]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-23T00:01:18.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:18 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:18.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:18 smithi103 ceph-mon[31370]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-23T00:01:18.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:18 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:20.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:20 smithi174 ceph-mon[36179]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-23T00:01:20.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:20 smithi105 ceph-mon[36426]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-23T00:01:20.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:20 smithi103 ceph-mon[31370]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-23T00:01:22.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:22 smithi174 ceph-mon[36179]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-05-23T00:01:22.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:22 smithi105 ceph-mon[36426]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-05-23T00:01:22.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:22 smithi103 ceph-mon[31370]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-05-23T00:01:23.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:23 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:23.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:23 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:23.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:23 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.825+0000 7ffbe7a00640 1 -- 172.21.15.103:0/1146693470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 msgr2=0x7ffbe010e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.825+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/1146693470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 0x7ffbe010e750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ffbd8009be0 tx=0x7ffbd802a6c0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.825+0000 7ffbe7a00640 1 -- 172.21.15.103:0/1146693470 shutdown_connections 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.825+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/1146693470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 0x7ffbe010e750 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.825+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/1146693470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbe0101880 0x7ffbe0101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.825+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/1146693470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbe0107890 0x7ffbe0107c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.825+0000 7ffbe7a00640 1 -- 172.21.15.103:0/1146693470 >> 172.21.15.103:0/1146693470 conn(0x7ffbe0078d00 msgr2=0x7ffbe0079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.825+0000 7ffbe7a00640 1 -- 172.21.15.103:0/1146693470 shutdown_connections 2024-05-23T00:01:23.829 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.826+0000 7ffbe7a00640 1 -- 172.21.15.103:0/1146693470 wait complete. 2024-05-23T00:01:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.827+0000 7ffbe7a00640 1 Processor -- start 2024-05-23T00:01:23.830 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.827+0000 7ffbe7a00640 1 -- start start 2024-05-23T00:01:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe7a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbe0101880 0x7ffbe0073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe7a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 0x7ffbe00740b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.832 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe7a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbe0107890 0x7ffbe00745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe7a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe0111c50 con 0x7ffbe0101880 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe7a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe0111ad0 con 0x7ffbe0102240 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe7a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe0111dd0 con 0x7ffbe0107890 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbe0101880 0x7ffbe0073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbe0101880 0x7ffbe0073b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48818/0 (socket says 172.21.15.103:48818) 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5000640 1 -- 172.21.15.103:0/2748384280 learned_addr learned my addr 172.21.15.103:0/2748384280 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 0x7ffbe00740b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbe0107890 0x7ffbe00745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5000640 1 -- 172.21.15.103:0/2748384280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbe0107890 msgr2=0x7ffbe00745f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5000640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbe0107890 0x7ffbe00745f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5000640 1 -- 172.21.15.103:0/2748384280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 msgr2=0x7ffbe00740b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.833 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5000640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 0x7ffbe00740b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5000640 1 -- 172.21.15.103:0/2748384280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe01a6510 con 0x7ffbe0101880 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbe5a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbe0107890 0x7ffbe00745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.828+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 0x7ffbe00740b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.829+0000 7ffbe5000640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbe0101880 0x7ffbe0073b70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9f0 tx=0x7ffbd400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.829+0000 7ffbdd600640 1 -- 172.21.15.103:0/2748384280 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400ccc0 con 0x7ffbe0101880 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.829+0000 7ffbdd600640 1 -- 172.21.15.103:0/2748384280 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbd400ce60 con 0x7ffbe0101880 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.829+0000 7ffbe7a00640 1 -- 172.21.15.103:0/2748384280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe01a6800 con 0x7ffbe0101880 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.829+0000 7ffbdd600640 1 -- 172.21.15.103:0/2748384280 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4010690 con 0x7ffbe0101880 2024-05-23T00:01:23.834 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.829+0000 7ffbe7a00640 1 -- 172.21.15.103:0/2748384280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe01a6d00 con 0x7ffbe0101880 2024-05-23T00:01:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.832+0000 7ffbea7f6740 1 Processor -- start 2024-05-23T00:01:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbea7f6740 1 -- start start 2024-05-23T00:01:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f940759130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 0x55f940763800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f940766130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.841 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f9406bd2e0 con 0x55f940758d50 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f940664e50 con 0x55f940759670 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f9406b92a0 con 0x55f940763d40 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbdd600640 1 -- 172.21.15.103:0/2748384280 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7ffbd40046c0 con 0x7ffbe0101880 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbe5a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f940759130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbdfe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f940766130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbdfe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f940766130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34556/0 (socket says 172.21.15.103:34556) 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.833+0000 7ffbdfe00640 1 -- 172.21.15.103:0/429830596 learned_addr learned my addr 172.21.15.103:0/429830596 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbdfe00640 1 -- 172.21.15.103:0/429830596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 msgr2=0x55f940763800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/429830596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 0x55f940763800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbdfe00640 1 -- 172.21.15.103:0/429830596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 msgr2=0x55f940759130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:23.842 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/429830596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f940759130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbdfe00640 1 -- 172.21.15.103:0/429830596 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9403d1210 con 0x55f940763d40 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/429830596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f940766130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0114f0 tx=0x7ffbcc0119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8456b26e9e6072bb server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbd2000640 1 -- 172.21.15.103:0/429830596 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc005020 con 0x55f940763d40 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbd2000640 1 -- 172.21.15.103:0/429830596 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbcc0047a0 con 0x55f940763d40 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.834+0000 7ffbd2000640 1 -- 172.21.15.103:0/429830596 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc00f9f0 con 0x55f940763d40 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 -- 172.21.15.103:0/429830596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 msgr2=0x55f940766130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/429830596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f940766130 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0114f0 tx=0x7ffbcc0119b0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 -- 172.21.15.103:0/429830596 shutdown_connections 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/429830596 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f940766130 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/429830596 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 0x55f940763800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/429830596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f940759130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.843 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 -- 172.21.15.103:0/429830596 >> 172.21.15.103:0/429830596 conn(0x55f940755070 msgr2=0x55f940755450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 -- 172.21.15.103:0/429830596 shutdown_connections 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbea7f6740 1 -- 172.21.15.103:0/429830596 wait complete. 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.835+0000 7ffbdd600640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ffbc80a2dd0 0x7ffbc80a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 1 -- start start 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbdd600640 1 -- 172.21.15.103:0/2748384280 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ffbd40cfeb0 con 0x7ffbe0101880 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbe5a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ffbc80a2dd0 0x7ffbc80a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 1 Processor -- start 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940758d50 0x55f940760c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f940761180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940763d40 0x55f9407616c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f940766bd0 con 0x55f940763d40 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55f940678210 con 0x55f940758d50 2024-05-23T00:01:23.844 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55f940766d50 con 0x55f940759670 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.836+0000 7ffbea7f6740 10 client.0 _ll_register_callbacks cb 0x55f94065b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f940761180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f940761180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:34558/0 (socket says 172.21.15.103:34558) 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 -- 172.21.15.103:0/1121652208 learned_addr learned my addr 172.21.15.103:0/1121652208 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbe5000640 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940758d50 0x55f940760c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbe5a00640 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940763d40 0x55f9407616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbe5a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ffbc80a2dd0 0x7ffbc80a5290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffbd80099a0 tx=0x7ffbd8005bf0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 -- 172.21.15.103:0/1121652208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940758d50 msgr2=0x55f940760c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940758d50 0x55f940760c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 -- 172.21.15.103:0/1121652208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940763d40 msgr2=0x55f9407616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940763d40 0x55f9407616c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 -- 172.21.15.103:0/1121652208 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f94049d620 con 0x55f940759670 2024-05-23T00:01:23.845 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f940761180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc01b040 tx=0x7ffbcc005230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb568a566fd10ddb server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.837+0000 7ffbc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:01:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.838+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1121652208 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc005630 con 0x55f940759670 2024-05-23T00:01:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.838+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1121652208 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbcc0047a0 con 0x55f940759670 2024-05-23T00:01:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.838+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1121652208 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ffbcc004030 con 0x55f940759670 2024-05-23T00:01:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.838+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1121652208 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffbcc033050 con 0x55f940759670 2024-05-23T00:01:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.838+0000 7ffbea7f6740 10 client.25804 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.838+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1121652208 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f940772dd0 con 0x55f940759670 2024-05-23T00:01:23.846 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.838+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1121652208 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbcc0047a0 con 0x55f940759670 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.839+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1121652208 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffbcc0035f0 con 0x55f940759670 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.839+0000 7ffbea7f6740 10 client.25804 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.839+0000 7ffbea7f6740 10 client.25804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.839+0000 7ffbea7f6740 20 client.25804 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.839+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55f940774240 0x55f940776630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.839+0000 7ffbea7f6740 4 client.25804 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.839+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1121652208 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f9406d0ec0 con 0x55f940774240 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.839+0000 7ffbe5000640 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55f940774240 0x55f940776630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.840+0000 7ffbc6a00640 10 client.25804 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.840+0000 7ffbe5000640 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55f940774240 0x55f940776630 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0a2a35ea89f46c7 server_cookie=7586fdfc8cff6149 in_seq=0 out_seq=0 2024-05-23T00:01:23.848 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.840+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1121652208 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f9406d0ec0 con 0x55f940774240 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.840+0000 7ffbc6a00640 10 client.25804 handle_command_reply: tid=0 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 client.25804 shutdown 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1121652208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 msgr2=0x55f940761180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f940761180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc01b040 tx=0x7ffbcc005230 comp rx=0 tx=0).stop 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1121652208 shutdown_connections 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55f940774240 0x55f940776630 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940763d40 0x55f9407616c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f940761180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1121652208 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940758d50 0x55f940760c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbc6a00640 0 client.25804 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1121652208 >> 172.21.15.103:0/1121652208 conn(0x55f940755070 msgr2=0x55f94075ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1121652208 shutdown_connections 2024-05-23T00:01:23.849 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1121652208 wait complete. 2024-05-23T00:01:23.850 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.841+0000 7ffbea7f6740 20 client.25804 trim_cache size 0 max 16384 2024-05-23T00:01:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.860+0000 7ffbea7f6740 1 Processor -- start 2024-05-23T00:01:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.860+0000 7ffbea7f6740 1 -- start start 2024-05-23T00:01:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.860+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94076fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.863 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.860+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 0x55f94077be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.860+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f94077c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.860+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe01b35e0 con 0x55f940758d50 2024-05-23T00:01:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.860+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe01b0eb0 con 0x55f940759670 2024-05-23T00:01:23.864 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.860+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe01aed00 con 0x55f940763d40 2024-05-23T00:01:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbe5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94076fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbdfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 0x55f94077be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbe5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94076fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48840/0 (socket says 172.21.15.103:48840) 2024-05-23T00:01:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbe5000640 1 -- 172.21.15.103:0/989039074 learned_addr learned my addr 172.21.15.103:0/989039074 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbe5a00640 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f94077c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbdfe00640 1 -- 172.21.15.103:0/989039074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 msgr2=0x55f94077c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:23.866 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f94077c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbdfe00640 1 -- 172.21.15.103:0/989039074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 msgr2=0x55f94076fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94076fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbdfe00640 1 -- 172.21.15.103:0/989039074 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f9407594a0 con 0x55f940759670 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbe5a00640 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f94077c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 0x55f94077be90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc003a90 tx=0x7ffbcc004f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da911e33f06ac7b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.861+0000 7ffbd1600640 1 -- 172.21.15.103:0/989039074 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc00fea0 con 0x55f940759670 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbd1600640 1 -- 172.21.15.103:0/989039074 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbcc016030 con 0x55f940759670 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbd1600640 1 -- 172.21.15.103:0/989039074 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc003c80 con 0x55f940759670 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbea7f6740 1 -- 172.21.15.103:0/989039074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 msgr2=0x55f94077be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 0x55f94077be90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc003a90 tx=0x7ffbcc004f10 comp rx=0 tx=0).stop 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbea7f6740 1 -- 172.21.15.103:0/989039074 shutdown_connections 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940763d40 0x55f94077c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940759670 0x55f94077be90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.867 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/989039074 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94076fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbea7f6740 1 -- 172.21.15.103:0/989039074 >> 172.21.15.103:0/989039074 conn(0x55f940755070 msgr2=0x55f940761490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.862+0000 7ffbea7f6740 1 -- 172.21.15.103:0/989039074 shutdown_connections 2024-05-23T00:01:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.863+0000 7ffbea7f6740 1 -- 172.21.15.103:0/989039074 wait complete. 2024-05-23T00:01:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.863+0000 7ffbea7f6740 1 -- start start 2024-05-23T00:01:23.868 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.863+0000 7ffbea7f6740 1 Processor -- start 2024-05-23T00:01:23.870 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.863+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94075d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.863+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f94075ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbea7f6740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940763d40 0x55f94075e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe01af370 con 0x55f940758d50 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe01b07e0 con 0x55f940763d40 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbea7f6740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe01afa40 con 0x55f940759670 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbea7f6740 10 client.0 _ll_register_callbacks cb 0x7ffbcc004ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94075d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94075d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48848/0 (socket says 172.21.15.103:48848) 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5000640 1 -- 172.21.15.103:0/1325414226 learned_addr learned my addr 172.21.15.103:0/1325414226 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f94075ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5a00640 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940763d40 0x55f94075e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5000640 1 -- 172.21.15.103:0/1325414226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 msgr2=0x55f94075ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:23.871 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5000640 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f94075ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5000640 1 -- 172.21.15.103:0/1325414226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940763d40 msgr2=0x55f94075e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5000640 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940763d40 0x55f94075e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.864+0000 7ffbe5000640 1 -- 172.21.15.103:0/1325414226 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f9407d9700 con 0x55f940758d50 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbe5000640 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94075d8b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40987b0 tx=0x7ffbd409bc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dce2047b8381c server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbc6a00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1325414226 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd40b9060 con 0x55f940758d50 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1325414226 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9350+0+0 (secure 0 0 0) 0x7ffbd40a45c0 con 0x55f940758d50 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1325414226 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffbcc016030 con 0x55f940758d50 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1325414226 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ffbd40a4a50 con 0x55f940758d50 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbea7f6740 10 client.15705 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1325414226 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f9407da9d0 con 0x55f940758d50 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.865+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1325414226 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffbcc016030 con 0x55f940758d50 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1325414226 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ffbd409b2c0 con 0x55f940758d50 2024-05-23T00:01:23.872 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbea7f6740 10 client.15705 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbea7f6740 10 client.15705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbea7f6740 20 client.15705 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55f9407db130 0x55f9407dd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbea7f6740 4 client.15705 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:01:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1325414226 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f9406caa10 con 0x55f9407db130 2024-05-23T00:01:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55f9407db130 0x55f9407dd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbdfe00640 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55f9407db130 0x55f9407dd550 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b64d998167fe6c1f server_cookie=1c2b8559e9d1f890 in_seq=0 out_seq=0 2024-05-23T00:01:23.873 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.866+0000 7ffbc6a00640 10 client.15705 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.870+0000 7ffbc6a00640 1 -- 172.21.15.103:0/1325414226 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f9406caa10 con 0x55f9407db130 2024-05-23T00:01:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.870+0000 7ffbc6a00640 10 client.15705 handle_command_reply: tid=0 2024-05-23T00:01:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.870+0000 7ffbea7f6740 1 client.15705 shutdown 2024-05-23T00:01:23.874 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.870+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1325414226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 msgr2=0x55f94075d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.870+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94075d8b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40987b0 tx=0x7ffbd409bc70 comp rx=0 tx=0).stop 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.870+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1325414226 shutdown_connections 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.870+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55f9407db130 0x55f9407dd550 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55f940763d40 0x55f94075e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55f940759670 0x55f94075ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbea7f6740 1 --2- 172.21.15.103:0/1325414226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f940758d50 0x55f94075d8b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1325414226 >> 172.21.15.103:0/1325414226 conn(0x55f940755070 msgr2=0x55f940761470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbc6a00640 0 client.15705 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1325414226 shutdown_connections 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbea7f6740 1 -- 172.21.15.103:0/1325414226 wait complete. 2024-05-23T00:01:23.875 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbea7f6740 20 client.15705 trim_cache size 0 max 16384 2024-05-23T00:01:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:01:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:01:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:01:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:01:23.876 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbe7a00640 1 -- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ffbc80a2dd0 msgr2=0x7ffbc80a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ffbc80a2dd0 0x7ffbc80a5290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffbd80099a0 tx=0x7ffbd8005bf0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbe7a00640 1 -- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbe0101880 msgr2=0x7ffbe0073b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.871+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbe0101880 0x7ffbe0073b70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400e9f0 tx=0x7ffbd400edf0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.872+0000 7ffbe7a00640 1 -- 172.21.15.103:0/2748384280 shutdown_connections 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.872+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ffbc80a2dd0 0x7ffbc80a5290 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.872+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbe0107890 0x7ffbe00745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.872+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffbe0102240 0x7ffbe00740b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.872+0000 7ffbe7a00640 1 --2- 172.21.15.103:0/2748384280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffbe0101880 0x7ffbe0073b70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.872+0000 7ffbe7a00640 1 -- 172.21.15.103:0/2748384280 >> 172.21.15.103:0/2748384280 conn(0x7ffbe0078d00 msgr2=0x7ffbe00ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.872+0000 7ffbe7a00640 1 -- 172.21.15.103:0/2748384280 shutdown_connections 2024-05-23T00:01:23.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:23.872+0000 7ffbe7a00640 1 -- 172.21.15.103:0/2748384280 wait complete. 2024-05-23T00:01:24.459 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:24 smithi105 ceph-mon[36426]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-05-23T00:01:24.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:24 smithi103 ceph-mon[31370]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-05-23T00:01:24.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:24 smithi174 ceph-mon[36179]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-05-23T00:01:26.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:26 smithi105 ceph-mon[36426]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-23T00:01:26.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:26 smithi103 ceph-mon[31370]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-23T00:01:26.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:26 smithi174 ceph-mon[36179]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-05-23T00:01:28.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:28 smithi105 ceph-mon[36426]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:01:28.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:28 smithi103 ceph-mon[31370]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:01:28.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:28 smithi174 ceph-mon[36179]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:01:30.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:30 smithi105 ceph-mon[36426]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-05-23T00:01:30.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:30 smithi103 ceph-mon[31370]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-05-23T00:01:30.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:30 smithi174 ceph-mon[36179]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 2.6 KiB/s wr, 9 op/s 2024-05-23T00:01:32.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:32 smithi105 ceph-mon[36426]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-23T00:01:32.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:32 smithi103 ceph-mon[31370]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-23T00:01:32.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:32 smithi174 ceph-mon[36179]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-23T00:01:33.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-23T00:01:33.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-23T00:01:33.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-23T00:01:33.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-05-23T00:01:33.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-05-23T00:01:33.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:33 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-23T00:01:33.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-23T00:01:33.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-23T00:01:33.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-23T00:01:33.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-05-23T00:01:33.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-05-23T00:01:33.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:33 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-23T00:01:33.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2024-05-23T00:01:33.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-23T00:01:33.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]: dispatch 2024-05-23T00:01:33.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]: dispatch 2024-05-23T00:01:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-05-23T00:01:33.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:33 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-05-23T00:01:34.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:34 smithi105 ceph-mon[36426]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-23T00:01:34.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-23T00:01:34.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-23T00:01:34.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-23T00:01:34.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-05-23T00:01:34.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-05-23T00:01:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:34 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-23T00:01:34.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:34 smithi105 ceph-mon[36426]: osdmap e95: 12 total, 12 up, 12 in 2024-05-23T00:01:34.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:34 smithi103 ceph-mon[31370]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-23T00:01:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-23T00:01:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-23T00:01:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-23T00:01:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-05-23T00:01:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-05-23T00:01:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:34 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-23T00:01:34.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:34 smithi103 ceph-mon[31370]: osdmap e95: 12 total, 12 up, 12 in 2024-05-23T00:01:34.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:34 smithi174 ceph-mon[36179]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-05-23T00:01:34.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2024-05-23T00:01:34.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-23T00:01:34.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [9, 1]}]': finished 2024-05-23T00:01:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [5, 8]}]': finished 2024-05-23T00:01:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-05-23T00:01:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:34 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-05-23T00:01:34.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:34 smithi174 ceph-mon[36179]: osdmap e95: 12 total, 12 up, 12 in 2024-05-23T00:01:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:35 smithi105 ceph-mon[36426]: osdmap e96: 12 total, 12 up, 12 in 2024-05-23T00:01:35.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:35 smithi103 ceph-mon[31370]: osdmap e96: 12 total, 12 up, 12 in 2024-05-23T00:01:35.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:35 smithi174 ceph-mon[36179]: osdmap e96: 12 total, 12 up, 12 in 2024-05-23T00:01:36.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:36 smithi105 ceph-mon[36426]: pgmap v36: 129 pgs: 6 peering, 123 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:01:36.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:36 smithi105 ceph-mon[36426]: Health check failed: Reduced data availability: 6 pgs peering (PG_AVAILABILITY) 2024-05-23T00:01:36.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:36 smithi103 ceph-mon[31370]: pgmap v36: 129 pgs: 6 peering, 123 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:01:36.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:36 smithi103 ceph-mon[31370]: Health check failed: Reduced data availability: 6 pgs peering (PG_AVAILABILITY) 2024-05-23T00:01:36.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:36 smithi174 ceph-mon[36179]: pgmap v36: 129 pgs: 6 peering, 123 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:01:36.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:36 smithi174 ceph-mon[36179]: Health check failed: Reduced data availability: 6 pgs peering (PG_AVAILABILITY) 2024-05-23T00:01:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.025+0000 7f838ca00640 1 -- 172.21.15.103:0/1256761047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 msgr2=0x7f8388103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.025+0000 7f838ca00640 1 --2- 172.21.15.103:0/1256761047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 0x7f8388103d20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8374009930 tx=0x7f837402a440 comp rx=0 tx=0).stop 2024-05-23T00:01:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.026+0000 7f838ca00640 1 -- 172.21.15.103:0/1256761047 shutdown_connections 2024-05-23T00:01:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.026+0000 7f838ca00640 1 --2- 172.21.15.103:0/1256761047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8388104260 0x7f8388110770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.026+0000 7f838ca00640 1 --2- 172.21.15.103:0/1256761047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 0x7f8388103d20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.026+0000 7f838ca00640 1 --2- 172.21.15.103:0/1256761047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f83881098b0 0x7f8388109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.026+0000 7f838ca00640 1 -- 172.21.15.103:0/1256761047 >> 172.21.15.103:0/1256761047 conn(0x7f83880ff350 msgr2=0x7f8388101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.026+0000 7f838ca00640 1 -- 172.21.15.103:0/1256761047 shutdown_connections 2024-05-23T00:01:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.026+0000 7f838ca00640 1 -- 172.21.15.103:0/1256761047 wait complete. 2024-05-23T00:01:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.027+0000 7f838ca00640 1 Processor -- start 2024-05-23T00:01:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.027+0000 7f838ca00640 1 -- start start 2024-05-23T00:01:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f838ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 0x7f838819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f838ca00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388104260 0x7f838819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f838ca00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83881098b0 0x7f838819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f838ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8388113c80 con 0x7f83881038a0 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f838ca00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8388113b00 con 0x7f8388104260 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f838ca00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8388113e00 con 0x7f83881098b0 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f8387400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 0x7f838819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f8387400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 0x7f838819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51956/0 (socket says 172.21.15.103:51956) 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f8387400640 1 -- 172.21.15.103:0/562915144 learned_addr learned my addr 172.21.15.103:0/562915144 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f8386a00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388104260 0x7f838819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.028+0000 7f8387e00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83881098b0 0x7f838819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f8387400640 1 -- 172.21.15.103:0/562915144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83881098b0 msgr2=0x7f838819ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f8387400640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83881098b0 0x7f838819ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f8387400640 1 -- 172.21.15.103:0/562915144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388104260 msgr2=0x7f838819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.032 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f8387400640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388104260 0x7f838819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f8387400640 1 -- 172.21.15.103:0/562915144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838819f1a0 con 0x7f83881038a0 2024-05-23T00:01:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f8387400640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 0x7f838819a200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f837800d950 tx=0x7f837800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f837fe00640 1 -- 172.21.15.103:0/562915144 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378017030 con 0x7f83881038a0 2024-05-23T00:01:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f837fe00640 1 -- 172.21.15.103:0/562915144 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8378004020 con 0x7f83881038a0 2024-05-23T00:01:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f837fe00640 1 -- 172.21.15.103:0/562915144 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83780047a0 con 0x7f83881038a0 2024-05-23T00:01:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f838ca00640 1 -- 172.21.15.103:0/562915144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f838819f490 con 0x7f83881038a0 2024-05-23T00:01:37.033 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.029+0000 7f838ca00640 1 -- 172.21.15.103:0/562915144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83881a6d20 con 0x7f83881038a0 2024-05-23T00:01:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.033+0000 7f837fe00640 1 -- 172.21.15.103:0/562915144 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f83780041d0 con 0x7f83881038a0 2024-05-23T00:01:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.033+0000 7f8390deb740 1 Processor -- start 2024-05-23T00:01:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.033+0000 7f8390deb740 1 -- start start 2024-05-23T00:01:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8390deb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c499310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8390deb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8390deb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8390deb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55897c3fd2e0 con 0x55897c498f30 2024-05-23T00:01:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8390deb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55897c3a4e50 con 0x55897c4a3f20 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8390deb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55897c3f92a0 con 0x55897c499850 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c499310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c499310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51964/0 (socket says 172.21.15.103:51964) 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387e00640 1 -- 172.21.15.103:0/304047152 learned_addr learned my addr 172.21.15.103:0/304047152 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8386a00640 1 --2- 172.21.15.103:0/304047152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387e00640 1 -- 172.21.15.103:0/304047152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 msgr2=0x55897c4a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387e00640 1 --2- 172.21.15.103:0/304047152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387e00640 1 -- 172.21.15.103:0/304047152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 msgr2=0x55897c4a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387e00640 1 --2- 172.21.15.103:0/304047152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.034+0000 7f8387e00640 1 -- 172.21.15.103:0/304047152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55897c111210 con 0x55897c498f30 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.035+0000 7f837fe00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f836c0a2e90 0x7f836c0a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.035+0000 7f8387e00640 1 --2- 172.21.15.103:0/304047152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c499310 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8380010370 tx=0x7f8380010770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6608dc2007d67c server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.035+0000 7f837fe00640 1 -- 172.21.15.103:0/562915144 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f83780cf320 con 0x7f83881038a0 2024-05-23T00:01:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.035+0000 7f8387400640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f836c0a2e90 0x7f836c0a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.035+0000 7f8373e00640 1 -- 172.21.15.103:0/304047152 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8380016070 con 0x55897c498f30 2024-05-23T00:01:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.035+0000 7f8373e00640 1 -- 172.21.15.103:0/304047152 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8380004d20 con 0x55897c498f30 2024-05-23T00:01:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.035+0000 7f8373e00640 1 -- 172.21.15.103:0/304047152 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83800127f0 con 0x55897c498f30 2024-05-23T00:01:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 -- 172.21.15.103:0/304047152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 msgr2=0x55897c499310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8387400640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f836c0a2e90 0x7f836c0a5350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83780cc560 tx=0x7f83780d2e20 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 --2- 172.21.15.103:0/304047152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c499310 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8380010370 tx=0x7f8380010770 comp rx=0 tx=0).stop 2024-05-23T00:01:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 -- 172.21.15.103:0/304047152 shutdown_connections 2024-05-23T00:01:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 --2- 172.21.15.103:0/304047152 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 --2- 172.21.15.103:0/304047152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 --2- 172.21.15.103:0/304047152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c499310 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 -- 172.21.15.103:0/304047152 >> 172.21.15.103:0/304047152 conn(0x55897c495250 msgr2=0x55897c495630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 -- 172.21.15.103:0/304047152 shutdown_connections 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.036+0000 7f8390deb740 1 -- 172.21.15.103:0/304047152 wait complete. 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.037+0000 7f8390deb740 1 -- start start 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.037+0000 7f8390deb740 1 Processor -- start 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8390deb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8390deb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8390deb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8390deb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55897c4a6db0 con 0x55897c498f30 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8390deb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55897c3b8210 con 0x55897c4a3f20 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8390deb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55897c4a6f30 con 0x55897c499850 2024-05-23T00:01:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8390deb740 10 client.0 _ll_register_callbacks cb 0x55897c39b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8386a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8387e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8386a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:36188/0 (socket says 172.21.15.103:36188) 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8386a00640 1 -- 172.21.15.103:0/1012357366 learned_addr learned my addr 172.21.15.103:0/1012357366 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8387e00640 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8386a00640 1 -- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 msgr2=0x55897c4a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8386a00640 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8386a00640 1 -- 172.21.15.103:0/1012357366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 msgr2=0x55897c4a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8386a00640 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8386a00640 1 -- 172.21.15.103:0/1012357366 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897c1dd620 con 0x55897c499850 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.038+0000 7f8387e00640 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.039+0000 7f8386a00640 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a1400 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8374033000 tx=0x7f8374031860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=847b4b05e309d386 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.039+0000 7f8370c00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.039+0000 7f8370c00640 1 -- 172.21.15.103:0/1012357366 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8374035040 con 0x55897c499850 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.039+0000 7f8390deb740 1 -- 172.21.15.103:0/1012357366 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8380004d20 con 0x55897c499850 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.039+0000 7f8370c00640 1 -- 172.21.15.103:0/1012357366 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f8374004460 con 0x55897c499850 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.039+0000 7f8370c00640 1 -- 172.21.15.103:0/1012357366 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f837404e050 con 0x55897c499850 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.039+0000 7f8390deb740 10 client.25822 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.039+0000 7f8390deb740 1 -- 172.21.15.103:0/1012357366 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55897c4b3070 con 0x55897c499850 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8370c00640 1 -- 172.21.15.103:0/1012357366 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8380004d20 con 0x55897c499850 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8370c00640 1 -- 172.21.15.103:0/1012357366 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f837403d850 con 0x55897c499850 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8390deb740 10 client.25822 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8390deb740 10 client.25822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8390deb740 20 client.25822 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8390deb740 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55897c4b4580 0x55897c4b69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8390deb740 4 client.25822 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:37.046 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8390deb740 1 -- 172.21.15.103:0/1012357366 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55897c410ec0 con 0x55897c4b4580 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.040+0000 7f8387e00640 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55897c4b4580 0x55897c4b69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.041+0000 7f8387e00640 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55897c4b4580 0x55897c4b69b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aabd2460a44ca1b8 server_cookie=377767fba2babf0a in_seq=0 out_seq=0 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.041+0000 7f8370c00640 10 client.25822 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8370c00640 1 -- 172.21.15.103:0/1012357366 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55897c410ec0 con 0x55897c4b4580 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8370c00640 10 client.25822 handle_command_reply: tid=0 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 client.25822 shutdown 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 -- 172.21.15.103:0/1012357366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 msgr2=0x55897c4a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a1400 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8374033000 tx=0x7f8374031860 comp rx=0 tx=0).stop 2024-05-23T00:01:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 -- 172.21.15.103:0/1012357366 shutdown_connections 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55897c4b4580 0x55897c4b69b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c4a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8370c00640 0 client.25822 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a1400 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 --2- 172.21.15.103:0/1012357366 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 -- 172.21.15.103:0/1012357366 >> 172.21.15.103:0/1012357366 conn(0x55897c495250 msgr2=0x55897c49b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 -- 172.21.15.103:0/1012357366 shutdown_connections 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.042+0000 7f8390deb740 1 -- 172.21.15.103:0/1012357366 wait complete. 2024-05-23T00:01:37.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.043+0000 7f8390deb740 20 client.25822 trim_cache size 0 max 16384 2024-05-23T00:01:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8390deb740 1 Processor -- start 2024-05-23T00:01:37.061 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8390deb740 1 -- start start 2024-05-23T00:01:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8390deb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c498f30 0x55897c4afc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8390deb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c499850 0x55897c49b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.062 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8390deb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c4a3f20 0x55897c4bc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8390deb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f83881b38d0 con 0x55897c499850 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8390deb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f83881b11a0 con 0x55897c498f30 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8390deb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f83881aeff0 con 0x55897c4a3f20 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8387e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c498f30 0x55897c4afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8386a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c499850 0x55897c49b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8387e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c498f30 0x55897c4afc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:34340/0 (socket says 172.21.15.103:34340) 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.059+0000 7f8387e00640 1 -- 172.21.15.103:0/2926537363 learned_addr learned my addr 172.21.15.103:0/2926537363 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8387e00640 1 --2- 172.21.15.103:0/2926537363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c4a3f20 0x55897c4bc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8387e00640 1 -- 172.21.15.103:0/2926537363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c4a3f20 msgr2=0x55897c4bc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8387e00640 1 --2- 172.21.15.103:0/2926537363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c4a3f20 0x55897c4bc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8387e00640 1 -- 172.21.15.103:0/2926537363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c499850 msgr2=0x55897c49b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:37.063 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8387e00640 1 --2- 172.21.15.103:0/2926537363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c499850 0x55897c49b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8387e00640 1 -- 172.21.15.103:0/2926537363 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55897c499640 con 0x55897c498f30 2024-05-23T00:01:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8387e00640 1 --2- 172.21.15.103:0/2926537363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c498f30 0x55897c4afc70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8380009f40 tx=0x7f8380005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b8ff05cbf9c86d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8373400640 1 -- 172.21.15.103:0/2926537363 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8380016040 con 0x55897c498f30 2024-05-23T00:01:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8373400640 1 -- 172.21.15.103:0/2926537363 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8380003e50 con 0x55897c498f30 2024-05-23T00:01:37.064 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.060+0000 7f8373400640 1 -- 172.21.15.103:0/2926537363 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83800113e0 con 0x55897c498f30 2024-05-23T00:01:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 -- 172.21.15.103:0/2926537363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c498f30 msgr2=0x55897c4afc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 --2- 172.21.15.103:0/2926537363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c498f30 0x55897c4afc70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8380009f40 tx=0x7f8380005fd0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 -- 172.21.15.103:0/2926537363 shutdown_connections 2024-05-23T00:01:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 --2- 172.21.15.103:0/2926537363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c4a3f20 0x55897c4bc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 --2- 172.21.15.103:0/2926537363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c499850 0x55897c49b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 --2- 172.21.15.103:0/2926537363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c498f30 0x55897c4afc70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 -- 172.21.15.103:0/2926537363 >> 172.21.15.103:0/2926537363 conn(0x55897c495250 msgr2=0x55897c4943b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 -- 172.21.15.103:0/2926537363 shutdown_connections 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.061+0000 7f8390deb740 1 -- 172.21.15.103:0/2926537363 wait complete. 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 1 -- start start 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 1 Processor -- start 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a81d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c519af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f83881af660 con 0x55897c498f30 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f83881b0ad0 con 0x55897c4a3f20 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f83881afd30 con 0x55897c499850 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.062+0000 7f8390deb740 10 client.0 _ll_register_callbacks cb 0x7f8374005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51992/0 (socket says 172.21.15.103:51992) 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 -- 172.21.15.103:0/3284285679 learned_addr learned my addr 172.21.15.103:0/3284285679 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c519af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8386a00640 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 -- 172.21.15.103:0/3284285679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 msgr2=0x55897c4a8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.069 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 -- 172.21.15.103:0/3284285679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 msgr2=0x55897c519af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c519af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 -- 172.21.15.103:0/3284285679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55897c51a030 con 0x55897c498f30 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8386a00640 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8387e00640 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a81d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f83800107b0 tx=0x7f83800045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9499375fdba285b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.063+0000 7f8370c00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8370c00640 1 -- 172.21.15.103:0/3284285679 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8380016070 con 0x55897c498f30 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8390deb740 1 -- 172.21.15.103:0/3284285679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8380003e50 con 0x55897c498f30 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8370c00640 1 -- 172.21.15.103:0/3284285679 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f8380003b80 con 0x55897c498f30 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8370c00640 1 -- 172.21.15.103:0/3284285679 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8380004000 con 0x55897c498f30 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8390deb740 10 client.15723 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8390deb740 1 -- 172.21.15.103:0/3284285679 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55897c51b240 con 0x55897c498f30 2024-05-23T00:01:37.070 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8370c00640 1 -- 172.21.15.103:0/3284285679 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8380003e50 con 0x55897c498f30 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8370c00640 1 -- 172.21.15.103:0/3284285679 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f838000f040 con 0x55897c498f30 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8390deb740 10 client.15723 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8390deb740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8390deb740 20 client.15723 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.064+0000 7f8390deb740 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55897c51b620 0x55897c51da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.065+0000 7f8390deb740 4 client.15723 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.065+0000 7f8390deb740 1 -- 172.21.15.103:0/3284285679 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55897c40aa10 con 0x55897c51b620 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.065+0000 7f8386a00640 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55897c51b620 0x55897c51da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.065+0000 7f8370c00640 10 client.15723 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:37.071 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.065+0000 7f8386a00640 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55897c51b620 0x55897c51da10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d97059a3d85c44ef server_cookie=1dd9470f502bd940 in_seq=0 out_seq=0 2024-05-23T00:01:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8370c00640 1 -- 172.21.15.103:0/3284285679 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55897c40aa10 con 0x55897c51b620 2024-05-23T00:01:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8370c00640 10 client.15723 handle_command_reply: tid=0 2024-05-23T00:01:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 client.15723 shutdown 2024-05-23T00:01:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 -- 172.21.15.103:0/3284285679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 msgr2=0x55897c4a81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.075 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a81d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f83800107b0 tx=0x7f83800045d0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 -- 172.21.15.103:0/3284285679 shutdown_connections 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55897c51b620 0x55897c51da10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55897c4a3f20 0x55897c519af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55897c499850 0x55897c4a8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8370c00640 0 client.15723 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 --2- 172.21.15.103:0/3284285679 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55897c498f30 0x55897c4a81d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 -- 172.21.15.103:0/3284285679 >> 172.21.15.103:0/3284285679 conn(0x55897c495250 msgr2=0x55897c4a26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 -- 172.21.15.103:0/3284285679 shutdown_connections 2024-05-23T00:01:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 1 -- 172.21.15.103:0/3284285679 wait complete. 2024-05-23T00:01:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.073+0000 7f8390deb740 20 client.15723 trim_cache size 0 max 16384 2024-05-23T00:01:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:01:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:01:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:01:37.077 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:01:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 -- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f836c0a2e90 msgr2=0x7f836c0a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f836c0a2e90 0x7f836c0a5350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f83780cc560 tx=0x7f83780d2e20 comp rx=0 tx=0).stop 2024-05-23T00:01:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 -- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 msgr2=0x7f838819a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 0x7f838819a200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f837800d950 tx=0x7f837800de10 comp rx=0 tx=0).stop 2024-05-23T00:01:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 -- 172.21.15.103:0/562915144 shutdown_connections 2024-05-23T00:01:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f836c0a2e90 0x7f836c0a5350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83881098b0 0x7f838819ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.078 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8388104260 0x7f838819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 --2- 172.21.15.103:0/562915144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f83881038a0 0x7f838819a200 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.074+0000 7f838ca00640 1 -- 172.21.15.103:0/562915144 >> 172.21.15.103:0/562915144 conn(0x7f83880ff350 msgr2=0x7f83880ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.075+0000 7f838ca00640 1 -- 172.21.15.103:0/562915144 shutdown_connections 2024-05-23T00:01:37.079 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:37.075+0000 7f838ca00640 1 -- 172.21.15.103:0/562915144 wait complete. 2024-05-23T00:01:38.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:38 smithi105 ceph-mon[36426]: pgmap v37: 129 pgs: 6 peering, 123 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-05-23T00:01:38.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:38 smithi103 ceph-mon[31370]: pgmap v37: 129 pgs: 6 peering, 123 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-05-23T00:01:38.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:38 smithi174 ceph-mon[36179]: pgmap v37: 129 pgs: 6 peering, 123 active+clean; 1.4 MiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 127 B/s wr, 3 op/s 2024-05-23T00:01:39.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:39 smithi105 ceph-mon[36426]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering) 2024-05-23T00:01:39.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:39 smithi105 ceph-mon[36426]: Cluster is now healthy 2024-05-23T00:01:39.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:39 smithi103 ceph-mon[31370]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering) 2024-05-23T00:01:39.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:39 smithi103 ceph-mon[31370]: Cluster is now healthy 2024-05-23T00:01:39.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:39 smithi174 ceph-mon[36179]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs peering) 2024-05-23T00:01:39.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:39 smithi174 ceph-mon[36179]: Cluster is now healthy 2024-05-23T00:01:40.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:40 smithi105 ceph-mon[36426]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-23T00:01:40.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:40 smithi103 ceph-mon[31370]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-23T00:01:40.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:40 smithi174 ceph-mon[36179]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-23T00:01:42.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:42 smithi105 ceph-mon[36426]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-23T00:01:42.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:42 smithi103 ceph-mon[31370]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-23T00:01:42.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:42 smithi174 ceph-mon[36179]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-23T00:01:43.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:43 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:01:43.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:43 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:01:43.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:43 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:01:44.380 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-23T00:01:44.381 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-23T00:01:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:44 smithi105 ceph-mon[36426]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-23T00:01:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:44 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:01:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:44 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:01:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:44 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:01:44.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:44 smithi103 ceph-mon[31370]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-23T00:01:44.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:44 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:01:44.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:44 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:01:44.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:44 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:01:44.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 -- 172.21.15.103:0/413360399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f427c104210 msgr2=0x7f427c110720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 --2- 172.21.15.103:0/413360399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f427c104210 0x7f427c110720 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f4274009ca0 tx=0x7f427402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 -- 172.21.15.103:0/413360399 shutdown_connections 2024-05-23T00:01:44.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 --2- 172.21.15.103:0/413360399 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f427c104210 0x7f427c110720 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 --2- 172.21.15.103:0/413360399 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f427c103850 0x7f427c103cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 --2- 172.21.15.103:0/413360399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f427c109860 0x7f427c109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 -- 172.21.15.103:0/413360399 >> 172.21.15.103:0/413360399 conn(0x7f427c0ff2e0 msgr2=0x7f427c101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:44.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 -- 172.21.15.103:0/413360399 shutdown_connections 2024-05-23T00:01:44.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.586+0000 7f4283000640 1 -- 172.21.15.103:0/413360399 wait complete. 2024-05-23T00:01:44.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.587+0000 7f4283000640 1 Processor -- start 2024-05-23T00:01:44.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.587+0000 7f4283000640 1 -- start start 2024-05-23T00:01:44.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.587+0000 7f4283000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f427c103850 0x7f427c199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.587+0000 7f4283000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f427c104210 0x7f427c19a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.587+0000 7f4283000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f427c109860 0x7f427c19e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.587+0000 7f4283000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f427c113c70 con 0x7f427c109860 2024-05-23T00:01:44.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.587+0000 7f4283000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f427c113af0 con 0x7f427c104210 2024-05-23T00:01:44.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.587+0000 7f4283000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f427c113df0 con 0x7f427c103850 2024-05-23T00:01:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f427c103850 0x7f427c199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f427c103850 0x7f427c199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59112/0 (socket says 172.21.15.103:59112) 2024-05-23T00:01:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281c00640 1 -- 172.21.15.103:0/4060006782 learned_addr learned my addr 172.21.15.103:0/4060006782 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4282600640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f427c109860 0x7f427c19e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281200640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f427c104210 0x7f427c19a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281c00640 1 -- 172.21.15.103:0/4060006782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f427c104210 msgr2=0x7f427c19a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281c00640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f427c104210 0x7f427c19a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281c00640 1 -- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f427c109860 msgr2=0x7f427c19e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281c00640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f427c109860 0x7f427c19e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.588+0000 7f4281c00640 1 -- 172.21.15.103:0/4060006782 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f427c19eea0 con 0x7f427c103850 2024-05-23T00:01:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.589+0000 7f4281c00640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f427c103850 0x7f427c199eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f427000e9f0 tx=0x7f427000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.589+0000 7f427a000640 1 -- 172.21.15.103:0/4060006782 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427000cd80 con 0x7f427c103850 2024-05-23T00:01:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.589+0000 7f4283000640 1 -- 172.21.15.103:0/4060006782 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f427c19f190 con 0x7f427c103850 2024-05-23T00:01:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.589+0000 7f427a000640 1 -- 172.21.15.103:0/4060006782 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4270005500 con 0x7f427c103850 2024-05-23T00:01:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.589+0000 7f427a000640 1 -- 172.21.15.103:0/4060006782 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270010690 con 0x7f427c103850 2024-05-23T00:01:44.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.589+0000 7f4283000640 1 -- 172.21.15.103:0/4060006782 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f427c1a6a20 con 0x7f427c103850 2024-05-23T00:01:44.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.594+0000 7f42873bf740 1 Processor -- start 2024-05-23T00:01:44.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.594+0000 7f42873bf740 1 -- start start 2024-05-23T00:01:44.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.594+0000 7f427a000640 1 -- 172.21.15.103:0/4060006782 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f42700046c0 con 0x7f427c103850 2024-05-23T00:01:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.595+0000 7f42873bf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062370 0x55bbca062750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.595+0000 7f42873bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062c90 0x55bbca06ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.595+0000 7f42873bf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca06d360 0x55bbca06f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.595+0000 7f42873bf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bbca01d320 con 0x55bbca06d360 2024-05-23T00:01:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.595+0000 7f42873bf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bbc9fbecb0 con 0x55bbca062370 2024-05-23T00:01:44.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.595+0000 7f42873bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bbc9f65fc0 con 0x55bbca062c90 2024-05-23T00:01:44.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062c90 0x55bbca06ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4282600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062370 0x55bbca062750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062c90 0x55bbca06ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59126/0 (socket says 172.21.15.103:59126) 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281c00640 1 -- 172.21.15.103:0/965805958 learned_addr learned my addr 172.21.15.103:0/965805958 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281200640 1 --2- 172.21.15.103:0/965805958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca06d360 0x55bbca06f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f427a000640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f42640a2e20 0x7f42640a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281c00640 1 -- 172.21.15.103:0/965805958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062370 msgr2=0x55bbca062750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281c00640 1 --2- 172.21.15.103:0/965805958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062370 0x55bbca062750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281c00640 1 -- 172.21.15.103:0/965805958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca06d360 msgr2=0x55bbca06f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281c00640 1 --2- 172.21.15.103:0/965805958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca06d360 0x55bbca06f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f4281c00640 1 -- 172.21.15.103:0/965805958 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbc9d9c7d0 con 0x55bbca062c90 2024-05-23T00:01:44.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.596+0000 7f427a000640 1 -- 172.21.15.103:0/4060006782 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f42700cff10 con 0x7f427c103850 2024-05-23T00:01:44.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.597+0000 7f4282600640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f42640a2e20 0x7f42640a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.597+0000 7f4281c00640 1 --2- 172.21.15.103:0/965805958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062c90 0x55bbca06ce20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f42700cd030 tx=0x7f4270004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70c2bb06be0060ef server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:44.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.597+0000 7f426ea00640 1 -- 172.21.15.103:0/965805958 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427009bb60 con 0x55bbca062c90 2024-05-23T00:01:44.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.597+0000 7f426ea00640 1 -- 172.21.15.103:0/965805958 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f427009c490 con 0x55bbca062c90 2024-05-23T00:01:44.600 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.597+0000 7f426ea00640 1 -- 172.21.15.103:0/965805958 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427009b6f0 con 0x55bbca062c90 2024-05-23T00:01:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 -- 172.21.15.103:0/965805958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062c90 msgr2=0x55bbca06ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f4282600640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f42640a2e20 0x7f42640a52e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4274005e30 tx=0x7f427402a8c0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 --2- 172.21.15.103:0/965805958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062c90 0x55bbca06ce20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f42700cd030 tx=0x7f4270004a40 comp rx=0 tx=0).stop 2024-05-23T00:01:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 -- 172.21.15.103:0/965805958 shutdown_connections 2024-05-23T00:01:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 --2- 172.21.15.103:0/965805958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca06d360 0x55bbca06f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 --2- 172.21.15.103:0/965805958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062c90 0x55bbca06ce20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 --2- 172.21.15.103:0/965805958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062370 0x55bbca062750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.601 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 -- 172.21.15.103:0/965805958 >> 172.21.15.103:0/965805958 conn(0x55bbca05e690 msgr2=0x55bbca05ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 -- 172.21.15.103:0/965805958 shutdown_connections 2024-05-23T00:01:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.598+0000 7f42873bf740 1 -- 172.21.15.103:0/965805958 wait complete. 2024-05-23T00:01:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 1 -- start start 2024-05-23T00:01:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 1 Processor -- start 2024-05-23T00:01:44.602 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca06a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.603 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 0x55bbca06a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca06ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bbc9f79410 con 0x55bbca062370 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55bbc9fbac70 con 0x55bbca062c90 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55bbca0700f0 con 0x55bbca06d360 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.599+0000 7f42873bf740 10 client.0 _ll_register_callbacks cb 0x55bbc9fb50d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca06ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4281c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca06a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca06ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59128/0 (socket says 172.21.15.103:59128) 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 -- 172.21.15.103:0/3256486101 learned_addr learned my addr 172.21.15.103:0/3256486101 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4281200640 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 0x55bbca06a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 -- 172.21.15.103:0/3256486101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 msgr2=0x55bbca06a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:44.604 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 0x55bbca06a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 -- 172.21.15.103:0/3256486101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 msgr2=0x55bbca06a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca06a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 -- 172.21.15.103:0/3256486101 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbc9cd23f0 con 0x55bbca06d360 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4281c00640 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca06a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4282600640 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca06ace0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4274035000 tx=0x7f4274031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b59de49e4ba7d19 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.600+0000 7f4263400640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.601+0000 7f4263400640 1 -- 172.21.15.103:0/3256486101 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274052060 con 0x55bbca06d360 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.601+0000 7f42873bf740 1 -- 172.21.15.103:0/3256486101 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f427009c490 con 0x55bbca06d360 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.601+0000 7f4263400640 1 -- 172.21.15.103:0/3256486101 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f4274003970 con 0x55bbca06d360 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.601+0000 7f4263400640 1 -- 172.21.15.103:0/3256486101 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4274003b70 con 0x55bbca06d360 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.601+0000 7f42873bf740 10 client.25837 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.601+0000 7f42873bf740 1 -- 172.21.15.103:0/3256486101 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bbc9c920f0 con 0x55bbca06d360 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.601+0000 7f4263400640 1 -- 172.21.15.103:0/3256486101 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f427009c490 con 0x55bbca06d360 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.602+0000 7f4263400640 1 -- 172.21.15.103:0/3256486101 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f427402f730 con 0x55bbca06d360 2024-05-23T00:01:44.605 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.602+0000 7f42873bf740 10 client.25837 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.602+0000 7f42873bf740 10 client.25837 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.602+0000 7f42873bf740 20 client.25837 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.602+0000 7f42873bf740 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bbca07d4d0 0x55bbca07f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.602+0000 7f42873bf740 4 client.25837 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:44.606 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.602+0000 7f42873bf740 1 -- 172.21.15.103:0/3256486101 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bbc9f7de00 con 0x55bbca07d4d0 2024-05-23T00:01:44.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.602+0000 7f4281c00640 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bbca07d4d0 0x55bbca07f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.603+0000 7f4281c00640 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bbca07d4d0 0x55bbca07f920 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b94801636798c2 server_cookie=24198bd781f25b1f in_seq=0 out_seq=0 2024-05-23T00:01:44.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.603+0000 7f4263400640 10 client.25837 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:44.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.603+0000 7f4263400640 1 -- 172.21.15.103:0/3256486101 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bbc9f7de00 con 0x55bbca07d4d0 2024-05-23T00:01:44.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.603+0000 7f4263400640 10 client.25837 handle_command_reply: tid=0 2024-05-23T00:01:44.607 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.603+0000 7f42873bf740 1 client.25837 shutdown 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 -- 172.21.15.103:0/3256486101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 msgr2=0x55bbca06ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca06ace0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4274035000 tx=0x7f4274031000 comp rx=0 tx=0).stop 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 -- 172.21.15.103:0/3256486101 shutdown_connections 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bbca07d4d0 0x55bbca07f920 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca06ace0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f4263400640 0 client.25837 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 0x55bbca06a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 --2- 172.21.15.103:0/3256486101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca06a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 -- 172.21.15.103:0/3256486101 >> 172.21.15.103:0/3256486101 conn(0x55bbca05e690 msgr2=0x55bbca055000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 -- 172.21.15.103:0/3256486101 shutdown_connections 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 1 -- 172.21.15.103:0/3256486101 wait complete. 2024-05-23T00:01:44.608 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.604+0000 7f42873bf740 20 client.25837 trim_cache size 0 max 16384 2024-05-23T00:01:44.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:44 smithi174 ceph-mon[36179]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-23T00:01:44.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:44 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:01:44.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:44 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:01:44.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:44 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:01:44.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.622+0000 7f42873bf740 1 Processor -- start 2024-05-23T00:01:44.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f42873bf740 1 -- start start 2024-05-23T00:01:44.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f42873bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062370 0x55bbca064b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f42873bf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062c90 0x55bbca05ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f42873bf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca06d360 0x55bbca069b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f42873bf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f427c1af170 con 0x55bbca062c90 2024-05-23T00:01:44.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f42873bf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f427c1aca40 con 0x55bbca06d360 2024-05-23T00:01:44.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f42873bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f427c1aa8f0 con 0x55bbca062370 2024-05-23T00:01:44.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f4281c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062370 0x55bbca064b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f4281200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062c90 0x55bbca05ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.626 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f4281c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062370 0x55bbca064b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59136/0 (socket says 172.21.15.103:59136) 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f4281c00640 1 -- 172.21.15.103:0/3274719169 learned_addr learned my addr 172.21.15.103:0/3274719169 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.623+0000 7f4282600640 1 --2- 172.21.15.103:0/3274719169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca06d360 0x55bbca069b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f4281c00640 1 -- 172.21.15.103:0/3274719169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca06d360 msgr2=0x55bbca069b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f4281c00640 1 --2- 172.21.15.103:0/3274719169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca06d360 0x55bbca069b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f4281c00640 1 -- 172.21.15.103:0/3274719169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062c90 msgr2=0x55bbca05ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f4281c00640 1 --2- 172.21.15.103:0/3274719169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062c90 0x55bbca05ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f4281c00640 1 -- 172.21.15.103:0/3274719169 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bbca07a2b0 con 0x55bbca062370 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f4281c00640 1 --2- 172.21.15.103:0/3274719169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062370 0x55bbca064b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f427009a040 tx=0x7f4270012f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb42f88b46d22f server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f426e000640 1 -- 172.21.15.103:0/3274719169 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42700b8070 con 0x55bbca062370 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f426e000640 1 -- 172.21.15.103:0/3274719169 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f427009c490 con 0x55bbca062370 2024-05-23T00:01:44.627 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.624+0000 7f426e000640 1 -- 172.21.15.103:0/3274719169 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427009b8d0 con 0x55bbca062370 2024-05-23T00:01:44.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 -- 172.21.15.103:0/3274719169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062370 msgr2=0x55bbca064b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 --2- 172.21.15.103:0/3274719169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062370 0x55bbca064b20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f427009a040 tx=0x7f4270012f50 comp rx=0 tx=0).stop 2024-05-23T00:01:44.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 -- 172.21.15.103:0/3274719169 shutdown_connections 2024-05-23T00:01:44.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 --2- 172.21.15.103:0/3274719169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca06d360 0x55bbca069b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 --2- 172.21.15.103:0/3274719169 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062c90 0x55bbca05ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.630 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 --2- 172.21.15.103:0/3274719169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca062370 0x55bbca064b20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 -- 172.21.15.103:0/3274719169 >> 172.21.15.103:0/3274719169 conn(0x55bbca05e690 msgr2=0x55bbca054770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 -- 172.21.15.103:0/3274719169 shutdown_connections 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.625+0000 7f42873bf740 1 -- 172.21.15.103:0/3274719169 wait complete. 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 1 -- start start 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 1 Processor -- start 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca075720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 0x55bbca070a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca070fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f427c1aaf40 con 0x55bbca062370 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f427c1ac370 con 0x55bbca062c90 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f427c1ab5d0 con 0x55bbca06d360 2024-05-23T00:01:44.631 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.626+0000 7f42873bf740 10 client.0 _ll_register_callbacks cb 0x7f4274009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4281c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca075720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca070fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca070fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59140/0 (socket says 172.21.15.103:59140) 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 -- 172.21.15.103:0/3242835993 learned_addr learned my addr 172.21.15.103:0/3242835993 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4281200640 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 0x55bbca070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 -- 172.21.15.103:0/3242835993 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 msgr2=0x55bbca070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 0x55bbca070a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 -- 172.21.15.103:0/3242835993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 msgr2=0x55bbca075720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca075720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 -- 172.21.15.103:0/3242835993 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bbca071800 con 0x55bbca06d360 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4281c00640 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4282600640 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca070fd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4274005d50 tx=0x7f427402f4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc7361a5848117 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.627+0000 7f4263400640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:01:44.632 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.628+0000 7f4263400640 1 -- 172.21.15.103:0/3242835993 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274062060 con 0x55bbca06d360 2024-05-23T00:01:44.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.628+0000 7f4263400640 1 -- 172.21.15.103:0/3242835993 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f42740494a0 con 0x55bbca06d360 2024-05-23T00:01:44.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.628+0000 7f42873bf740 1 -- 172.21.15.103:0/3242835993 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f427009c490 con 0x55bbca06d360 2024-05-23T00:01:44.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.628+0000 7f4263400640 1 -- 172.21.15.103:0/3242835993 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4274052040 con 0x55bbca06d360 2024-05-23T00:01:44.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.628+0000 7f42873bf740 10 client.25843 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:44.633 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.628+0000 7f42873bf740 1 -- 172.21.15.103:0/3242835993 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bbca0e3aa0 con 0x55bbca06d360 2024-05-23T00:01:44.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.628+0000 7f4263400640 1 -- 172.21.15.103:0/3242835993 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f427009c490 con 0x55bbca06d360 2024-05-23T00:01:44.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.629+0000 7f4263400640 1 -- 172.21.15.103:0/3242835993 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f427405d670 con 0x55bbca06d360 2024-05-23T00:01:44.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.629+0000 7f42873bf740 10 client.25843 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:44.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.629+0000 7f42873bf740 10 client.25843 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:44.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.629+0000 7f42873bf740 20 client.25843 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:44.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.629+0000 7f42873bf740 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bbca07d4d0 0x55bbca0e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.629+0000 7f42873bf740 4 client.25843 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-23T00:01:44.635 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.629+0000 7f42873bf740 1 -- 172.21.15.103:0/3242835993 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bbc9fcc3e0 con 0x55bbca07d4d0 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.629+0000 7f4281c00640 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bbca07d4d0 0x55bbca0e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f4281c00640 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bbca07d4d0 0x55bbca0e6240 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aaa2e5f4aeeb04c server_cookie=b19cdf3f9d98ca3b in_seq=0 out_seq=0 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f4263400640 10 client.25843 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f4263400640 1 -- 172.21.15.103:0/3242835993 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bbc9fcc3e0 con 0x55bbca07d4d0 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f4263400640 10 client.25843 handle_command_reply: tid=0 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 client.25843 shutdown 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 -- 172.21.15.103:0/3242835993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 msgr2=0x55bbca070fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca070fd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4274005d50 tx=0x7f427402f4b0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 -- 172.21.15.103:0/3242835993 shutdown_connections 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55bbca07d4d0 0x55bbca0e6240 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55bbca06d360 0x55bbca070fd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55bbca062c90 0x55bbca070a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.636 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f4263400640 0 client.25843 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 --2- 172.21.15.103:0/3242835993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bbca062370 0x55bbca075720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 -- 172.21.15.103:0/3242835993 >> 172.21.15.103:0/3242835993 conn(0x55bbca05e690 msgr2=0x55bbca055000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 -- 172.21.15.103:0/3242835993 shutdown_connections 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.630+0000 7f42873bf740 1 -- 172.21.15.103:0/3242835993 wait complete. 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f42873bf740 20 client.25843 trim_cache size 0 max 16384 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "no active scrubs running", 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stdout: "scrubs": {} 2024-05-23T00:01:44.637 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:01:44.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 -- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f42640a2e20 msgr2=0x7f42640a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f42640a2e20 0x7f42640a52e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4274005e30 tx=0x7f427402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 -- 172.21.15.103:0/4060006782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f427c103850 msgr2=0x7f427c199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f427c103850 0x7f427c199eb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f427000e9f0 tx=0x7f427000edf0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 -- 172.21.15.103:0/4060006782 shutdown_connections 2024-05-23T00:01:44.638 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f42640a2e20 0x7f42640a52e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f427c109860 0x7f427c19e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f427c104210 0x7f427c19a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 --2- 172.21.15.103:0/4060006782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f427c103850 0x7f427c199eb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.631+0000 7f4283000640 1 -- 172.21.15.103:0/4060006782 >> 172.21.15.103:0/4060006782 conn(0x7f427c0ff2e0 msgr2=0x7f427c0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:44.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.632+0000 7f4283000640 1 -- 172.21.15.103:0/4060006782 shutdown_connections 2024-05-23T00:01:44.639 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.632+0000 7f4283000640 1 -- 172.21.15.103:0/4060006782 wait complete. 2024-05-23T00:01:44.649 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-23T00:01:44.650 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:01:44.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 -- 172.21.15.103:0/2368167965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff981076a0 msgr2=0x7eff98107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 --2- 172.21.15.103:0/2368167965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff981076a0 0x7eff98107a80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7eff88009860 tx=0x7eff8802a3d0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 -- 172.21.15.103:0/2368167965 shutdown_connections 2024-05-23T00:01:44.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 --2- 172.21.15.103:0/2368167965 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff98069150 0x7eff9810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 --2- 172.21.15.103:0/2368167965 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff98107fc0 0x7eff98068a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 --2- 172.21.15.103:0/2368167965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff981076a0 0x7eff98107a80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 -- 172.21.15.103:0/2368167965 >> 172.21.15.103:0/2368167965 conn(0x7eff980760b0 msgr2=0x7eff980764c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:44.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 -- 172.21.15.103:0/2368167965 shutdown_connections 2024-05-23T00:01:44.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.813+0000 7eff9d800640 1 -- 172.21.15.103:0/2368167965 wait complete. 2024-05-23T00:01:44.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.815+0000 7eff9d800640 1 Processor -- start 2024-05-23T00:01:44.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.815+0000 7eff9d800640 1 -- start start 2024-05-23T00:01:44.817 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.815+0000 7eff9d800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff98069150 0x7eff98100930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.815+0000 7eff9d800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff981076a0 0x7eff98100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.815+0000 7eff9d800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff98107fc0 0x7eff981013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.815+0000 7eff9d800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff98111a70 con 0x7eff98069150 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.815+0000 7eff9d800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7eff981118f0 con 0x7eff98107fc0 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.815+0000 7eff9d800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7eff98111bf0 con 0x7eff981076a0 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff97e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff98069150 0x7eff98100930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff97400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff981076a0 0x7eff98100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff97e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff98069150 0x7eff98100930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48774/0 (socket says 172.21.15.103:48774) 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff97e00640 1 -- 172.21.15.103:0/1407522271 learned_addr learned my addr 172.21.15.103:0/1407522271 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:44.818 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff9ce00640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff98107fc0 0x7eff981013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff9ce00640 1 -- 172.21.15.103:0/1407522271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff981076a0 msgr2=0x7eff98100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff9ce00640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff981076a0 0x7eff98100e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.819 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff9ce00640 1 -- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff98069150 msgr2=0x7eff98100930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff9ce00640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff98069150 0x7eff98100930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff9ce00640 1 -- 172.21.15.103:0/1407522271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff981a64f0 con 0x7eff98107fc0 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.816+0000 7eff9ce00640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff98107fc0 0x7eff981013b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7eff9000ec50 tx=0x7eff9000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.817+0000 7eff94c00640 1 -- 172.21.15.103:0/1407522271 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff90003b70 con 0x7eff98107fc0 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.817+0000 7eff94c00640 1 -- 172.21.15.103:0/1407522271 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff9000ee90 con 0x7eff98107fc0 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.817+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff981a67e0 con 0x7eff98107fc0 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.817+0000 7eff94c00640 1 -- 172.21.15.103:0/1407522271 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff900107a0 con 0x7eff98107fc0 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.817+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff981a6ce0 con 0x7eff98107fc0 2024-05-23T00:01:44.820 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.818+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff980685b0 con 0x7eff98107fc0 2024-05-23T00:01:44.822 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.820+0000 7eff94c00640 1 -- 172.21.15.103:0/1407522271 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7eff90004020 con 0x7eff98107fc0 2024-05-23T00:01:44.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.821+0000 7eff94c00640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7eff7c0a2e20 0x7eff7c0a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:44.823 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.821+0000 7eff94c00640 1 -- 172.21.15.103:0/1407522271 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7eff900d0090 con 0x7eff98107fc0 2024-05-23T00:01:44.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.822+0000 7eff97e00640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7eff7c0a2e20 0x7eff7c0a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:44.824 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.823+0000 7eff97e00640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7eff7c0a2e20 0x7eff7c0a52e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7eff88005e00 tx=0x7eff880058b0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:44.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:44.825+0000 7eff94c00640 1 -- 172.21.15.103:0/1407522271 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7eff9009c610 con 0x7eff98107fc0 2024-05-23T00:01:45.065 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.063+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff9802d200 con 0x7eff98107fc0 2024-05-23T00:01:45.066 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.064+0000 7eff94c00640 1 -- 172.21.15.103:0/1407522271 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11899 (secure 0 0 0) 0x7eff9009bf10 con 0x7eff98107fc0 2024-05-23T00:01:45.066 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:01:45.066 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-23T00:01:21.950881+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24559,24563,24575]},"id":1}]} 2024-05-23T00:01:45.067 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 27 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.066+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7eff7c0a2e20 msgr2=0x7eff7c0a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.066+0000 7eff9d800640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7eff7c0a2e20 0x7eff7c0a52e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7eff88005e00 tx=0x7eff880058b0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.066+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff98107fc0 msgr2=0x7eff981013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.066+0000 7eff9d800640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff98107fc0 0x7eff981013b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7eff9000ec50 tx=0x7eff9000c6a0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.067+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 shutdown_connections 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.067+0000 7eff9d800640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7eff7c0a2e20 0x7eff7c0a52e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.067+0000 7eff9d800640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7eff98107fc0 0x7eff981013b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.067+0000 7eff9d800640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7eff981076a0 0x7eff98100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.067+0000 7eff9d800640 1 --2- 172.21.15.103:0/1407522271 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff98069150 0x7eff98100930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.067+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 >> 172.21.15.103:0/1407522271 conn(0x7eff980760b0 msgr2=0x7eff980ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.067+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 shutdown_connections 2024-05-23T00:01:45.069 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.067+0000 7eff9d800640 1 -- 172.21.15.103:0/1407522271 wait complete. 2024-05-23T00:01:45.087 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.251+0000 7fa9e7200640 1 -- 172.21.15.103:0/394306908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e8100e90 msgr2=0x7fa9e810dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.251+0000 7fa9e7200640 1 --2- 172.21.15.103:0/394306908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e8100e90 0x7fa9e810dac0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009900 tx=0x7fa9d402a3f0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.251+0000 7fa9e7200640 1 -- 172.21.15.103:0/394306908 shutdown_connections 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.251+0000 7fa9e7200640 1 --2- 172.21.15.103:0/394306908 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e810e090 0x7fa9e8110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.251+0000 7fa9e7200640 1 --2- 172.21.15.103:0/394306908 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e8100e90 0x7fa9e810dac0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.251+0000 7fa9e7200640 1 --2- 172.21.15.103:0/394306908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e81004e0 0x7fa9e81008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.251+0000 7fa9e7200640 1 -- 172.21.15.103:0/394306908 >> 172.21.15.103:0/394306908 conn(0x7fa9e80fc0f0 msgr2=0x7fa9e80fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.252+0000 7fa9e7200640 1 -- 172.21.15.103:0/394306908 shutdown_connections 2024-05-23T00:01:45.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.252+0000 7fa9e7200640 1 -- 172.21.15.103:0/394306908 wait complete. 2024-05-23T00:01:45.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.253+0000 7fa9e7200640 1 Processor -- start 2024-05-23T00:01:45.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.253+0000 7fa9e7200640 1 -- start start 2024-05-23T00:01:45.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.253+0000 7fa9e7200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e81004e0 0x7fa9e819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e8100e90 0x7fa9e819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e7200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e810e090 0x7fa9e819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8113d00 con 0x7fa9e8100e90 2024-05-23T00:01:45.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e7200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8113b80 con 0x7fa9e81004e0 2024-05-23T00:01:45.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e7200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8113e80 con 0x7fa9e810e090 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e81004e0 0x7fa9e819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e5e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e81004e0 0x7fa9e819a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49844/0 (socket says 172.21.15.103:49844) 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e5e00640 1 -- 172.21.15.103:0/4238767800 learned_addr learned my addr 172.21.15.103:0/4238767800 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e5400640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e8100e90 0x7fa9e819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e6800640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e810e090 0x7fa9e819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e5e00640 1 -- 172.21.15.103:0/4238767800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e810e090 msgr2=0x7fa9e819eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.254+0000 7fa9e5e00640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e810e090 0x7fa9e819eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.255+0000 7fa9e5e00640 1 -- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e8100e90 msgr2=0x7fa9e819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.255+0000 7fa9e5e00640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e8100e90 0x7fa9e819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.255+0000 7fa9e5e00640 1 -- 172.21.15.103:0/4238767800 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e819f2c0 con 0x7fa9e81004e0 2024-05-23T00:01:45.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.255+0000 7fa9e5e00640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e81004e0 0x7fa9e819a2d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800e9f0 tx=0x7fa9d800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.255+0000 7fa9dea00640 1 -- 172.21.15.103:0/4238767800 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d800cd80 con 0x7fa9e81004e0 2024-05-23T00:01:45.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.255+0000 7fa9dea00640 1 -- 172.21.15.103:0/4238767800 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d8010910 con 0x7fa9e81004e0 2024-05-23T00:01:45.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.255+0000 7fa9dea00640 1 -- 172.21.15.103:0/4238767800 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d8018950 con 0x7fa9e81004e0 2024-05-23T00:01:45.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.255+0000 7fa9e7200640 1 -- 172.21.15.103:0/4238767800 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e819f5b0 con 0x7fa9e81004e0 2024-05-23T00:01:45.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.256+0000 7fa9e7200640 1 -- 172.21.15.103:0/4238767800 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e81a6e40 con 0x7fa9e81004e0 2024-05-23T00:01:45.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.259+0000 7fa9f0004740 1 Processor -- start 2024-05-23T00:01:45.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.259+0000 7fa9f0004740 1 -- start start 2024-05-23T00:01:45.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.259+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.259+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3ac0 0x55cbd42bdb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.259+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42be130 0x55cbd42c0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.259+0000 7fa9f0004740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cbd4217b20 con 0x55cbd42be130 2024-05-23T00:01:45.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.259+0000 7fa9f0004740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cbd41bf1b0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.259+0000 7fa9f0004740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cbd4213ae0 con 0x55cbd42b3110 2024-05-23T00:01:45.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42be130 0x55cbd42c0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42be130 0x55cbd42c0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48796/0 (socket says 172.21.15.103:48796) 2024-05-23T00:01:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e5400640 1 -- 172.21.15.103:0/3263157753 learned_addr learned my addr 172.21.15.103:0/3263157753 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e6800640 1 --2- 172.21.15.103:0/3263157753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e5400640 1 -- 172.21.15.103:0/3263157753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 msgr2=0x55cbd42b34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e5400640 1 --2- 172.21.15.103:0/3263157753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42b34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e5400640 1 -- 172.21.15.103:0/3263157753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3ac0 msgr2=0x55cbd42bdb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-23T00:01:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e5400640 1 --2- 172.21.15.103:0/3263157753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3ac0 0x55cbd42bdb60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.260+0000 7fa9e5400640 1 -- 172.21.15.103:0/3263157753 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbd427b610 con 0x55cbd42be130 2024-05-23T00:01:45.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.261+0000 7fa9dea00640 1 -- 172.21.15.103:0/4238767800 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7fa9d8010ac0 con 0x7fa9e81004e0 2024-05-23T00:01:45.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.261+0000 7fa9e5400640 1 --2- 172.21.15.103:0/3263157753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42be130 0x55cbd42c0580 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009900 tx=0x7fa9d4005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecae0ab379f2280 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.261+0000 7fa9d3400640 1 -- 172.21.15.103:0/3263157753 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4040070 con 0x55cbd42be130 2024-05-23T00:01:45.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.261+0000 7fa9d3400640 1 -- 172.21.15.103:0/3263157753 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4031870 con 0x55cbd42be130 2024-05-23T00:01:45.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.261+0000 7fa9d3400640 1 -- 172.21.15.103:0/3263157753 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4035040 con 0x55cbd42be130 2024-05-23T00:01:45.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 -- 172.21.15.103:0/3263157753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42be130 msgr2=0x55cbd42c0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.264 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3263157753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42be130 0x55cbd42c0580 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009900 tx=0x7fa9d4005be0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 -- 172.21.15.103:0/3263157753 shutdown_connections 2024-05-23T00:01:45.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3263157753 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42be130 0x55cbd42c0580 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3263157753 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3ac0 0x55cbd42bdb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3263157753 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42b34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 -- 172.21.15.103:0/3263157753 >> 172.21.15.103:0/3263157753 conn(0x55cbd42af3d0 msgr2=0x55cbd42af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 -- 172.21.15.103:0/3263157753 shutdown_connections 2024-05-23T00:01:45.265 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.262+0000 7fa9f0004740 1 -- 172.21.15.103:0/3263157753 wait complete. 2024-05-23T00:01:45.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.263+0000 7fa9f0004740 1 -- start start 2024-05-23T00:01:45.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.264+0000 7fa9f0004740 1 Processor -- start 2024-05-23T00:01:45.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.264+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.266 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.264+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42bb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.264+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.264+0000 7fa9f0004740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cbd42c1040 con 0x55cbd42b3ac0 2024-05-23T00:01:45.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.264+0000 7fa9f0004740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55cbd41d2600 con 0x55cbd42bbee0 2024-05-23T00:01:45.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.264+0000 7fa9f0004740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55cbd42c11c0 con 0x55cbd42b3110 2024-05-23T00:01:45.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.264+0000 7fa9f0004740 10 client.0 _ll_register_callbacks cb 0x55cbd420d160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:45.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.265+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.267 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.265+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42ca4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49848/0 (socket says 172.21.15.103:49848) 2024-05-23T00:01:45.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.265+0000 7fa9e5400640 1 -- 172.21.15.103:0/513464416 learned_addr learned my addr 172.21.15.103:0/513464416 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.268 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.265+0000 7fa9e6800640 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.265+0000 7fa9e6800640 1 -- 172.21.15.103:0/513464416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 msgr2=0x55cbd42ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.265+0000 7fa9e6800640 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.265+0000 7fa9e6800640 1 -- 172.21.15.103:0/513464416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 msgr2=0x55cbd42bb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.266+0000 7fa9e5e00640 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42bb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.266+0000 7fa9e6800640 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42bb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.269 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.266+0000 7fa9e6800640 1 -- 172.21.15.103:0/513464416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbd41dfa90 con 0x55cbd42b3110 2024-05-23T00:01:45.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.266+0000 7fa9e5e00640 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42bb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:01:45.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.266+0000 7fa9e6800640 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bb300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8101430 tx=0x7fa9e0011df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4af30459857a61d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.266+0000 7fa9c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:01:45.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.267+0000 7fa9c7e00640 1 -- 172.21.15.103:0/513464416 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0010cf0 con 0x55cbd42b3110 2024-05-23T00:01:45.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.267+0000 7fa9c7e00640 1 -- 172.21.15.103:0/513464416 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa9e0004b20 con 0x55cbd42b3110 2024-05-23T00:01:45.270 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.267+0000 7fa9dea00640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9cc0a2db0 0x7fa9cc0a5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.268+0000 7fa9f0004740 1 -- 172.21.15.103:0/513464416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d4031870 con 0x55cbd42b3110 2024-05-23T00:01:45.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.268+0000 7fa9dea00640 1 -- 172.21.15.103:0/4238767800 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa9d80cfee0 con 0x7fa9e81004e0 2024-05-23T00:01:45.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.268+0000 7fa9e5e00640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9cc0a2db0 0x7fa9cc0a5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.271 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.269+0000 7fa9e5e00640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9cc0a2db0 0x7fa9cc0a5270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa9d80cd0c0 tx=0x7fa9d8018420 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.271+0000 7fa9c7e00640 1 -- 172.21.15.103:0/513464416 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa9e0010e90 con 0x55cbd42b3110 2024-05-23T00:01:45.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.271+0000 7fa9f0004740 10 client.25861 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:45.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.271+0000 7fa9f0004740 1 -- 172.21.15.103:0/513464416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cbd41c70d0 con 0x55cbd42b3110 2024-05-23T00:01:45.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.272+0000 7fa9c7e00640 1 -- 172.21.15.103:0/513464416 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4031870 con 0x55cbd42b3110 2024-05-23T00:01:45.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.272+0000 7fa9c7e00640 1 -- 172.21.15.103:0/513464416 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa9e0024470 con 0x55cbd42b3110 2024-05-23T00:01:45.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.272+0000 7fa9f0004740 10 client.25861 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:45.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.272+0000 7fa9f0004740 10 client.25861 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-23T00:01:45.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.272+0000 7fa9f0004740 20 client.25861 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:45.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.272+0000 7fa9f0004740 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55cbd42ce920 0x55cbd42d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.272+0000 7fa9f0004740 4 client.25861 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:45.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.272+0000 7fa9f0004740 1 -- 172.21.15.103:0/513464416 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cbd4225250 con 0x55cbd42ce920 2024-05-23T00:01:45.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.273+0000 7fa9e5400640 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55cbd42ce920 0x55cbd42d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.275 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.274+0000 7fa9e5400640 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55cbd42ce920 0x55cbd42d0d30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=218886f28e85bf4b server_cookie=5d18fe48b35a4232 in_seq=0 out_seq=0 2024-05-23T00:01:45.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.274+0000 7fa9c7e00640 10 client.25861 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-23T00:01:45.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.274+0000 7fa9c7e00640 1 -- 172.21.15.103:0/513464416 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cbd4225250 con 0x55cbd42ce920 2024-05-23T00:01:45.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.274+0000 7fa9c7e00640 10 client.25861 handle_command_reply: tid=0 2024-05-23T00:01:45.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 client.25861 shutdown 2024-05-23T00:01:45.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 -- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 msgr2=0x55cbd42bb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bb300 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8101430 tx=0x7fa9e0011df0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 -- 172.21.15.103:0/513464416 shutdown_connections 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55cbd42ce920 0x55cbd42d0d30 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42bb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 --2- 172.21.15.103:0/513464416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bb300 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 -- 172.21.15.103:0/513464416 >> 172.21.15.103:0/513464416 conn(0x55cbd42af3d0 msgr2=0x55cbd42b5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9c7e00640 0 client.25861 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.275+0000 7fa9f0004740 1 -- 172.21.15.103:0/513464416 shutdown_connections 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.276+0000 7fa9f0004740 1 -- 172.21.15.103:0/513464416 wait complete. 2024-05-23T00:01:45.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.276+0000 7fa9f0004740 20 client.25861 trim_cache size 0 max 16384 2024-05-23T00:01:45.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.289+0000 7fa9f0004740 1 Processor -- start 2024-05-23T00:01:45.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9f0004740 1 -- start start 2024-05-23T00:01:45.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42d6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42d6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9f0004740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b38d0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9f0004740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b11a0 con 0x55cbd42bbee0 2024-05-23T00:01:45.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9f0004740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81af010 con 0x55cbd42b3110 2024-05-23T00:01:45.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42d6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42d6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48822/0 (socket says 172.21.15.103:48822) 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9e5400640 1 -- 172.21.15.103:0/2048125264 learned_addr learned my addr 172.21.15.103:0/2048125264 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.290+0000 7fa9e6800640 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9e6800640 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42d6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9e5400640 1 -- 172.21.15.103:0/2048125264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 msgr2=0x55cbd42bdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9e5400640 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9e5400640 1 -- 172.21.15.103:0/2048125264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 msgr2=0x55cbd42d6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9e5400640 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42d6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9e5400640 1 -- 172.21.15.103:0/2048125264 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cbd3f2b3f0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9e5400640 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42d6600 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4003ae0 tx=0x7fa9d4003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc9b09ef1b12a6a server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9d2a00640 1 -- 172.21.15.103:0/2048125264 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4035040 con 0x55cbd42b3ac0 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9d2a00640 1 -- 172.21.15.103:0/2048125264 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d40037f0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.291+0000 7fa9d2a00640 1 -- 172.21.15.103:0/2048125264 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4040040 con 0x55cbd42b3ac0 2024-05-23T00:01:45.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.292+0000 7fa9f0004740 1 -- 172.21.15.103:0/2048125264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 msgr2=0x55cbd42d6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.292+0000 7fa9f0004740 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42d6600 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4003ae0 tx=0x7fa9d4003a70 comp rx=0 tx=0).stop 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.293+0000 7fa9f0004740 1 -- 172.21.15.103:0/2048125264 shutdown_connections 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.293+0000 7fa9f0004740 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42bbee0 0x55cbd42d6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.293+0000 7fa9f0004740 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42b3ac0 0x55cbd42d6600 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.293+0000 7fa9f0004740 1 --2- 172.21.15.103:0/2048125264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3110 0x55cbd42bdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.293+0000 7fa9f0004740 1 -- 172.21.15.103:0/2048125264 >> 172.21.15.103:0/2048125264 conn(0x55cbd42af3d0 msgr2=0x55cbd42aa580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.293+0000 7fa9f0004740 1 -- 172.21.15.103:0/2048125264 shutdown_connections 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.293+0000 7fa9f0004740 1 -- 172.21.15.103:0/2048125264 wait complete. 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.293+0000 7fa9f0004740 1 -- start start 2024-05-23T00:01:45.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9f0004740 1 Processor -- start 2024-05-23T00:01:45.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3110 0x55cbd42c6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3ac0 0x55cbd42c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9f0004740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42bbee0 0x55cbd42b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9f0004740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81af660 con 0x55cbd42bbee0 2024-05-23T00:01:45.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9f0004740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81b0ad0 con 0x55cbd42b3110 2024-05-23T00:01:45.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9f0004740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81afd30 con 0x55cbd42b3ac0 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9f0004740 10 client.0 _ll_register_callbacks cb 0x7fa9e0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3ac0 0x55cbd42c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.294+0000 7fa9e6800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3110 0x55cbd42c6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e5400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3ac0 0x55cbd42c6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59212/0 (socket says 172.21.15.103:59212) 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e5400640 1 -- 172.21.15.103:0/3053792574 learned_addr learned my addr 172.21.15.103:0/3053792574 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e6800640 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42bbee0 0x55cbd42b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e5400640 1 -- 172.21.15.103:0/3053792574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3110 msgr2=0x55cbd42c6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e5400640 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3110 0x55cbd42c6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e5400640 1 -- 172.21.15.103:0/3053792574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42bbee0 msgr2=0x55cbd42b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e5400640 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42bbee0 0x55cbd42b79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e5400640 1 -- 172.21.15.103:0/3053792574 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cbd3ff57d0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e6800640 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42bbee0 0x55cbd42b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:45.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9e5400640 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3ac0 0x55cbd42c6950 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009850 tx=0x7fa9d4005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f1c77cbcf89cf99 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.295+0000 7fa9c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.296+0000 7fa9c7e00640 1 -- 172.21.15.103:0/3053792574 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4040070 con 0x55cbd42b3ac0 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.296+0000 7fa9f0004740 1 -- 172.21.15.103:0/3053792574 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9d40037f0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.296+0000 7fa9c7e00640 1 -- 172.21.15.103:0/3053792574 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa9d4005260 con 0x55cbd42b3ac0 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.296+0000 7fa9c7e00640 1 -- 172.21.15.103:0/3053792574 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa9d4063050 con 0x55cbd42b3ac0 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.296+0000 7fa9f0004740 10 client.25867 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.296+0000 7fa9f0004740 1 -- 172.21.15.103:0/3053792574 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cbd433d0c0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.296+0000 7fa9c7e00640 1 -- 172.21.15.103:0/3053792574 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9d40037f0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.297+0000 7fa9c7e00640 1 -- 172.21.15.103:0/3053792574 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa9d40520d0 con 0x55cbd42b3ac0 2024-05-23T00:01:45.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.297+0000 7fa9f0004740 10 client.25867 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.297+0000 7fa9f0004740 10 client.25867 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.297+0000 7fa9f0004740 20 client.25867 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.297+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55cbd433d390 0x55cbd433f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.297+0000 7fa9f0004740 4 client.25867 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.297+0000 7fa9f0004740 1 -- 172.21.15.103:0/3053792574 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cbd421f5a0 con 0x55cbd433d390 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.297+0000 7fa9e6800640 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55cbd433d390 0x55cbd433f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9e6800640 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55cbd433d390 0x55cbd433f780 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=34f67b78f06f559a server_cookie=a2ff47b9bf815c81 in_seq=0 out_seq=0 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9c7e00640 10 client.25867 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9c7e00640 1 -- 172.21.15.103:0/3053792574 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cbd421f5a0 con 0x55cbd433d390 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9c7e00640 10 client.25867 handle_command_reply: tid=0 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 client.25867 shutdown 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 -- 172.21.15.103:0/3053792574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3ac0 msgr2=0x55cbd42c6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3ac0 0x55cbd42c6950 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009850 tx=0x7fa9d4005dd0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 -- 172.21.15.103:0/3053792574 shutdown_connections 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x55cbd433d390 0x55cbd433f780 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cbd42bbee0 0x55cbd42b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9c7e00640 0 client.25867 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55cbd42b3ac0 0x55cbd42c6950 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 --2- 172.21.15.103:0/3053792574 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55cbd42b3110 0x55cbd42c6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.298+0000 7fa9f0004740 1 -- 172.21.15.103:0/3053792574 >> 172.21.15.103:0/3053792574 conn(0x55cbd42af3d0 msgr2=0x55cbd42b9b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.299+0000 7fa9f0004740 1 -- 172.21.15.103:0/3053792574 shutdown_connections 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.299+0000 7fa9f0004740 1 -- 172.21.15.103:0/3053792574 wait complete. 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.299+0000 7fa9f0004740 20 client.25867 trim_cache size 0 max 16384 2024-05-23T00:01:45.303 INFO:teuthology.orchestra.run.smithi103.stdout:[] 2024-05-23T00:01:45.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.299+0000 7fa9e7200640 1 -- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9cc0a2db0 msgr2=0x7fa9cc0a5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.299+0000 7fa9e7200640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9cc0a2db0 0x7fa9cc0a5270 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa9d80cd0c0 tx=0x7fa9d8018420 comp rx=0 tx=0).stop 2024-05-23T00:01:45.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.299+0000 7fa9e7200640 1 -- 172.21.15.103:0/4238767800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e81004e0 msgr2=0x7fa9e819a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.299+0000 7fa9e7200640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e81004e0 0x7fa9e819a2d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800e9f0 tx=0x7fa9d800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.300+0000 7fa9e7200640 1 -- 172.21.15.103:0/4238767800 shutdown_connections 2024-05-23T00:01:45.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.300+0000 7fa9e7200640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa9cc0a2db0 0x7fa9cc0a5270 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.300+0000 7fa9e7200640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9e810e090 0x7fa9e819eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.300+0000 7fa9e7200640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9e8100e90 0x7fa9e819a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.300+0000 7fa9e7200640 1 --2- 172.21.15.103:0/4238767800 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9e81004e0 0x7fa9e819a2d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.300+0000 7fa9e7200640 1 -- 172.21.15.103:0/4238767800 >> 172.21.15.103:0/4238767800 conn(0x7fa9e80fc0f0 msgr2=0x7fa9e80fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.300+0000 7fa9e7200640 1 -- 172.21.15.103:0/4238767800 shutdown_connections 2024-05-23T00:01:45.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.300+0000 7fa9e7200640 1 -- 172.21.15.103:0/4238767800 wait complete. 2024-05-23T00:01:45.325 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-23T00:01:45.378 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:45 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1407522271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:01:45.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.497+0000 7fdb44000640 1 -- 172.21.15.103:0/1753100727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c06a830 msgr2=0x7fdb3c10e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.497+0000 7fdb44000640 1 --2- 172.21.15.103:0/1753100727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c06a830 0x7fdb3c10e710 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009900 tx=0x7fdb3402a3f0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.500 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.498+0000 7fdb44000640 1 -- 172.21.15.103:0/1753100727 shutdown_connections 2024-05-23T00:01:45.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.498+0000 7fdb44000640 1 --2- 172.21.15.103:0/1753100727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c06a830 0x7fdb3c10e710 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.498+0000 7fdb44000640 1 --2- 172.21.15.103:0/1753100727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3c069e70 0x7fdb3c06a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.498+0000 7fdb44000640 1 --2- 172.21.15.103:0/1753100727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb3c107850 0x7fdb3c107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.498+0000 7fdb44000640 1 -- 172.21.15.103:0/1753100727 >> 172.21.15.103:0/1753100727 conn(0x7fdb3c076570 msgr2=0x7fdb3c076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.498+0000 7fdb44000640 1 -- 172.21.15.103:0/1753100727 shutdown_connections 2024-05-23T00:01:45.501 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.498+0000 7fdb44000640 1 -- 172.21.15.103:0/1753100727 wait complete. 2024-05-23T00:01:45.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.499+0000 7fdb44000640 1 Processor -- start 2024-05-23T00:01:45.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.499+0000 7fdb44000640 1 -- start start 2024-05-23T00:01:45.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.499+0000 7fdb44000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3c069e70 0x7fdb3c072fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.499+0000 7fdb44000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb3c06a830 0x7fdb3c073520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.502 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb44000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c107850 0x7fdb3c06c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb44000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c111cb0 con 0x7fdb3c06a830 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb44000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c111b30 con 0x7fdb3c107850 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb44000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c111e30 con 0x7fdb3c069e70 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb40c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb3c06a830 0x7fdb3c073520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb42000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c107850 0x7fdb3c06c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb42000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c107850 0x7fdb3c06c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49890/0 (socket says 172.21.15.103:49890) 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb42000640 1 -- 172.21.15.103:0/234352240 learned_addr learned my addr 172.21.15.103:0/234352240 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb41600640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3c069e70 0x7fdb3c072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.503 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb42000640 1 -- 172.21.15.103:0/234352240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3c069e70 msgr2=0x7fdb3c072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb42000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3c069e70 0x7fdb3c072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb42000640 1 -- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb3c06a830 msgr2=0x7fdb3c073520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb42000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb3c06a830 0x7fdb3c073520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.500+0000 7fdb42000640 1 -- 172.21.15.103:0/234352240 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3c06cb00 con 0x7fdb3c107850 2024-05-23T00:01:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.501+0000 7fdb42000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c107850 0x7fdb3c06c240 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdb3402a910 tx=0x7fdb34003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.501+0000 7fdb3a000640 1 -- 172.21.15.103:0/234352240 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb3403f070 con 0x7fdb3c107850 2024-05-23T00:01:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.501+0000 7fdb3a000640 1 -- 172.21.15.103:0/234352240 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb34031c20 con 0x7fdb3c107850 2024-05-23T00:01:45.504 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.501+0000 7fdb3a000640 1 -- 172.21.15.103:0/234352240 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb34035030 con 0x7fdb3c107850 2024-05-23T00:01:45.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.501+0000 7fdb44000640 1 -- 172.21.15.103:0/234352240 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb3c06cd90 con 0x7fdb3c107850 2024-05-23T00:01:45.505 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.501+0000 7fdb44000640 1 -- 172.21.15.103:0/234352240 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb3c070ce0 con 0x7fdb3c107850 2024-05-23T00:01:45.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.505+0000 7fdb46ddf740 1 Processor -- start 2024-05-23T00:01:45.507 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.506+0000 7fdb3a000640 1 -- 172.21.15.103:0/234352240 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7fdb34042070 con 0x7fdb3c107850 2024-05-23T00:01:45.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.506+0000 7fdb46ddf740 1 -- start start 2024-05-23T00:01:45.508 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.506+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee920320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.506+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.506+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.506+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557bee884b20 con 0x557bee9208f0 2024-05-23T00:01:45.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.506+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557bee82c1b0 con 0x557bee92af60 2024-05-23T00:01:45.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.506+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557bee880ae0 con 0x557bee91ff40 2024-05-23T00:01:45.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb40c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee920320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.509 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb40c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee920320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59234/0 (socket says 172.21.15.103:59234) 2024-05-23T00:01:45.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb40c00640 1 -- 172.21.15.103:0/3614479232 learned_addr learned my addr 172.21.15.103:0/3614479232 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb40c00640 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.510 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb41600640 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb40c00640 1 -- 172.21.15.103:0/3614479232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 msgr2=0x557bee92d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb40c00640 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb40c00640 1 -- 172.21.15.103:0/3614479232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 msgr2=0x557bee92a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.507+0000 7fdb40c00640 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.508+0000 7fdb40c00640 1 -- 172.21.15.103:0/3614479232 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bee8e8610 con 0x557bee91ff40 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.508+0000 7fdb3a000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdb240a2e20 0x7fdb240a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.508+0000 7fdb40c00640 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee920320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdb280058c0 tx=0x7fdb28005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=393f250d5c375c6c server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.508+0000 7fdb3a000640 1 -- 172.21.15.103:0/234352240 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdb340f00b0 con 0x7fdb3c107850 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.508+0000 7fdb2e600640 1 -- 172.21.15.103:0/3614479232 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28005020 con 0x557bee91ff40 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.508+0000 7fdb2e600640 1 -- 172.21.15.103:0/3614479232 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb280047a0 con 0x557bee91ff40 2024-05-23T00:01:45.511 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.508+0000 7fdb2e600640 1 -- 172.21.15.103:0/3614479232 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800f9e0 con 0x557bee91ff40 2024-05-23T00:01:45.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb41600640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdb240a2e20 0x7fdb240a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb46ddf740 1 -- 172.21.15.103:0/3614479232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 msgr2=0x557bee920320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.512 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee920320 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdb280058c0 tx=0x7fdb28005850 comp rx=0 tx=0).stop 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb46ddf740 1 -- 172.21.15.103:0/3614479232 shutdown_connections 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/3614479232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee920320 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb46ddf740 1 -- 172.21.15.103:0/3614479232 >> 172.21.15.103:0/3614479232 conn(0x557bee91c200 msgr2=0x557bee91c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.509+0000 7fdb41600640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdb240a2e20 0x7fdb240a52e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x557bee91d270 tx=0x7fdb30009500 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.510+0000 7fdb46ddf740 1 -- 172.21.15.103:0/3614479232 shutdown_connections 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.510+0000 7fdb46ddf740 1 -- 172.21.15.103:0/3614479232 wait complete. 2024-05-23T00:01:45.513 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.510+0000 7fdb46ddf740 1 -- start start 2024-05-23T00:01:45.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.510+0000 7fdb46ddf740 1 Processor -- start 2024-05-23T00:01:45.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee91ff40 0x557bee927f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee9208f0 0x557bee928480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee92af60 0x557bee9289c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb41600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee91ff40 0x557bee927f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.514 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb42000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee92af60 0x557bee9289c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.516 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb41600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee91ff40 0x557bee927f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49922/0 (socket says 172.21.15.103:49922) 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557bee92ded0 con 0x557bee92af60 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x557bee83f600 con 0x557bee91ff40 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x557bee92e050 con 0x557bee9208f0 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb41600640 1 -- 172.21.15.103:0/2901159595 learned_addr learned my addr 172.21.15.103:0/2901159595 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb40c00640 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee9208f0 0x557bee928480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.511+0000 7fdb46ddf740 10 client.0 _ll_register_callbacks cb 0x557bee87a160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb41600640 1 -- 172.21.15.103:0/2901159595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee9208f0 msgr2=0x557bee928480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb41600640 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee9208f0 0x557bee928480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb41600640 1 -- 172.21.15.103:0/2901159595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee92af60 msgr2=0x557bee9289c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb41600640 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee92af60 0x557bee9289c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.517 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb41600640 1 -- 172.21.15.103:0/2901159595 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bee84ca90 con 0x557bee91ff40 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb41600640 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee91ff40 0x557bee927f40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdb30012930 tx=0x7fdb30004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f2980c59ee5a163 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb22a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb22a00640 1 -- 172.21.15.103:0/2901159595 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb30010df0 con 0x557bee91ff40 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.512+0000 7fdb46ddf740 1 -- 172.21.15.103:0/2901159595 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb280047a0 con 0x557bee91ff40 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb22a00640 1 -- 172.21.15.103:0/2901159595 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdb3001b460 con 0x557bee91ff40 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb22a00640 1 -- 172.21.15.103:0/2901159595 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdb3001b660 con 0x557bee91ff40 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb46ddf740 10 client.25871 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb46ddf740 1 -- 172.21.15.103:0/2901159595 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557bee8340d0 con 0x557bee91ff40 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb22a00640 1 -- 172.21.15.103:0/2901159595 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb280047a0 con 0x557bee91ff40 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb22a00640 1 -- 172.21.15.103:0/2901159595 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdb30024070 con 0x557bee91ff40 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb46ddf740 10 client.25871 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb46ddf740 10 client.25871 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-23T00:01:45.518 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.513+0000 7fdb46ddf740 20 client.25871 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:45.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.514+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x557bee93afb0 0x557bee93d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.514+0000 7fdb46ddf740 4 client.25871 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:45.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.514+0000 7fdb46ddf740 1 -- 172.21.15.103:0/2901159595 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bee892250 con 0x557bee93afb0 2024-05-23T00:01:45.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.514+0000 7fdb40c00640 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x557bee93afb0 0x557bee93d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.515+0000 7fdb40c00640 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x557bee93afb0 0x557bee93d410 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8c0b28a631090d2d server_cookie=ab2ce94b765fcf19 in_seq=0 out_seq=0 2024-05-23T00:01:45.519 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.515+0000 7fdb22a00640 10 client.25871 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-23T00:01:45.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.515+0000 7fdb22a00640 1 -- 172.21.15.103:0/2901159595 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557bee892250 con 0x557bee93afb0 2024-05-23T00:01:45.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.515+0000 7fdb22a00640 10 client.25871 handle_command_reply: tid=0 2024-05-23T00:01:45.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.515+0000 7fdb46ddf740 1 client.25871 shutdown 2024-05-23T00:01:45.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 -- 172.21.15.103:0/2901159595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee91ff40 msgr2=0x557bee927f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee91ff40 0x557bee927f40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdb30012930 tx=0x7fdb30004a40 comp rx=0 tx=0).stop 2024-05-23T00:01:45.520 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 -- 172.21.15.103:0/2901159595 shutdown_connections 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee92af60 0x557bee9289c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x557bee93afb0 0x557bee93d410 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee9208f0 0x557bee928480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/2901159595 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee91ff40 0x557bee927f40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb22a00640 0 client.25871 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 -- 172.21.15.103:0/2901159595 >> 172.21.15.103:0/2901159595 conn(0x557bee91c200 msgr2=0x557bee922070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 -- 172.21.15.103:0/2901159595 shutdown_connections 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.516+0000 7fdb46ddf740 1 -- 172.21.15.103:0/2901159595 wait complete. 2024-05-23T00:01:45.521 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.517+0000 7fdb46ddf740 20 client.25871 trim_cache size 0 max 16384 2024-05-23T00:01:45.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.531+0000 7fdb46ddf740 1 Processor -- start 2024-05-23T00:01:45.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.531+0000 7fdb46ddf740 1 -- start start 2024-05-23T00:01:45.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.531+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee91ff40 0x557bee936d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.531+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee9208f0 0x557bee91b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.531+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee92af60 0x557bee942c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.531+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c1b7d70 con 0x557bee91ff40 2024-05-23T00:01:45.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.531+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c1b5640 con 0x557bee9208f0 2024-05-23T00:01:45.535 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.531+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c1b3490 con 0x557bee92af60 2024-05-23T00:01:45.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb41600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee91ff40 0x557bee936d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee9208f0 0x557bee91b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee9208f0 0x557bee91b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49928/0 (socket says 172.21.15.103:49928) 2024-05-23T00:01:45.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb41600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee91ff40 0x557bee936d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48890/0 (socket says 172.21.15.103:48890) 2024-05-23T00:01:45.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 -- 172.21.15.103:0/4274753133 learned_addr learned my addr 172.21.15.103:0/4274753133 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb42000640 1 --2- 172.21.15.103:0/4274753133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee92af60 0x557bee942c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 -- 172.21.15.103:0/4274753133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee92af60 msgr2=0x557bee942c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 --2- 172.21.15.103:0/4274753133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee92af60 0x557bee942c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 -- 172.21.15.103:0/4274753133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee91ff40 msgr2=0x557bee936d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 --2- 172.21.15.103:0/4274753133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee91ff40 0x557bee936d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 -- 172.21.15.103:0/4274753133 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bee5983f0 con 0x557bee9208f0 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.532+0000 7fdb40c00640 1 --2- 172.21.15.103:0/4274753133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee9208f0 0x557bee91b320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdb280117b0 tx=0x7fdb280046e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5fb0c2eb1d0ac82 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.533+0000 7fdb2dc00640 1 -- 172.21.15.103:0/4274753133 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28030070 con 0x557bee9208f0 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.533+0000 7fdb2dc00640 1 -- 172.21.15.103:0/4274753133 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdb28004520 con 0x557bee9208f0 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.533+0000 7fdb2dc00640 1 -- 172.21.15.103:0/4274753133 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2800f9f0 con 0x557bee9208f0 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.533+0000 7fdb46ddf740 1 -- 172.21.15.103:0/4274753133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee9208f0 msgr2=0x557bee91b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.533+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/4274753133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee9208f0 0x557bee91b320 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdb280117b0 tx=0x7fdb280046e0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 -- 172.21.15.103:0/4274753133 shutdown_connections 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/4274753133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee92af60 0x557bee942c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/4274753133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee9208f0 0x557bee91b320 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/4274753133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee91ff40 0x557bee936d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 -- 172.21.15.103:0/4274753133 >> 172.21.15.103:0/4274753133 conn(0x557bee91c200 msgr2=0x557bee91ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 -- 172.21.15.103:0/4274753133 shutdown_connections 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 -- 172.21.15.103:0/4274753133 wait complete. 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 -- start start 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.534+0000 7fdb46ddf740 1 Processor -- start 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee933620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb46ddf740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c1b3b00 con 0x557bee9208f0 2024-05-23T00:01:45.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c1b4f70 con 0x557bee92af60 2024-05-23T00:01:45.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb46ddf740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3c1b41d0 con 0x557bee91ff40 2024-05-23T00:01:45.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb46ddf740 10 client.0 _ll_register_callbacks cb 0x7fdb30003f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb42000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb42000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49932/0 (socket says 172.21.15.103:49932) 2024-05-23T00:01:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb41600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee933620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb40c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.535+0000 7fdb42000640 1 -- 172.21.15.103:0/638723620 learned_addr learned my addr 172.21.15.103:0/638723620 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb42000640 1 -- 172.21.15.103:0/638723620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 msgr2=0x557bee933620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb42000640 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee933620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb42000640 1 -- 172.21.15.103:0/638723620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 msgr2=0x557bee92e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb42000640 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb42000640 1 -- 172.21.15.103:0/638723620 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bee6627d0 con 0x557bee92af60 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb40c00640 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb42000640 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92eed0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdb3402a9b0 tx=0x7fdb340bbbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0ce58dab3204135 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb22a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb22a00640 1 -- 172.21.15.103:0/638723620 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb340d4070 con 0x557bee92af60 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb46ddf740 1 -- 172.21.15.103:0/638723620 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557bee88c5a0 con 0x557bee92af60 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.536+0000 7fdb22a00640 1 -- 172.21.15.103:0/638723620 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fdb340c49e0 con 0x557bee92af60 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.537+0000 7fdb22a00640 1 -- 172.21.15.103:0/638723620 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fdb34049040 con 0x557bee92af60 2024-05-23T00:01:45.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.537+0000 7fdb46ddf740 10 client.25877 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.537+0000 7fdb46ddf740 1 -- 172.21.15.103:0/638723620 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557bee9a12f0 con 0x557bee92af60 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.537+0000 7fdb22a00640 1 -- 172.21.15.103:0/638723620 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557bee88c5a0 con 0x557bee92af60 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.538+0000 7fdb22a00640 1 -- 172.21.15.103:0/638723620 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fdb340ed510 con 0x557bee92af60 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.538+0000 7fdb46ddf740 10 client.25877 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.538+0000 7fdb46ddf740 10 client.25877 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.538+0000 7fdb46ddf740 20 client.25877 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.538+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x557bee9a1a10 0x557bee9a3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.538+0000 7fdb46ddf740 4 client.25877 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.538+0000 7fdb46ddf740 1 -- 172.21.15.103:0/638723620 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557bee8096e0 con 0x557bee9a1a10 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.538+0000 7fdb41600640 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x557bee9a1a10 0x557bee9a3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb41600640 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x557bee9a1a10 0x557bee9a3e30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cea61d3bbcfd4a28 server_cookie=defefeb5720ff631 in_seq=0 out_seq=0 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb22a00640 10 client.25877 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb22a00640 1 -- 172.21.15.103:0/638723620 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557bee8096e0 con 0x557bee9a1a10 2024-05-23T00:01:45.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb22a00640 10 client.25877 handle_command_reply: tid=0 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 client.25877 shutdown 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 -- 172.21.15.103:0/638723620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 msgr2=0x557bee92eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92eed0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdb3402a9b0 tx=0x7fdb340bbbc0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 -- 172.21.15.103:0/638723620 shutdown_connections 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x557bee92af60 0x557bee92eed0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557bee9208f0 0x557bee92e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x557bee9a1a10 0x557bee9a3e30 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 --2- 172.21.15.103:0/638723620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x557bee91ff40 0x557bee933620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb46ddf740 1 -- 172.21.15.103:0/638723620 >> 172.21.15.103:0/638723620 conn(0x557bee91c200 msgr2=0x557bee928770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.539+0000 7fdb22a00640 0 client.25877 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-23T00:01:45.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.540+0000 7fdb46ddf740 1 -- 172.21.15.103:0/638723620 shutdown_connections 2024-05-23T00:01:45.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.540+0000 7fdb46ddf740 1 -- 172.21.15.103:0/638723620 wait complete. 2024-05-23T00:01:45.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.540+0000 7fdb46ddf740 20 client.25877 trim_cache size 0 max 16384 2024-05-23T00:01:45.546 INFO:teuthology.orchestra.run.smithi103.stdout:[] 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.540+0000 7fdb44000640 1 -- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdb240a2e20 msgr2=0x7fdb240a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.540+0000 7fdb44000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdb240a2e20 0x7fdb240a52e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x557bee91d270 tx=0x7fdb30009500 comp rx=0 tx=0).stop 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.540+0000 7fdb44000640 1 -- 172.21.15.103:0/234352240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c107850 msgr2=0x7fdb3c06c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.540+0000 7fdb44000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c107850 0x7fdb3c06c240 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdb3402a910 tx=0x7fdb34003b90 comp rx=0 tx=0).stop 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.541+0000 7fdb44000640 1 -- 172.21.15.103:0/234352240 shutdown_connections 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.541+0000 7fdb44000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fdb240a2e20 0x7fdb240a52e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.541+0000 7fdb44000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb3c107850 0x7fdb3c06c240 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.541+0000 7fdb44000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdb3c06a830 0x7fdb3c073520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.541+0000 7fdb44000640 1 --2- 172.21.15.103:0/234352240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3c069e70 0x7fdb3c072fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.541+0000 7fdb44000640 1 -- 172.21.15.103:0/234352240 >> 172.21.15.103:0/234352240 conn(0x7fdb3c076570 msgr2=0x7fdb3c0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.541+0000 7fdb44000640 1 -- 172.21.15.103:0/234352240 shutdown_connections 2024-05-23T00:01:45.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.541+0000 7fdb44000640 1 -- 172.21.15.103:0/234352240 wait complete. 2024-05-23T00:01:45.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:45 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1407522271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:01:45.559 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-23T00:01:45.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:45 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1407522271' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:01:45.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.725+0000 7f459b000640 1 -- 172.21.15.103:0/565652539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4594103250 msgr2=0x7f459410b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.725+0000 7f459b000640 1 --2- 172.21.15.103:0/565652539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4594103250 0x7f459410b730 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4580009900 tx=0x7f458002a490 comp rx=0 tx=0).stop 2024-05-23T00:01:45.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.726+0000 7f459b000640 1 -- 172.21.15.103:0/565652539 shutdown_connections 2024-05-23T00:01:45.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.726+0000 7f459b000640 1 --2- 172.21.15.103:0/565652539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f459410bd00 0x7f459410e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.728 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.726+0000 7f459b000640 1 --2- 172.21.15.103:0/565652539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4594103250 0x7f459410b730 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.726+0000 7f459b000640 1 --2- 172.21.15.103:0/565652539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594102930 0x7f4594102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.726+0000 7f459b000640 1 -- 172.21.15.103:0/565652539 >> 172.21.15.103:0/565652539 conn(0x7f45940fc0f0 msgr2=0x7f45940fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.726+0000 7f459b000640 1 -- 172.21.15.103:0/565652539 shutdown_connections 2024-05-23T00:01:45.729 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.726+0000 7f459b000640 1 -- 172.21.15.103:0/565652539 wait complete. 2024-05-23T00:01:45.730 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.727+0000 7f459b000640 1 Processor -- start 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.727+0000 7f459b000640 1 -- start start 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.727+0000 7f459b000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594102930 0x7f459419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.727+0000 7f459b000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594103250 0x7f459419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f459b000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f459410bd00 0x7f459419eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f459b000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4594111b30 con 0x7f4594102930 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f459b000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f45941119b0 con 0x7f459410bd00 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f459b000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4594111cb0 con 0x7f4594103250 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f4599c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594102930 0x7f459419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f4599200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594103250 0x7f459419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f4599200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594103250 0x7f459419a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59280/0 (socket says 172.21.15.103:59280) 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f4599200640 1 -- 172.21.15.103:0/2881076705 learned_addr learned my addr 172.21.15.103:0/2881076705 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.731 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.728+0000 7f459a600640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f459410bd00 0x7f459419eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4599c00640 1 -- 172.21.15.103:0/2881076705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594103250 msgr2=0x7f459419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4599c00640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594103250 0x7f459419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4599c00640 1 -- 172.21.15.103:0/2881076705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f459410bd00 msgr2=0x7f459419eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4599c00640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f459410bd00 0x7f459419eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.732 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4599c00640 1 -- 172.21.15.103:0/2881076705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f459419f220 con 0x7f4594102930 2024-05-23T00:01:45.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4599c00640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594102930 0x7f459419a230 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f458800e9f0 tx=0x7f458800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4592000640 1 -- 172.21.15.103:0/2881076705 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458800ccc0 con 0x7f4594102930 2024-05-23T00:01:45.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4592000640 1 -- 172.21.15.103:0/2881076705 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4588005020 con 0x7f4594102930 2024-05-23T00:01:45.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f459b000640 1 -- 172.21.15.103:0/2881076705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f459419f510 con 0x7f4594102930 2024-05-23T00:01:45.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f4592000640 1 -- 172.21.15.103:0/2881076705 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45880049e0 con 0x7f4594102930 2024-05-23T00:01:45.733 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.729+0000 7f459b000640 1 -- 172.21.15.103:0/2881076705 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45941a6da0 con 0x7f4594102930 2024-05-23T00:01:45.735 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.734+0000 7f459f3bb740 1 Processor -- start 2024-05-23T00:01:45.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.734+0000 7f459f3bb740 1 -- start start 2024-05-23T00:01:45.736 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.734+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.734+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1acb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.734+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.734+0000 7f459f3bb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5570fa106b20 con 0x5570fa1a2110 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.734+0000 7f459f3bb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5570fa0ae1b0 con 0x5570fa1a2ac0 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.734+0000 7f459f3bb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5570fa102ae0 con 0x5570fa1ad130 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.735+0000 7f459a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1a24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.735+0000 7f459a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1a24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48932/0 (socket says 172.21.15.103:48932) 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.735+0000 7f459a600640 1 -- 172.21.15.103:0/767884653 learned_addr learned my addr 172.21.15.103:0/767884653 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.737 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.735+0000 7f4592000640 1 -- 172.21.15.103:0/2881076705 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f4588005560 con 0x7f4594102930 2024-05-23T00:01:45.738 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.735+0000 7f4599200640 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.735+0000 7f4599c00640 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.735+0000 7f459a600640 1 -- 172.21.15.103:0/767884653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 msgr2=0x5570fa1af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.735+0000 7f459a600640 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.736+0000 7f459a600640 1 -- 172.21.15.103:0/767884653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 msgr2=0x5570fa1acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.736+0000 7f459a600640 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.736+0000 7f459a600640 1 -- 172.21.15.103:0/767884653 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570fa16a610 con 0x5570fa1a2110 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.736+0000 7f4599c00640 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.736+0000 7f459a600640 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1a24f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f458c009f50 tx=0x7f458c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb2df7aea11decd server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.739 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.737+0000 7f4586a00640 1 -- 172.21.15.103:0/767884653 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c004d20 con 0x5570fa1a2110 2024-05-23T00:01:45.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.737+0000 7f4586a00640 1 -- 172.21.15.103:0/767884653 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f458c004050 con 0x5570fa1a2110 2024-05-23T00:01:45.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.737+0000 7f4586a00640 1 -- 172.21.15.103:0/767884653 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c005280 con 0x5570fa1a2110 2024-05-23T00:01:45.740 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.737+0000 7f4592000640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f457c0a2ea0 0x7f457c0a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.737+0000 7f4592000640 1 -- 172.21.15.103:0/2881076705 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f45880cfab0 con 0x7f4594102930 2024-05-23T00:01:45.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.737+0000 7f4599c00640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f457c0a2ea0 0x7f457c0a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.737+0000 7f459f3bb740 1 -- 172.21.15.103:0/767884653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 msgr2=0x5570fa1a24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.737+0000 7f459f3bb740 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1a24f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f458c009f50 tx=0x7f458c0118d0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.738+0000 7f459f3bb740 1 -- 172.21.15.103:0/767884653 shutdown_connections 2024-05-23T00:01:45.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.738+0000 7f459f3bb740 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.743 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.738+0000 7f459f3bb740 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.738+0000 7f459f3bb740 1 --2- 172.21.15.103:0/767884653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1a24f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.738+0000 7f459f3bb740 1 -- 172.21.15.103:0/767884653 >> 172.21.15.103:0/767884653 conn(0x5570fa19e3d0 msgr2=0x5570fa19e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.738+0000 7f4599c00640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f457c0a2ea0 0x7f457c0a5360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f45880ccc90 tx=0x7f45880054d0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.738+0000 7f459f3bb740 1 -- 172.21.15.103:0/767884653 shutdown_connections 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.738+0000 7f459f3bb740 1 -- 172.21.15.103:0/767884653 wait complete. 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.739+0000 7f459f3bb740 1 -- start start 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.739+0000 7f459f3bb740 1 Processor -- start 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.740+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2110 0x5570fa1aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.740+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 0x5570fa1aa610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.740+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1aab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.740+0000 7f459f3bb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5570fa1b00a0 con 0x5570fa1a2ac0 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.740+0000 7f459f3bb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5570fa0c1600 con 0x5570fa1a2110 2024-05-23T00:01:45.744 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.740+0000 7f459f3bb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5570fa1b0220 con 0x5570fa1ad130 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.740+0000 7f459f3bb740 10 client.0 _ll_register_callbacks cb 0x5570fa0fc160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2110 0x5570fa1aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2110 0x5570fa1aa0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49952/0 (socket says 172.21.15.103:49952) 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 -- 172.21.15.103:0/2421235513 learned_addr learned my addr 172.21.15.103:0/2421235513 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f4599200640 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 0x5570fa1aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 -- 172.21.15.103:0/2421235513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 msgr2=0x5570fa1aab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1aab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 -- 172.21.15.103:0/2421235513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 msgr2=0x5570fa1aa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 0x5570fa1aa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.745 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 -- 172.21.15.103:0/2421235513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570fa0cea90 con 0x5570fa1a2110 2024-05-23T00:01:45.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.741+0000 7f459a600640 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2110 0x5570fa1aa0d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f458c005c40 tx=0x7f458c005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8189c516f1afcc6b server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.742+0000 7f457b400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:45.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.742+0000 7f457b400640 1 -- 172.21.15.103:0/2421235513 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c02b070 con 0x5570fa1a2110 2024-05-23T00:01:45.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.742+0000 7f457b400640 1 -- 172.21.15.103:0/2421235513 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f458c027440 con 0x5570fa1a2110 2024-05-23T00:01:45.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.742+0000 7f459f3bb740 1 -- 172.21.15.103:0/2421235513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f458c004050 con 0x5570fa1a2110 2024-05-23T00:01:45.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.742+0000 7f457b400640 1 -- 172.21.15.103:0/2421235513 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f458c034050 con 0x5570fa1a2110 2024-05-23T00:01:45.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.742+0000 7f459f3bb740 10 client.25886 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:45.748 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.742+0000 7f459f3bb740 1 -- 172.21.15.103:0/2421235513 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5570fa0b60d0 con 0x5570fa1a2110 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.743+0000 7f457b400640 1 -- 172.21.15.103:0/2421235513 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f458c004050 con 0x5570fa1a2110 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.743+0000 7f457b400640 1 -- 172.21.15.103:0/2421235513 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f458c00f5a0 con 0x5570fa1a2110 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.743+0000 7f459f3bb740 10 client.25886 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.743+0000 7f459f3bb740 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.743+0000 7f459f3bb740 20 client.25886 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.743+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5570fa1bd240 0x5570fa1bf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.743+0000 7f459f3bb740 4 client.25886 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.743+0000 7f459f3bb740 1 -- 172.21.15.103:0/2421235513 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570fa114250 con 0x5570fa1bd240 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.744+0000 7f4599200640 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5570fa1bd240 0x5570fa1bf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.744+0000 7f4599200640 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5570fa1bd240 0x5570fa1bf630 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8517156f48d1267 server_cookie=e36f9fc35d08b2cb in_seq=0 out_seq=0 2024-05-23T00:01:45.749 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.744+0000 7f457b400640 10 client.25886 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f457b400640 1 -- 172.21.15.103:0/2421235513 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5570fa114250 con 0x5570fa1bd240 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f457b400640 10 client.25886 handle_command_reply: tid=0 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 client.25886 shutdown 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 -- 172.21.15.103:0/2421235513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2110 msgr2=0x5570fa1aa0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2110 0x5570fa1aa0d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f458c005c40 tx=0x7f458c005bd0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 -- 172.21.15.103:0/2421235513 shutdown_connections 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5570fa1bd240 0x5570fa1bf630 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1aab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 0x5570fa1aa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2421235513 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2110 0x5570fa1aa0d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.745+0000 7f459f3bb740 1 -- 172.21.15.103:0/2421235513 >> 172.21.15.103:0/2421235513 conn(0x5570fa19e3d0 msgr2=0x5570fa1a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.746+0000 7f457b400640 0 client.25886 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.746+0000 7f459f3bb740 1 -- 172.21.15.103:0/2421235513 shutdown_connections 2024-05-23T00:01:45.750 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.746+0000 7f459f3bb740 1 -- 172.21.15.103:0/2421235513 wait complete. 2024-05-23T00:01:45.751 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.746+0000 7f459f3bb740 20 client.25886 trim_cache size 0 max 16384 2024-05-23T00:01:45.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.760+0000 7f459f3bb740 1 Processor -- start 2024-05-23T00:01:45.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.760+0000 7f459f3bb740 1 -- start start 2024-05-23T00:01:45.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.760+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1a2110 0x5570fa1b8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.760+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 0x5570fa1aa090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.760+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1ad130 0x5570fa1a9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.760+0000 7f459f3bb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45941b3870 con 0x5570fa1a2ac0 2024-05-23T00:01:45.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.760+0000 7f459f3bb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f45941b1140 con 0x5570fa1ad130 2024-05-23T00:01:45.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.760+0000 7f459f3bb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45941aef90 con 0x5570fa1a2110 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1a2110 0x5570fa1b8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1a2110 0x5570fa1b8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59314/0 (socket says 172.21.15.103:59314) 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 -- 172.21.15.103:0/2956633215 learned_addr learned my addr 172.21.15.103:0/2956633215 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f4599200640 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 0x5570fa1aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1ad130 0x5570fa1a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 -- 172.21.15.103:0/2956633215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1ad130 msgr2=0x5570fa1a9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1ad130 0x5570fa1a9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 -- 172.21.15.103:0/2956633215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 msgr2=0x5570fa1aa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 0x5570fa1aa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 -- 172.21.15.103:0/2956633215 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570fa1aa770 con 0x5570fa1a2110 2024-05-23T00:01:45.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.761+0000 7f459a600640 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1a2110 0x5570fa1b8f60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f458c009bb0 tx=0x7f458c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1deb079bead5676f server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.762+0000 7f4586000640 1 -- 172.21.15.103:0/2956633215 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c02b070 con 0x5570fa1a2110 2024-05-23T00:01:45.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.762+0000 7f4586000640 1 -- 172.21.15.103:0/2956633215 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f458c003c40 con 0x5570fa1a2110 2024-05-23T00:01:45.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.762+0000 7f4586000640 1 -- 172.21.15.103:0/2956633215 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c039940 con 0x5570fa1a2110 2024-05-23T00:01:45.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.762+0000 7f459f3bb740 1 -- 172.21.15.103:0/2956633215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1a2110 msgr2=0x5570fa1b8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.762+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1a2110 0x5570fa1b8f60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f458c009bb0 tx=0x7f458c0118d0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.766 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.762+0000 7f459f3bb740 1 -- 172.21.15.103:0/2956633215 shutdown_connections 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.762+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1ad130 0x5570fa1a9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.763+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2ac0 0x5570fa1aa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.763+0000 7f459f3bb740 1 --2- 172.21.15.103:0/2956633215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1a2110 0x5570fa1b8f60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.763+0000 7f459f3bb740 1 -- 172.21.15.103:0/2956633215 >> 172.21.15.103:0/2956633215 conn(0x5570fa19e3d0 msgr2=0x5570fa19ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.763+0000 7f459f3bb740 1 -- 172.21.15.103:0/2956633215 shutdown_connections 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.763+0000 7f459f3bb740 1 -- 172.21.15.103:0/2956633215 wait complete. 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.763+0000 7f459f3bb740 1 -- start start 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.763+0000 7f459f3bb740 1 Processor -- start 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1b51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1b5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459f3bb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1b0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459f3bb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f45941af600 con 0x5570fa1a2110 2024-05-23T00:01:45.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459f3bb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f45941b0a70 con 0x5570fa1a2ac0 2024-05-23T00:01:45.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459f3bb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45941afcd0 con 0x5570fa1ad130 2024-05-23T00:01:45.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459f3bb740 10 client.0 _ll_register_callbacks cb 0x7f4588005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:45.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1b51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1b51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:48972/0 (socket says 172.21.15.103:48972) 2024-05-23T00:01:45.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459a600640 1 -- 172.21.15.103:0/1119700774 learned_addr learned my addr 172.21.15.103:0/1119700774 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:45.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f459a600640 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.764+0000 7f4599200640 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f459a600640 1 -- 172.21.15.103:0/1119700774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 msgr2=0x5570fa1b0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f459a600640 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1b0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f459a600640 1 -- 172.21.15.103:0/1119700774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 msgr2=0x5570fa1b5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:45.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f459a600640 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1b5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f459a600640 1 -- 172.21.15.103:0/1119700774 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570f9e1a3f0 con 0x5570fa1a2110 2024-05-23T00:01:45.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f459a600640 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1b51c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f458c0395e0 tx=0x7f458c039440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc8f9c85fed3f64 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:45.770 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f457b400640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f457b400640 1 -- 172.21.15.103:0/1119700774 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f458c02b070 con 0x5570fa1a2110 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f459f3bb740 1 -- 172.21.15.103:0/1119700774 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f458c003c40 con 0x5570fa1a2110 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.765+0000 7f457b400640 1 -- 172.21.15.103:0/1119700774 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f458c00f680 con 0x5570fa1a2110 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f457b400640 1 -- 172.21.15.103:0/1119700774 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f458c04e050 con 0x5570fa1a2110 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f459f3bb740 10 client.15798 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f459f3bb740 1 -- 172.21.15.103:0/1119700774 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5570f9ee47d0 con 0x5570fa1a2110 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f457b400640 1 -- 172.21.15.103:0/1119700774 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f458c003c40 con 0x5570fa1a2110 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f457b400640 1 -- 172.21.15.103:0/1119700774 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f458c0457a0 con 0x5570fa1a2110 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f459f3bb740 10 client.15798 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f459f3bb740 10 client.15798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f459f3bb740 20 client.15798 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:45.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f459f3bb740 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5570fa223520 0x5570fa225940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f459f3bb740 4 client.15798 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.766+0000 7f459f3bb740 1 -- 172.21.15.103:0/1119700774 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5570fa10e5a0 con 0x5570fa223520 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.767+0000 7f4599200640 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5570fa223520 0x5570fa225940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f457b400640 10 client.15798 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f4599200640 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5570fa223520 0x5570fa225940 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b2b0d24a22d8248 server_cookie=9d26bb7d3f25d785 in_seq=0 out_seq=0 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f457b400640 1 -- 172.21.15.103:0/1119700774 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5570fa10e5a0 con 0x5570fa223520 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f457b400640 10 client.15798 handle_command_reply: tid=0 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 client.15798 shutdown 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 -- 172.21.15.103:0/1119700774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 msgr2=0x5570fa1b51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1b51c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f458c0395e0 tx=0x7f458c039440 comp rx=0 tx=0).stop 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 -- 172.21.15.103:0/1119700774 shutdown_connections 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5570fa223520 0x5570fa225940 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5570fa1ad130 0x5570fa1b0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5570fa1a2ac0 0x5570fa1b5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 --2- 172.21.15.103:0/1119700774 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570fa1a2110 0x5570fa1b51c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f459f3bb740 1 -- 172.21.15.103:0/1119700774 >> 172.21.15.103:0/1119700774 conn(0x5570fa19e3d0 msgr2=0x5570fa19d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.768+0000 7f457b400640 0 client.15798 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:45.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.769+0000 7f459f3bb740 1 -- 172.21.15.103:0/1119700774 shutdown_connections 2024-05-23T00:01:45.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.769+0000 7f459f3bb740 1 -- 172.21.15.103:0/1119700774 wait complete. 2024-05-23T00:01:45.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.769+0000 7f459f3bb740 20 client.15798 trim_cache size 0 max 16384 2024-05-23T00:01:45.774 INFO:teuthology.orchestra.run.smithi103.stdout:[] 2024-05-23T00:01:45.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.769+0000 7f459b000640 1 -- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f457c0a2ea0 msgr2=0x7f457c0a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.769+0000 7f459b000640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f457c0a2ea0 0x7f457c0a5360 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f45880ccc90 tx=0x7f45880054d0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.769+0000 7f459b000640 1 -- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594102930 msgr2=0x7f459419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.769+0000 7f459b000640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594102930 0x7f459419a230 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f458800e9f0 tx=0x7f458800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.770+0000 7f459b000640 1 -- 172.21.15.103:0/2881076705 shutdown_connections 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.770+0000 7f459b000640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f457c0a2ea0 0x7f457c0a5360 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.770+0000 7f459b000640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f459410bd00 0x7f459419eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.770+0000 7f459b000640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4594103250 0x7f459419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.770+0000 7f459b000640 1 --2- 172.21.15.103:0/2881076705 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4594102930 0x7f459419a230 secure :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f458800e9f0 tx=0x7f458800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.770+0000 7f459b000640 1 -- 172.21.15.103:0/2881076705 >> 172.21.15.103:0/2881076705 conn(0x7f45940fc0f0 msgr2=0x7f45941004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.770+0000 7f459b000640 1 -- 172.21.15.103:0/2881076705 shutdown_connections 2024-05-23T00:01:45.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:45.770+0000 7f459b000640 1 -- 172.21.15.103:0/2881076705 wait complete. 2024-05-23T00:01:46.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:46 smithi105 ceph-mon[36426]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-23T00:01:46.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:46 smithi103 ceph-mon[31370]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-23T00:01:46.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:46 smithi174 ceph-mon[36179]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-23T00:01:46.791 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-23T00:01:46.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:01:46.792 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-23T00:01:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.997+0000 7faf9a400640 1 -- 172.21.15.103:0/3556600555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf941021e0 msgr2=0x7faf9410e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.997+0000 7faf9a400640 1 --2- 172.21.15.103:0/3556600555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf941021e0 0x7faf9410e6f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faf8c009c10 tx=0x7faf8c02a6e0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.997+0000 7faf9a400640 1 -- 172.21.15.103:0/3556600555 shutdown_connections 2024-05-23T00:01:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.997+0000 7faf9a400640 1 --2- 172.21.15.103:0/3556600555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf941021e0 0x7faf9410e6f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.997+0000 7faf9a400640 1 --2- 172.21.15.103:0/3556600555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf94101820 0x7faf94101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.997+0000 7faf9a400640 1 --2- 172.21.15.103:0/3556600555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf94107830 0x7faf94107c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.997+0000 7faf9a400640 1 -- 172.21.15.103:0/3556600555 >> 172.21.15.103:0/3556600555 conn(0x7faf940fd2f0 msgr2=0x7faf940ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.998+0000 7faf9a400640 1 -- 172.21.15.103:0/3556600555 shutdown_connections 2024-05-23T00:01:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.998+0000 7faf9a400640 1 -- 172.21.15.103:0/3556600555 wait complete. 2024-05-23T00:01:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf9a400640 1 Processor -- start 2024-05-23T00:01:47.001 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf9a400640 1 -- start start 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf9a400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf94101820 0x7faf94108770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf9a400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf941021e0 0x7faf94108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf9a400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf94107830 0x7faf941091f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf9a400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faf94111c40 con 0x7faf94107830 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf9a400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf94111ac0 con 0x7faf94101820 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf9a400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf94111dc0 con 0x7faf941021e0 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf93400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf94101820 0x7faf94108770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:46.999+0000 7faf93400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf94101820 0x7faf94108770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49982/0 (socket says 172.21.15.103:49982) 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf92a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf941021e0 0x7faf94108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf93400640 1 -- 172.21.15.103:0/3749886687 learned_addr learned my addr 172.21.15.103:0/3749886687 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.002 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf93e00640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf94107830 0x7faf941091f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf93400640 1 -- 172.21.15.103:0/3749886687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf941021e0 msgr2=0x7faf94108cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf93400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf941021e0 0x7faf94108cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.003 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf93400640 1 -- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf94107830 msgr2=0x7faf941091f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf93400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf94107830 0x7faf941091f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf93400640 1 -- 172.21.15.103:0/3749886687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9419a0d0 con 0x7faf94101820 2024-05-23T00:01:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.000+0000 7faf93400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf94101820 0x7faf94108770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faf8400e9f0 tx=0x7faf8400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.001+0000 7faf8be00640 1 -- 172.21.15.103:0/3749886687 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8400ccc0 con 0x7faf94101820 2024-05-23T00:01:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.001+0000 7faf9a400640 1 -- 172.21.15.103:0/3749886687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9419a3c0 con 0x7faf94101820 2024-05-23T00:01:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.001+0000 7faf8be00640 1 -- 172.21.15.103:0/3749886687 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf8400ce60 con 0x7faf94101820 2024-05-23T00:01:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.001+0000 7faf8be00640 1 -- 172.21.15.103:0/3749886687 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84010690 con 0x7faf94101820 2024-05-23T00:01:47.004 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.001+0000 7faf9a400640 1 -- 172.21.15.103:0/3749886687 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9419a8c0 con 0x7faf94101820 2024-05-23T00:01:47.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf9d39e740 1 Processor -- start 2024-05-23T00:01:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf9d39e740 1 -- start start 2024-05-23T00:01:47.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf01b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf026000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf028930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf9d39e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5580cef7f7c0 con 0x5580cf026540 2024-05-23T00:01:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf9d39e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5580cef27270 con 0x5580cf01b550 2024-05-23T00:01:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf9d39e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580cef7b780 con 0x5580cf01be70 2024-05-23T00:01:47.008 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.005+0000 7faf8be00640 1 -- 172.21.15.103:0/3749886687 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7faf840046c0 con 0x7faf94101820 2024-05-23T00:01:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.006+0000 7faf93e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf01b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.006+0000 7faf93e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf01b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:49990/0 (socket says 172.21.15.103:49990) 2024-05-23T00:01:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.006+0000 7faf93e00640 1 -- 172.21.15.103:0/2209583941 learned_addr learned my addr 172.21.15.103:0/2209583941 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.006+0000 7faf92a00640 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf028930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.006+0000 7faf93400640 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf026000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.006+0000 7faf93400640 1 -- 172.21.15.103:0/2209583941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 msgr2=0x5580cf01b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.006+0000 7faf93400640 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf01b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.006+0000 7faf93400640 1 -- 172.21.15.103:0/2209583941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 msgr2=0x5580cf028930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.007+0000 7faf93400640 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf028930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.007+0000 7faf93400640 1 -- 172.21.15.103:0/2209583941 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580ced5b7e0 con 0x5580cf01be70 2024-05-23T00:01:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.007+0000 7faf93e00640 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf01b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.007+0000 7faf92a00640 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf028930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.007+0000 7faf93400640 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf026000 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faf840cd040 tx=0x7faf840d3ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a0d1115b7869554 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.007+0000 7faf8be00640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7faf780a2e20 0x7faf780a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.007+0000 7faf8be00640 1 -- 172.21.15.103:0/3749886687 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7faf840cfe60 con 0x7faf94101820 2024-05-23T00:01:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.008+0000 7faf93e00640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7faf780a2e20 0x7faf780a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.008+0000 7faf88c00640 1 -- 172.21.15.103:0/2209583941 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf840e3070 con 0x5580cf01be70 2024-05-23T00:01:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.008+0000 7faf88c00640 1 -- 172.21.15.103:0/2209583941 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf8409c440 con 0x5580cf01be70 2024-05-23T00:01:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.008+0000 7faf88c00640 1 -- 172.21.15.103:0/2209583941 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8409b6a0 con 0x5580cf01be70 2024-05-23T00:01:47.011 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.009+0000 7faf9d39e740 1 -- 172.21.15.103:0/2209583941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 msgr2=0x5580cf026000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.009+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf026000 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faf840cd040 tx=0x7faf840d3ef0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.009+0000 7faf93e00640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7faf780a2e20 0x7faf780a52e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faf8c004af0 tx=0x7faf8c005fd0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.009+0000 7faf9d39e740 1 -- 172.21.15.103:0/2209583941 shutdown_connections 2024-05-23T00:01:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.009+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf028930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.009+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf026000 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.009+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2209583941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf01b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.009+0000 7faf9d39e740 1 -- 172.21.15.103:0/2209583941 >> 172.21.15.103:0/2209583941 conn(0x5580cf017870 msgr2=0x5580cf017c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.010+0000 7faf9d39e740 1 -- 172.21.15.103:0/2209583941 shutdown_connections 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.010+0000 7faf9d39e740 1 -- 172.21.15.103:0/2209583941 wait complete. 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.010+0000 7faf9d39e740 1 -- start start 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.010+0000 7faf9d39e740 1 Processor -- start 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01b550 0x5580cf0235b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf023af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf026540 0x5580cf024030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf9d39e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5580cf029330 con 0x5580cf01b550 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf9d39e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5580cef3a620 con 0x5580cf026540 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf9d39e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580cf0294b0 con 0x5580cf01be70 2024-05-23T00:01:47.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf9d39e740 10 client.0 _ll_register_callbacks cb 0x5580cef288b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01b550 0x5580cf0235b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01b550 0x5580cf0235b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49016/0 (socket says 172.21.15.103:49016) 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93400640 1 -- 172.21.15.103:0/2459009756 learned_addr learned my addr 172.21.15.103:0/2459009756 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf92a00640 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf023af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93e00640 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf026540 0x5580cf024030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93400640 1 -- 172.21.15.103:0/2459009756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 msgr2=0x5580cf023af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93400640 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf023af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93400640 1 -- 172.21.15.103:0/2459009756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf026540 msgr2=0x5580cf024030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93400640 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf026540 0x5580cf024030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.011+0000 7faf93400640 1 -- 172.21.15.103:0/2459009756 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580cedbbf70 con 0x5580cf01b550 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.012+0000 7faf93400640 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01b550 0x5580cf0235b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7faf84098710 tx=0x7faf84005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d50b78fe743ea016 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.012+0000 7faf7d600640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.012+0000 7faf7d600640 1 -- 172.21.15.103:0/2459009756 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8409b6a0 con 0x5580cf01b550 2024-05-23T00:01:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.012+0000 7faf7d600640 1 -- 172.21.15.103:0/2459009756 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7faf8409c1a0 con 0x5580cf01b550 2024-05-23T00:01:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.012+0000 7faf9d39e740 1 -- 172.21.15.103:0/2459009756 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf8409c440 con 0x5580cf01b550 2024-05-23T00:01:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.013+0000 7faf7d600640 1 -- 172.21.15.103:0/2459009756 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf8409bb10 con 0x5580cf01b550 2024-05-23T00:01:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.013+0000 7faf9d39e740 10 client.15810 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.013+0000 7faf9d39e740 1 -- 172.21.15.103:0/2459009756 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5580cec53170 con 0x5580cf01b550 2024-05-23T00:01:47.017 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.013+0000 7faf7d600640 1 -- 172.21.15.103:0/2459009756 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf8409c440 con 0x5580cf01b550 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.013+0000 7faf7d600640 1 -- 172.21.15.103:0/2459009756 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7faf840a4590 con 0x5580cf01b550 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.014+0000 7faf9d39e740 10 client.15810 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.014+0000 7faf9d39e740 10 client.15810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.014+0000 7faf9d39e740 20 client.15810 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.014+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580cf036830 0x5580cf038c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.014+0000 7faf9d39e740 4 client.15810 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.014+0000 7faf9d39e740 1 -- 172.21.15.103:0/2459009756 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580cef931e0 con 0x5580cf036830 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.014+0000 7faf92a00640 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580cf036830 0x5580cf038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.015+0000 7faf92a00640 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580cf036830 0x5580cf038c20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77c6e795680234d2 server_cookie=9bc52a8c6bc69d70 in_seq=0 out_seq=0 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.015+0000 7faf7d600640 10 client.15810 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.015+0000 7faf7d600640 1 -- 172.21.15.103:0/2459009756 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5580cef931e0 con 0x5580cf036830 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.015+0000 7faf7d600640 10 client.15810 handle_command_reply: tid=0 2024-05-23T00:01:47.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.015+0000 7faf9d39e740 1 client.15810 shutdown 2024-05-23T00:01:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 -- 172.21.15.103:0/2459009756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01b550 msgr2=0x5580cf0235b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01b550 0x5580cf0235b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7faf84098710 tx=0x7faf84005720 comp rx=0 tx=0).stop 2024-05-23T00:01:47.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 -- 172.21.15.103:0/2459009756 shutdown_connections 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580cf036830 0x5580cf038c20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf026540 0x5580cf024030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01be70 0x5580cf023af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2459009756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01b550 0x5580cf0235b0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 -- 172.21.15.103:0/2459009756 >> 172.21.15.103:0/2459009756 conn(0x5580cf017870 msgr2=0x5580cf01d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf7d600640 0 client.15810 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 -- 172.21.15.103:0/2459009756 shutdown_connections 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 1 -- 172.21.15.103:0/2459009756 wait complete. 2024-05-23T00:01:47.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.016+0000 7faf9d39e740 20 client.15810 trim_cache size 0 max 16384 2024-05-23T00:01:47.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.034+0000 7faf9d39e740 1 Processor -- start 2024-05-23T00:01:47.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf9d39e740 1 -- start start 2024-05-23T00:01:47.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01b550 0x5580cf01dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01be70 0x5580cf03bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.037 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf03c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf9d39e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faf941c0620 con 0x5580cf026540 2024-05-23T00:01:47.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf9d39e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf941bdef0 con 0x5580cf01be70 2024-05-23T00:01:47.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf9d39e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf94067e00 con 0x5580cf01b550 2024-05-23T00:01:47.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf93400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01b550 0x5580cf01dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf93e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf03c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf93400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01b550 0x5580cf01dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59372/0 (socket says 172.21.15.103:59372) 2024-05-23T00:01:47.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf93400640 1 -- 172.21.15.103:0/3021631515 learned_addr learned my addr 172.21.15.103:0/3021631515 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.038 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.035+0000 7faf92a00640 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01be70 0x5580cf03bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.039 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf93400640 1 -- 172.21.15.103:0/3021631515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01be70 msgr2=0x5580cf03bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf93400640 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01be70 0x5580cf03bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf93400640 1 -- 172.21.15.103:0/3021631515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 msgr2=0x5580cf03c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf93400640 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf03c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf93400640 1 -- 172.21.15.103:0/3021631515 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580cf03ea60 con 0x5580cf01b550 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf93e00640 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf03c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf93400640 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01b550 0x5580cf01dda0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faf8400c6a0 tx=0x7faf8409a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcbe70dc42d0e828 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf7fe00640 1 -- 172.21.15.103:0/3021631515 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf84097fe0 con 0x5580cf01b550 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf7fe00640 1 -- 172.21.15.103:0/3021631515 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf840ccf80 con 0x5580cf01b550 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.036+0000 7faf7fe00640 1 -- 172.21.15.103:0/3021631515 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf840cd5d0 con 0x5580cf01b550 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.037+0000 7faf9d39e740 1 -- 172.21.15.103:0/3021631515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01b550 msgr2=0x5580cf01dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.037+0000 7faf9d39e740 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01b550 0x5580cf01dda0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faf8400c6a0 tx=0x7faf8409a250 comp rx=0 tx=0).stop 2024-05-23T00:01:47.040 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.037+0000 7faf9d39e740 1 -- 172.21.15.103:0/3021631515 shutdown_connections 2024-05-23T00:01:47.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.037+0000 7faf9d39e740 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf026540 0x5580cf03c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.037+0000 7faf9d39e740 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01be70 0x5580cf03bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.037+0000 7faf9d39e740 1 --2- 172.21.15.103:0/3021631515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf01b550 0x5580cf01dda0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.037+0000 7faf9d39e740 1 -- 172.21.15.103:0/3021631515 >> 172.21.15.103:0/3021631515 conn(0x5580cf017870 msgr2=0x5580cf031f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.037+0000 7faf9d39e740 1 -- 172.21.15.103:0/3021631515 shutdown_connections 2024-05-23T00:01:47.041 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.038+0000 7faf9d39e740 1 -- 172.21.15.103:0/3021631515 wait complete. 2024-05-23T00:01:47.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.038+0000 7faf9d39e740 1 -- start start 2024-05-23T00:01:47.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.038+0000 7faf9d39e740 1 Processor -- start 2024-05-23T00:01:47.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf039da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01be70 0x5580cf03a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf9d39e740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf026540 0x5580cf09ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf9d39e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faf941bc3d0 con 0x5580cf01be70 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf9d39e740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7faf941bd820 con 0x5580cf01b550 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf9d39e740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf941bca80 con 0x5580cf026540 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf9d39e740 10 client.0 _ll_register_callbacks cb 0x7faf840e1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf92a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01be70 0x5580cf03a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf92a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01be70 0x5580cf03a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49038/0 (socket says 172.21.15.103:49038) 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf92a00640 1 -- 172.21.15.103:0/2463297852 learned_addr learned my addr 172.21.15.103:0/2463297852 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf93e00640 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf026540 0x5580cf09ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.039+0000 7faf93400640 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf039da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.040+0000 7faf92a00640 1 -- 172.21.15.103:0/2463297852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf026540 msgr2=0x5580cf09ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.040+0000 7faf92a00640 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf026540 0x5580cf09ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.040+0000 7faf92a00640 1 -- 172.21.15.103:0/2463297852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 msgr2=0x5580cf039da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.040+0000 7faf92a00640 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf039da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.040+0000 7faf92a00640 1 -- 172.21.15.103:0/2463297852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580cf09bf50 con 0x5580cf01be70 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.040+0000 7faf92a00640 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01be70 0x5580cf03a2e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faf80004590 tx=0x7faf80009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6930a5c04d1e1a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.040+0000 7faf7d600640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.041+0000 7faf7d600640 1 -- 172.21.15.103:0/2463297852 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf80005220 con 0x5580cf01be70 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.041+0000 7faf7d600640 1 -- 172.21.15.103:0/2463297852 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7faf800053c0 con 0x5580cf01be70 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.041+0000 7faf9d39e740 1 -- 172.21.15.103:0/2463297852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf840ccf80 con 0x5580cf01be70 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.041+0000 7faf7d600640 1 -- 172.21.15.103:0/2463297852 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faf80030050 con 0x5580cf01be70 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.041+0000 7faf9d39e740 10 client.15816 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.041+0000 7faf9d39e740 1 -- 172.21.15.103:0/2463297852 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5580cf09d2f0 con 0x5580cf01be70 2024-05-23T00:01:47.045 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf7d600640 1 -- 172.21.15.103:0/2463297852 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faf840ccf80 con 0x5580cf01be70 2024-05-23T00:01:47.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf7d600640 1 -- 172.21.15.103:0/2463297852 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7faf800203b0 con 0x5580cf01be70 2024-05-23T00:01:47.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf9d39e740 10 client.15816 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:47.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf9d39e740 10 client.15816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:47.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf9d39e740 20 client.15816 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:47.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580cf09db00 0x5580cf09ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf9d39e740 4 client.15816 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-23T00:01:47.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf9d39e740 1 -- 172.21.15.103:0/2463297852 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5580cef8cef0 con 0x5580cf09db00 2024-05-23T00:01:47.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.042+0000 7faf93400640 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580cf09db00 0x5580cf09ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.043+0000 7faf93400640 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580cf09db00 0x5580cf09ff20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39434829fb61182d server_cookie=419cc89631f8075 in_seq=0 out_seq=0 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.043+0000 7faf7d600640 10 client.15816 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf7d600640 1 -- 172.21.15.103:0/2463297852 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5580cef8cef0 con 0x5580cf09db00 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf7d600640 10 client.15816 handle_command_reply: tid=0 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 client.15816 shutdown 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 -- 172.21.15.103:0/2463297852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01be70 msgr2=0x5580cf03a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01be70 0x5580cf03a2e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7faf80004590 tx=0x7faf80009f30 comp rx=0 tx=0).stop 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 -- 172.21.15.103:0/2463297852 shutdown_connections 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580cf09db00 0x5580cf09ff20 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580cf026540 0x5580cf09ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580cf01be70 0x5580cf03a2e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 --2- 172.21.15.103:0/2463297852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580cf01b550 0x5580cf039da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf9d39e740 1 -- 172.21.15.103:0/2463297852 >> 172.21.15.103:0/2463297852 conn(0x5580cf017870 msgr2=0x5580cf023550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.044+0000 7faf7d600640 0 client.15816 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:47.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.045+0000 7faf9d39e740 1 -- 172.21.15.103:0/2463297852 shutdown_connections 2024-05-23T00:01:47.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.045+0000 7faf9d39e740 1 -- 172.21.15.103:0/2463297852 wait complete. 2024-05-23T00:01:47.049 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.045+0000 7faf9d39e740 20 client.15816 trim_cache size 0 max 16384 2024-05-23T00:01:47.049 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:01:47.049 INFO:teuthology.orchestra.run.smithi103.stdout: "return_code": 0, 2024-05-23T00:01:47.049 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_tag": "77093ea9-2fe4-41d7-82c5-895936555030", 2024-05-23T00:01:47.050 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": "asynchronous" 2024-05-23T00:01:47.050 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:01:47.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 -- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7faf780a2e20 msgr2=0x7faf780a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7faf780a2e20 0x7faf780a52e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faf8c004af0 tx=0x7faf8c005fd0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.050 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 -- 172.21.15.103:0/3749886687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf94101820 msgr2=0x7faf94108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf94101820 0x7faf94108770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faf8400e9f0 tx=0x7faf8400edf0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 -- 172.21.15.103:0/3749886687 shutdown_connections 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7faf780a2e20 0x7faf780a52e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faf94107830 0x7faf941091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf941021e0 0x7faf94108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 --2- 172.21.15.103:0/3749886687 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf94101820 0x7faf94108770 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.046+0000 7faf9a400640 1 -- 172.21.15.103:0/3749886687 >> 172.21.15.103:0/3749886687 conn(0x7faf940fd2f0 msgr2=0x7faf9410cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.047+0000 7faf9a400640 1 -- 172.21.15.103:0/3749886687 shutdown_connections 2024-05-23T00:01:47.051 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.047+0000 7faf9a400640 1 -- 172.21.15.103:0/3749886687 wait complete. 2024-05-23T00:01:47.072 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-23T00:01:47.072 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-23T00:01:47.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.237+0000 7f93e9400640 1 -- 172.21.15.103:0/2722343086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93e4103170 msgr2=0x7f93e410b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.237+0000 7f93e9400640 1 --2- 172.21.15.103:0/2722343086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93e4103170 0x7f93e410b650 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f93d0009860 tx=0x7f93d002a3d0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.238+0000 7f93e9400640 1 -- 172.21.15.103:0/2722343086 shutdown_connections 2024-05-23T00:01:47.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.238+0000 7f93e9400640 1 --2- 172.21.15.103:0/2722343086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e410bc20 0x7f93e410e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.238+0000 7f93e9400640 1 --2- 172.21.15.103:0/2722343086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93e4103170 0x7f93e410b650 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.238+0000 7f93e9400640 1 --2- 172.21.15.103:0/2722343086 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93e4102850 0x7f93e4102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.238+0000 7f93e9400640 1 -- 172.21.15.103:0/2722343086 >> 172.21.15.103:0/2722343086 conn(0x7f93e4077350 msgr2=0x7f93e40fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.238+0000 7f93e9400640 1 -- 172.21.15.103:0/2722343086 shutdown_connections 2024-05-23T00:01:47.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.238+0000 7f93e9400640 1 -- 172.21.15.103:0/2722343086 wait complete. 2024-05-23T00:01:47.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.239+0000 7f93e9400640 1 Processor -- start 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.239+0000 7f93e9400640 1 -- start start 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.239+0000 7f93e9400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e4102850 0x7f93e4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.239+0000 7f93e9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93e4103170 0x7f93e4198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.239+0000 7f93e9400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93e410bc20 0x7f93e419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.239+0000 7f93e9400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93e4111a00 con 0x7f93e4102850 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.239+0000 7f93e9400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f93e4111880 con 0x7f93e4103170 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.239+0000 7f93e9400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93e4111b80 con 0x7f93e410bc20 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e4102850 0x7f93e4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e8a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93e410bc20 0x7f93e419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e8a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93e410bc20 0x7f93e419c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59390/0 (socket says 172.21.15.103:59390) 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e8a00640 1 -- 172.21.15.103:0/715571183 learned_addr learned my addr 172.21.15.103:0/715571183 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e4102850 0x7f93e4197f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49048/0 (socket says 172.21.15.103:49048) 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e3400640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93e4103170 0x7f93e4198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e3e00640 1 -- 172.21.15.103:0/715571183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93e410bc20 msgr2=0x7f93e419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e3e00640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93e410bc20 0x7f93e419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e3e00640 1 -- 172.21.15.103:0/715571183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93e4103170 msgr2=0x7f93e4198460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e3e00640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93e4103170 0x7f93e4198460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.240+0000 7f93e3e00640 1 -- 172.21.15.103:0/715571183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e419cf10 con 0x7f93e4102850 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.241+0000 7f93e3e00640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e4102850 0x7f93e4197f20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f93d400e9f0 tx=0x7f93d400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.241+0000 7f93e0c00640 1 -- 172.21.15.103:0/715571183 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d400ccc0 con 0x7f93e4102850 2024-05-23T00:01:47.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.241+0000 7f93e9400640 1 -- 172.21.15.103:0/715571183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e419d200 con 0x7f93e4102850 2024-05-23T00:01:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.241+0000 7f93e0c00640 1 -- 172.21.15.103:0/715571183 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93d4010910 con 0x7f93e4102850 2024-05-23T00:01:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.241+0000 7f93e0c00640 1 -- 172.21.15.103:0/715571183 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4018960 con 0x7f93e4102850 2024-05-23T00:01:47.245 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.241+0000 7f93e9400640 1 -- 172.21.15.103:0/715571183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e41003d0 con 0x7f93e4102850 2024-05-23T00:01:47.246 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.245+0000 7f93ed813740 1 Processor -- start 2024-05-23T00:01:47.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.245+0000 7f93ed813740 1 -- start start 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.245+0000 7f93ed813740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.245+0000 7f93ed813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.245+0000 7f93ed813740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef80750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.245+0000 7f93ed813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a14ef2e320 con 0x55a14ef6b480 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.245+0000 7f93ed813740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a14eecfcb0 con 0x55a14ef6ab60 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.245+0000 7f93ed813740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a14ee76fc0 con 0x55a14ef7e360 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.246+0000 7f93e8a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.246+0000 7f93e8a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:50040/0 (socket says 172.21.15.103:50040) 2024-05-23T00:01:47.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.246+0000 7f93e8a00640 1 -- 172.21.15.103:0/449439473 learned_addr learned my addr 172.21.15.103:0/449439473 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.246+0000 7f93e3400640 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef80750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.246+0000 7f93e0c00640 1 -- 172.21.15.103:0/715571183 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7f93d4010ac0 con 0x7f93e4102850 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.246+0000 7f93e3400640 1 -- 172.21.15.103:0/449439473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 msgr2=0x55a14ef6af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.246+0000 7f93e3400640 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.246+0000 7f93e3400640 1 -- 172.21.15.103:0/449439473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 msgr2=0x55a14ef7de20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.247+0000 7f93e3e00640 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.247+0000 7f93e3400640 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.247+0000 7f93e3400640 1 -- 172.21.15.103:0/449439473 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a14ecad7d0 con 0x55a14ef7e360 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.247+0000 7f93e8a00640 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.247+0000 7f93e3400640 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef80750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f93d0005e00 tx=0x7f93d0032f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92a13cd2e4bdeff1 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.247+0000 7f93d9200640 1 -- 172.21.15.103:0/449439473 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d0030cb0 con 0x55a14ef7e360 2024-05-23T00:01:47.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.247+0000 7f93d9200640 1 -- 172.21.15.103:0/449439473 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93d0030e50 con 0x55a14ef7e360 2024-05-23T00:01:47.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.247+0000 7f93d9200640 1 -- 172.21.15.103:0/449439473 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d00048c0 con 0x55a14ef7e360 2024-05-23T00:01:47.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.248+0000 7f93e0c00640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f93c80a2ea0 0x7f93c80a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.248+0000 7f93e0c00640 1 -- 172.21.15.103:0/715571183 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f93d40cfdf0 con 0x7f93e4102850 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.248+0000 7f93ed813740 1 -- 172.21.15.103:0/449439473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 msgr2=0x55a14ef80750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.248+0000 7f93ed813740 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef80750 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f93d0005e00 tx=0x7f93d0032f60 comp rx=0 tx=0).stop 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.248+0000 7f93e8a00640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f93c80a2ea0 0x7f93c80a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.249+0000 7f93ed813740 1 -- 172.21.15.103:0/449439473 shutdown_connections 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.249+0000 7f93ed813740 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef80750 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.249+0000 7f93ed813740 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.249+0000 7f93ed813740 1 --2- 172.21.15.103:0/449439473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.249+0000 7f93ed813740 1 -- 172.21.15.103:0/449439473 >> 172.21.15.103:0/449439473 conn(0x55a14ef66e80 msgr2=0x55a14ef67260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.249+0000 7f93ed813740 1 -- 172.21.15.103:0/449439473 shutdown_connections 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.249+0000 7f93ed813740 1 -- 172.21.15.103:0/449439473 wait complete. 2024-05-23T00:01:47.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.249+0000 7f93ed813740 1 -- start start 2024-05-23T00:01:47.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93ed813740 1 Processor -- start 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93e8a00640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f93c80a2ea0 0x7f93c80a5360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009540 tx=0x7f93dc010040 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93ed813740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef7b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93ed813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93ed813740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef7bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93ed813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a14ee8a410 con 0x55a14ef6b480 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93ed813740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55a14eecbc70 con 0x55a14ef6ab60 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93ed813740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55a14ef810f0 con 0x55a14ef7e360 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.250+0000 7f93ed813740 10 client.0 _ll_register_callbacks cb 0x55a14eec60d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:47.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49062/0 (socket says 172.21.15.103:49062) 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e3400640 1 -- 172.21.15.103:0/2774807205 learned_addr learned my addr 172.21.15.103:0/2774807205 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e3e00640 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef7b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e8a00640 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef7bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e8a00640 1 -- 172.21.15.103:0/2774807205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 msgr2=0x55a14ef7b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e8a00640 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef7b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e8a00640 1 -- 172.21.15.103:0/2774807205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 msgr2=0x55a14ef7b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.251+0000 7f93e8a00640 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.252+0000 7f93e8a00640 1 -- 172.21.15.103:0/2774807205 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a14ebe33f0 con 0x55a14ef7e360 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.252+0000 7f93e3400640 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.252+0000 7f93e8a00640 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef7bce0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f93dc005c40 tx=0x7f93dc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb8ce755515a2ce server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.252+0000 7f93cd600640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.252+0000 7f93cd600640 1 -- 172.21.15.103:0/2774807205 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc004b20 con 0x55a14ef7e360 2024-05-23T00:01:47.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.252+0000 7f93ed813740 1 -- 172.21.15.103:0/2774807205 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93d0030e50 con 0x55a14ef7e360 2024-05-23T00:01:47.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.252+0000 7f93cd600640 1 -- 172.21.15.103:0/2774807205 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f93dc020040 con 0x55a14ef7e360 2024-05-23T00:01:47.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93cd600640 1 -- 172.21.15.103:0/2774807205 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f93dc032050 con 0x55a14ef7e360 2024-05-23T00:01:47.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93ed813740 10 client.25933 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:47.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93ed813740 1 -- 172.21.15.103:0/2774807205 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a14eba30f0 con 0x55a14ef7e360 2024-05-23T00:01:47.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93cd600640 1 -- 172.21.15.103:0/2774807205 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93d0030e50 con 0x55a14ef7e360 2024-05-23T00:01:47.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93cd600640 1 -- 172.21.15.103:0/2774807205 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f93dc024070 con 0x55a14ef7e360 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93ed813740 10 client.25933 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93ed813740 10 client.25933 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93ed813740 20 client.25933 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.253+0000 7f93ed813740 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55a14ef8e5a0 0x55a14ef909d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.254+0000 7f93ed813740 4 client.25933 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.254+0000 7f93ed813740 1 -- 172.21.15.103:0/2774807205 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a14ee8ee00 con 0x55a14ef8e5a0 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.254+0000 7f93e3e00640 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55a14ef8e5a0 0x55a14ef909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.255+0000 7f93cd600640 10 client.25933 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.255+0000 7f93e3e00640 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55a14ef8e5a0 0x55a14ef909d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29d7519546fd8249 server_cookie=18ae7e5fed1c02b8 in_seq=0 out_seq=0 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.255+0000 7f93cd600640 1 -- 172.21.15.103:0/2774807205 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a14ee8ee00 con 0x55a14ef8e5a0 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.255+0000 7f93cd600640 10 client.25933 handle_command_reply: tid=0 2024-05-23T00:01:47.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 client.25933 shutdown 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 -- 172.21.15.103:0/2774807205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 msgr2=0x55a14ef7bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef7bce0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f93dc005c40 tx=0x7f93dc004ab0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 -- 172.21.15.103:0/2774807205 shutdown_connections 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55a14ef8e5a0 0x55a14ef909d0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef7bce0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93cd600640 0 client.25933 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6b480 0x55a14ef7b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 --2- 172.21.15.103:0/2774807205 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef7b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 -- 172.21.15.103:0/2774807205 >> 172.21.15.103:0/2774807205 conn(0x55a14ef66e80 msgr2=0x55a14ef66000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 -- 172.21.15.103:0/2774807205 shutdown_connections 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 1 -- 172.21.15.103:0/2774807205 wait complete. 2024-05-23T00:01:47.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.256+0000 7f93ed813740 20 client.25933 trim_cache size 0 max 16384 2024-05-23T00:01:47.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.292+0000 7f93ed813740 1 Processor -- start 2024-05-23T00:01:47.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.292+0000 7f93ed813740 1 -- start start 2024-05-23T00:01:47.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93ed813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6ab60 0x55a14ef89e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.295 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93ed813740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6b480 0x55a14ef67260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93ed813740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef6d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93ed813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93e41af310 con 0x55a14ef6ab60 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93ed813740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f93e41acbe0 con 0x55a14ef6b480 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93ed813740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93e41aaa30 con 0x55a14ef7e360 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93e3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6ab60 0x55a14ef89e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93e3400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6b480 0x55a14ef67260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93e3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6ab60 0x55a14ef89e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:49064/0 (socket says 172.21.15.103:49064) 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.293+0000 7f93e3e00640 1 -- 172.21.15.103:0/1876658815 learned_addr learned my addr 172.21.15.103:0/1876658815 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.296 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.294+0000 7f93e8a00640 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef6d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.294+0000 7f93e3e00640 1 -- 172.21.15.103:0/1876658815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 msgr2=0x55a14ef6d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.294+0000 7f93e3e00640 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef6d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.294+0000 7f93e3e00640 1 -- 172.21.15.103:0/1876658815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6b480 msgr2=0x55a14ef67260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.294+0000 7f93e3e00640 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6b480 0x55a14ef67260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.294+0000 7f93e3e00640 1 -- 172.21.15.103:0/1876658815 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a14ef70fa0 con 0x55a14ef6ab60 2024-05-23T00:01:47.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.294+0000 7f93e8a00640 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:47.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.294+0000 7f93e3e00640 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6ab60 0x55a14ef89e60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f93d40cd0e0 tx=0x7f93d4009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9116c21c95eb121b server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.295+0000 7f93cfe00640 1 -- 172.21.15.103:0/1876658815 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40a4940 con 0x55a14ef6ab60 2024-05-23T00:01:47.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.295+0000 7f93cfe00640 1 -- 172.21.15.103:0/1876658815 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93d409a080 con 0x55a14ef6ab60 2024-05-23T00:01:47.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.295+0000 7f93cfe00640 1 -- 172.21.15.103:0/1876658815 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d409b430 con 0x55a14ef6ab60 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 -- 172.21.15.103:0/1876658815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6ab60 msgr2=0x55a14ef89e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6ab60 0x55a14ef89e60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f93d40cd0e0 tx=0x7f93d4009700 comp rx=0 tx=0).stop 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 -- 172.21.15.103:0/1876658815 shutdown_connections 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef7e360 0x55a14ef6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6b480 0x55a14ef67260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 --2- 172.21.15.103:0/1876658815 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef6ab60 0x55a14ef89e60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 -- 172.21.15.103:0/1876658815 >> 172.21.15.103:0/1876658815 conn(0x55a14ef66e80 msgr2=0x55a14ef65770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 -- 172.21.15.103:0/1876658815 shutdown_connections 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 -- 172.21.15.103:0/1876658815 wait complete. 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.296+0000 7f93ed813740 1 -- start start 2024-05-23T00:01:47.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.297+0000 7f93ed813740 1 Processor -- start 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.297+0000 7f93ed813740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.297+0000 7f93ed813740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef6b480 0x55a14ef6ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.297+0000 7f93ed813740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef7e360 0x55a14eff3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.297+0000 7f93ed813740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f93e41ab0a0 con 0x55a14ef7e360 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.297+0000 7f93ed813740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f93e41ac510 con 0x55a14ef6ab60 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.297+0000 7f93ed813740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93e41ab770 con 0x55a14ef6b480 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.297+0000 7f93ed813740 10 client.0 _ll_register_callbacks cb 0x7f93dc00d580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:50088/0 (socket says 172.21.15.103:50088) 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 -- 172.21.15.103:0/2773278191 learned_addr learned my addr 172.21.15.103:0/2773278191 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3400640 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef6b480 0x55a14ef6ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.301 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e8a00640 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef7e360 0x55a14eff3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 -- 172.21.15.103:0/2773278191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef6b480 msgr2=0x55a14ef6ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef6b480 0x55a14ef6ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 -- 172.21.15.103:0/2773278191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef7e360 msgr2=0x55a14eff3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:47.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef7e360 0x55a14eff3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 -- 172.21.15.103:0/2773278191 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a14ef70870 con 0x55a14ef6ab60 2024-05-23T00:01:47.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3400640 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef6b480 0x55a14ef6ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:47.302 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.298+0000 7f93e3e00640 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6fa60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f93d40a4410 tx=0x7f93d409bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e79ba87e9a3b2fc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.299+0000 7f93cd600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.299+0000 7f93cd600640 1 -- 172.21.15.103:0/2773278191 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40cd4b0 con 0x55a14ef6ab60 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.299+0000 7f93cd600640 1 -- 172.21.15.103:0/2773278191 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f93d409b1d0 con 0x55a14ef6ab60 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.299+0000 7f93ed813740 1 -- 172.21.15.103:0/2773278191 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93d409a080 con 0x55a14ef6ab60 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.300+0000 7f93cd600640 1 -- 172.21.15.103:0/2773278191 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f93d40a4940 con 0x55a14ef6ab60 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.300+0000 7f93ed813740 10 client.25928 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.300+0000 7f93ed813740 1 -- 172.21.15.103:0/2773278191 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55a14eff51a0 con 0x55a14ef6ab60 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.300+0000 7f93cd600640 1 -- 172.21.15.103:0/2773278191 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93d409a080 con 0x55a14ef6ab60 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.301+0000 7f93cd600640 1 -- 172.21.15.103:0/2773278191 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f93d4018c00 con 0x55a14ef6ab60 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.301+0000 7f93ed813740 10 client.25928 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:47.304 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.301+0000 7f93ed813740 10 client.25928 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.301+0000 7f93ed813740 20 client.25928 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.301+0000 7f93ed813740 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55a14eff57a0 0x55a14eff7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.301+0000 7f93ed813740 4 client.25928 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.301+0000 7f93ed813740 1 -- 172.21.15.103:0/2773278191 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a14eedd3e0 con 0x55a14eff57a0 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.301+0000 7f93e3400640 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55a14eff57a0 0x55a14eff7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93e3400640 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55a14eff57a0 0x55a14eff7bc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4adaede52b064693 server_cookie=d95fe8ade9cdbbc2 in_seq=0 out_seq=0 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93cd600640 10 client.25928 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93cd600640 1 -- 172.21.15.103:0/2773278191 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a14eedd3e0 con 0x55a14eff57a0 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93cd600640 10 client.25928 handle_command_reply: tid=0 2024-05-23T00:01:47.305 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93ed813740 1 client.25928 shutdown 2024-05-23T00:01:47.306 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93ed813740 1 -- 172.21.15.103:0/2773278191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 msgr2=0x55a14ef6fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93ed813740 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6fa60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f93d40a4410 tx=0x7f93d409bec0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93ed813740 1 -- 172.21.15.103:0/2773278191 shutdown_connections 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.302+0000 7f93ed813740 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55a14eff57a0 0x55a14eff7bc0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.303+0000 7f93ed813740 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a14ef7e360 0x55a14eff3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.303+0000 7f93ed813740 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55a14ef6b480 0x55a14ef6ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.303+0000 7f93cd600640 0 client.25928 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.303+0000 7f93ed813740 1 --2- 172.21.15.103:0/2773278191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55a14ef6ab60 0x55a14ef6fa60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.303+0000 7f93ed813740 1 -- 172.21.15.103:0/2773278191 >> 172.21.15.103:0/2773278191 conn(0x55a14ef66e80 msgr2=0x55a14ef66000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.303+0000 7f93ed813740 1 -- 172.21.15.103:0/2773278191 shutdown_connections 2024-05-23T00:01:47.308 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.303+0000 7f93ed813740 1 -- 172.21.15.103:0/2773278191 wait complete. 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.303+0000 7f93ed813740 20 client.25928 trim_cache size 0 max 16384 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stdout: "scrubs": { 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stdout: "77093ea9-2fe4-41d7-82c5-895936555030": { 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stdout: "path": "/", 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stdout: "tag": "77093ea9-2fe4-41d7-82c5-895936555030", 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stdout: "options": "recursive,force" 2024-05-23T00:01:47.309 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:01:47.310 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:01:47.310 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 -- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f93c80a2ea0 msgr2=0x7f93c80a5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f93c80a2ea0 0x7f93c80a5360 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f93dc009540 tx=0x7f93dc010040 comp rx=0 tx=0).stop 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 -- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e4102850 msgr2=0x7f93e4197f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e4102850 0x7f93e4197f20 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f93d400e9f0 tx=0x7f93d400eeb0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 -- 172.21.15.103:0/715571183 shutdown_connections 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f93c80a2ea0 0x7f93c80a5360 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93e410bc20 0x7f93e419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f93e4103170 0x7f93e4198460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 --2- 172.21.15.103:0/715571183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f93e4102850 0x7f93e4197f20 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 -- 172.21.15.103:0/715571183 >> 172.21.15.103:0/715571183 conn(0x7f93e4077350 msgr2=0x7f93e40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:47.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 -- 172.21.15.103:0/715571183 shutdown_connections 2024-05-23T00:01:47.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:01:47.304+0000 7f93e9400640 1 -- 172.21.15.103:0/715571183 wait complete. 2024-05-23T00:01:47.323 INFO:tasks.cephfs.filesystem:scrub status for tag:77093ea9-2fe4-41d7-82c5-895936555030 - {'path': '/', 'tag': '77093ea9-2fe4-41d7-82c5-895936555030', 'options': 'recursive,force'} 2024-05-23T00:01:47.367 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:01:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: 2024-05-23T00:01:47.044+0000 7f785f66a700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000001fb [...3,head] /volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4/ auth v22 ap=1 f(v0 m2024-05-23T00:00:44.374428+0000 1=0+1) n(v0 rc2024-05-23T00:00:44.379428+0000 2=0+2) old_inodes=1 (inest lock) (ifile excl) caps={25399=pAsLsXsFsx/p@1},l=25399 | lock=0 importingcaps=0 dirfrag=1 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x559b32768000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x100000001fb:[<0x100000001fa/eed760b1-8739-432b-b036-087fb73667d4 v18>,<0x10000000001/sv_1 v41>,<0x10000000000/qa v28>,<0x1/volumes v26>]//[]","memoryvalue":"(2)0x100000001fb:[<0x100000001fa/eed760b1-8739-432b-b036-087fb73667d4 v22>,<0x10000000001/sv_1 v46>,<0x10000000000/qa v30>,<0x1/volumes v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-05-23T00:00:44.374428+0000 1=0+1)","ondisk_value.rstat":"n(v0 rc2024-05-23T00:00:44.379428+0000 2=0+2)","memory_value.dirstat":"f(v0 m2024-05-23T00:00:44.374428+0000 1=0+1)","memory_value.rstat":"n(v0 rc2024-05-23T00:00:44.379428+0000 2=0+2)","error_str":""},"return_code":0} 2024-05-23T00:01:47.367 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:01:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: 2024-05-23T00:01:47.045+0000 7f785f66a700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x100000001fc [2,head] /volumes/qa/sv_1/.meta auth v14 ap=1 s=117 n(v0 rc2024-05-22T23:58:07.280064+0000 b117 1=1+0) | replicated=0 authpin=1 scrubqueue=0 0x559b3251ab00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x100000001fc:[<0x100000001fa/.meta v14>,<0x10000000001/sv_1 v41>,<0x10000000000/qa v28>,<0x1/volumes v26>]//[3]","memoryvalue":"(4)0x100000001fc:[<0x100000001fa/.meta v14>,<0x10000000001/sv_1 v46>,<0x10000000000/qa v30>,<0x1/volumes v22>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-23T00:01:47.367 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:01:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: 2024-05-23T00:01:47.045+0000 7f785f66a700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000005 [...3,head] /volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4/client.0/ auth v6 ap=1 f() n(v0 rc2024-05-23T00:00:44.379428+0000 1=0+1) old_inodes=1 (inest lock) (ifile excl) caps={25399=pAsLsXsFsx/AsLsXsFsx@2},l=25399 | ptrwaiter=0 lock=0 importingcaps=0 dirfrag=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x559b3251a580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(2)0x10000000005:[<0x100000001fb/client.0 v2>,<0x100000001fa/eed760b1-8739-432b-b036-087fb73667d4 v18>,<0x10000000001/sv_1 v41>,<0x10000000000/qa v28>,<0x1/volumes v26>]//[]","memoryvalue":"(2)0x10000000005:[<0x100000001fb/client.0 v6>,<0x100000001fa/eed760b1-8739-432b-b036-087fb73667d4 v22>,<0x10000000001/sv_1 v46>,<0x10000000000/qa v30>,<0x1/volumes v22>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 rc2024-05-23T00:00:44.379428+0000 1=0+1)","error_str":""},"return_code":0} 2024-05-23T00:01:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-23T00:01:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: scrub queued for path: / 2024-05-23T00:01:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: scrub summary: active paths [/] 2024-05-23T00:01:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: scrub summary: active paths [/] 2024-05-23T00:01:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: Scrub error on inode 0x100000001fb (/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: Scrub error on inode 0x100000001fc (/volumes/qa/sv_1/.meta) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: Scrub error on inode 0x10000000005 (/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4/client.0) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:48 smithi105 ceph-mon[36426]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:48.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-23T00:01:48.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: scrub queued for path: / 2024-05-23T00:01:48.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:48.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: scrub summary: active paths [/] 2024-05-23T00:01:48.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: scrub summary: active paths [/] 2024-05-23T00:01:48.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: Scrub error on inode 0x100000001fb (/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: Scrub error on inode 0x100000001fc (/volumes/qa/sv_1/.meta) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: Scrub error on inode 0x10000000005 (/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4/client.0) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:48.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:48 smithi103 ceph-mon[31370]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-05-23T00:01:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: scrub queued for path: / 2024-05-23T00:01:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: scrub summary: active paths [/] 2024-05-23T00:01:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: scrub summary: active paths [/] 2024-05-23T00:01:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: Scrub error on inode 0x100000001fb (/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: Scrub error on inode 0x100000001fc (/volumes/qa/sv_1/.meta) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: Scrub error on inode 0x10000000005 (/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4/client.0) see mds.i log and `damage ls` output for details 2024-05-23T00:01:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:48.618 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:48 smithi174 ceph-mon[36179]: scrub summary: idle+waiting paths [/] 2024-05-23T00:01:50.262 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.259+0000 7efefb200640 1 -- 172.21.15.103:0/3999415703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 msgr2=0x7efefc110770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.262 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.259+0000 7efefb200640 1 --2- 172.21.15.103:0/3999415703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 0x7efefc110770 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efef4009be0 tx=0x7efef402a6c0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.262 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.260+0000 7efefb200640 1 -- 172.21.15.103:0/3999415703 shutdown_connections 2024-05-23T00:01:50.262 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.260+0000 7efefb200640 1 --2- 172.21.15.103:0/3999415703 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 0x7efefc110770 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.260+0000 7efefb200640 1 --2- 172.21.15.103:0/3999415703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efefc1038a0 0x7efefc103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.260+0000 7efefb200640 1 --2- 172.21.15.103:0/3999415703 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc1098b0 0x7efefc109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.260+0000 7efefb200640 1 -- 172.21.15.103:0/3999415703 >> 172.21.15.103:0/3999415703 conn(0x7efefc0ff350 msgr2=0x7efefc101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.260+0000 7efefb200640 1 -- 172.21.15.103:0/3999415703 shutdown_connections 2024-05-23T00:01:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.260+0000 7efefb200640 1 -- 172.21.15.103:0/3999415703 wait complete. 2024-05-23T00:01:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.261+0000 7efefb200640 1 Processor -- start 2024-05-23T00:01:50.263 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.261+0000 7efefb200640 1 -- start start 2024-05-23T00:01:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efefb200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efefc1038a0 0x7efefc19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efefb200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 0x7efefc19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efefb200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc1098b0 0x7efefc19e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efefb200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efefc113d10 con 0x7efefc1098b0 2024-05-23T00:01:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efefb200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efefc113b90 con 0x7efefc104260 2024-05-23T00:01:50.264 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efefb200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efefc113e90 con 0x7efefc1038a0 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efefa800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc1098b0 0x7efefc19e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efef9e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efefc1038a0 0x7efefc19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.262+0000 7efef9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 0x7efefc19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efefc1038a0 0x7efefc19a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59438/0 (socket says 172.21.15.103:59438) 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9e00640 1 -- 172.21.15.103:0/1123371225 learned_addr learned my addr 172.21.15.103:0/1123371225 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 0x7efefc19a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:50110/0 (socket says 172.21.15.103:50110) 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9400640 1 -- 172.21.15.103:0/1123371225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efefc1038a0 msgr2=0x7efefc19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9400640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efefc1038a0 0x7efefc19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9400640 1 -- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc1098b0 msgr2=0x7efefc19e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.265 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9400640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc1098b0 0x7efefc19e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.266 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9400640 1 -- 172.21.15.103:0/1123371225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efefc19f0e0 con 0x7efefc104260 2024-05-23T00:01:50.266 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.263+0000 7efef9400640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 0x7efefc19a630 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efee800e9f0 tx=0x7efee800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:50.266 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.264+0000 7efef2a00640 1 -- 172.21.15.103:0/1123371225 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee800ccc0 con 0x7efefc104260 2024-05-23T00:01:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.264+0000 7efef2a00640 1 -- 172.21.15.103:0/1123371225 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efee8005020 con 0x7efefc104260 2024-05-23T00:01:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.264+0000 7efefb200640 1 -- 172.21.15.103:0/1123371225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efefc19f3d0 con 0x7efefc104260 2024-05-23T00:01:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.264+0000 7efef2a00640 1 -- 172.21.15.103:0/1123371225 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee80049e0 con 0x7efefc104260 2024-05-23T00:01:50.267 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.264+0000 7efefb200640 1 -- 172.21.15.103:0/1123371225 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efefc1a6c60 con 0x7efefc104260 2024-05-23T00:01:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.268+0000 7eff03844740 1 Processor -- start 2024-05-23T00:01:50.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.268+0000 7eff03844740 1 -- start start 2024-05-23T00:01:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7eff03844740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134a8d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7eff03844740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a979e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7eff03844740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7eff03844740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5591349f12e0 con 0x559134a8d850 2024-05-23T00:01:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7eff03844740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x559134998e50 con 0x559134a97f20 2024-05-23T00:01:50.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7eff03844740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5591349ed2a0 con 0x559134a8cf30 2024-05-23T00:01:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7efef2a00640 1 -- 172.21.15.103:0/1123371225 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7efee8005560 con 0x7efefc104260 2024-05-23T00:01:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7efef9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.269+0000 7efef9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.270+0000 7efef9400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:50114/0 (socket says 172.21.15.103:50114) 2024-05-23T00:01:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.270+0000 7efef9400640 1 -- 172.21.15.103:0/1051142650 learned_addr learned my addr 172.21.15.103:0/1051142650 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:50.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.270+0000 7efefa800640 1 --2- 172.21.15.103:0/1051142650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134a8d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.270+0000 7efef9400640 1 -- 172.21.15.103:0/1051142650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 msgr2=0x559134a8d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.270+0000 7efef9400640 1 --2- 172.21.15.103:0/1051142650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134a8d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.270+0000 7efef9400640 1 -- 172.21.15.103:0/1051142650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 msgr2=0x559134a979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.270+0000 7efef9400640 1 --2- 172.21.15.103:0/1051142650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.270+0000 7efef9400640 1 -- 172.21.15.103:0/1051142650 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559134705210 con 0x559134a97f20 2024-05-23T00:01:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.271+0000 7efef9400640 1 --2- 172.21.15.103:0/1051142650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9a310 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efee80ccb30 tx=0x7efee80d5cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae412b05b7f0bf0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.271+0000 7efef2a00640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7efee00a2f60 0x7efee00a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.272+0000 7efee7000640 1 -- 172.21.15.103:0/1051142650 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee80de070 con 0x559134a97f20 2024-05-23T00:01:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.272+0000 7efefa800640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7efee00a2f60 0x7efee00a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.272+0000 7efee7000640 1 -- 172.21.15.103:0/1051142650 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efee80051d0 con 0x559134a97f20 2024-05-23T00:01:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.272+0000 7efee7000640 1 -- 172.21.15.103:0/1051142650 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee809b660 con 0x559134a97f20 2024-05-23T00:01:50.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.272+0000 7efef2a00640 1 -- 172.21.15.103:0/1123371225 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7efee80cfa10 con 0x7efefc104260 2024-05-23T00:01:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 -- 172.21.15.103:0/1051142650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 msgr2=0x559134a9a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 --2- 172.21.15.103:0/1051142650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9a310 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efee80ccb30 tx=0x7efee80d5cd0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7efefa800640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7efee00a2f60 0x7efee00a5420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efef4006010 tx=0x7efef4009860 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 -- 172.21.15.103:0/1051142650 shutdown_connections 2024-05-23T00:01:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 --2- 172.21.15.103:0/1051142650 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9a310 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 --2- 172.21.15.103:0/1051142650 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 --2- 172.21.15.103:0/1051142650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134a8d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 -- 172.21.15.103:0/1051142650 >> 172.21.15.103:0/1051142650 conn(0x559134a89250 msgr2=0x559134a89630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 -- 172.21.15.103:0/1051142650 shutdown_connections 2024-05-23T00:01:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.273+0000 7eff03844740 1 -- 172.21.15.103:0/1051142650 wait complete. 2024-05-23T00:01:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.274+0000 7eff03844740 1 -- start start 2024-05-23T00:01:50.276 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.274+0000 7eff03844740 1 Processor -- start 2024-05-23T00:01:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7eff03844740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134a94e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7eff03844740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a95340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7eff03844740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a95880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7eff03844740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559134a9adb0 con 0x559134a8d850 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7eff03844740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5591349ac210 con 0x559134a97f20 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7eff03844740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559134a9af30 con 0x559134a8cf30 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7eff03844740 10 client.0 _ll_register_callbacks cb 0x55913498f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7efefa800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a95880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7efef9e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134a94e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7efefa800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a95880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:50124/0 (socket says 172.21.15.103:50124) 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7efefa800640 1 -- 172.21.15.103:0/2989536061 learned_addr learned my addr 172.21.15.103:0/2989536061 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:50.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.275+0000 7efef9400640 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a95340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.276+0000 7efefa800640 1 -- 172.21.15.103:0/2989536061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 msgr2=0x559134a94e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.276+0000 7efefa800640 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134a94e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.276+0000 7efefa800640 1 -- 172.21.15.103:0/2989536061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 msgr2=0x559134a95340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:50.279 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.276+0000 7efefa800640 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a95340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.276+0000 7efefa800640 1 -- 172.21.15.103:0/2989536061 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591347d1620 con 0x559134a97f20 2024-05-23T00:01:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.276+0000 7efef9400640 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a95340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.277+0000 7efefa800640 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a95880 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efef4033000 tx=0x7efef402a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56a964c73b8a97a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.277+0000 7efedb400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.277+0000 7efedb400640 1 -- 172.21.15.103:0/2989536061 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4050020 con 0x559134a97f20 2024-05-23T00:01:50.280 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.277+0000 7eff03844740 1 -- 172.21.15.103:0/2989536061 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efee80051d0 con 0x559134a97f20 2024-05-23T00:01:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.281+0000 7efedb400640 1 -- 172.21.15.103:0/2989536061 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7efef4003df0 con 0x559134a97f20 2024-05-23T00:01:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.281+0000 7efedb400640 1 -- 172.21.15.103:0/2989536061 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efef4036040 con 0x559134a97f20 2024-05-23T00:01:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.281+0000 7eff03844740 10 client.25940 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:50.283 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.281+0000 7eff03844740 1 -- 172.21.15.103:0/2989536061 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559134aa7040 con 0x559134a97f20 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.281+0000 7efedb400640 1 -- 172.21.15.103:0/2989536061 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efee80051d0 con 0x559134a97f20 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.281+0000 7efedb400640 1 -- 172.21.15.103:0/2989536061 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7efef405d020 con 0x559134a97f20 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.282+0000 7eff03844740 10 client.25940 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.282+0000 7eff03844740 10 client.25940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.282+0000 7eff03844740 20 client.25940 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.282+0000 7eff03844740 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x559134aa8570 0x559134aaa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.282+0000 7eff03844740 4 client.25940 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.282+0000 7eff03844740 1 -- 172.21.15.103:0/2989536061 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559134a04ec0 con 0x559134aa8570 2024-05-23T00:01:50.284 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.282+0000 7efef9e00640 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x559134aa8570 0x559134aaa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.283+0000 7efef9e00640 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x559134aa8570 0x559134aaa9a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afa3ebcd270ed19c server_cookie=ced95ee406677e92 in_seq=0 out_seq=0 2024-05-23T00:01:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.283+0000 7efedb400640 10 client.25940 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.283+0000 7efedb400640 1 -- 172.21.15.103:0/2989536061 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559134a04ec0 con 0x559134aa8570 2024-05-23T00:01:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.283+0000 7efedb400640 10 client.25940 handle_command_reply: tid=0 2024-05-23T00:01:50.285 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.283+0000 7eff03844740 1 client.25940 shutdown 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 -- 172.21.15.103:0/2989536061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 msgr2=0x559134a95880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a95880 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7efef4033000 tx=0x7efef402a6c0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 -- 172.21.15.103:0/2989536061 shutdown_connections 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x559134aa8570 0x559134aaa9a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a95880 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a95340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 --2- 172.21.15.103:0/2989536061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134a94e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 -- 172.21.15.103:0/2989536061 >> 172.21.15.103:0/2989536061 conn(0x559134a89250 msgr2=0x559134a883b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7efedb400640 0 client.25940 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 -- 172.21.15.103:0/2989536061 shutdown_connections 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 1 -- 172.21.15.103:0/2989536061 wait complete. 2024-05-23T00:01:50.287 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.284+0000 7eff03844740 20 client.25940 trim_cache size 0 max 16384 2024-05-23T00:01:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.314+0000 7eff03844740 1 Processor -- start 2024-05-23T00:01:50.316 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.314+0000 7eff03844740 1 -- start start 2024-05-23T00:01:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.314+0000 7eff03844740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a8cf30 0x559134aa3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.314+0000 7eff03844740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134ab0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.314+0000 7eff03844740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a97f20 0x559134ab0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.314+0000 7eff03844740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1b3780 con 0x559134a8d850 2024-05-23T00:01:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.314+0000 7eff03844740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1b1050 con 0x559134a8cf30 2024-05-23T00:01:50.317 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.314+0000 7eff03844740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1aeef0 con 0x559134a97f20 2024-05-23T00:01:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efefa800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a97f20 0x559134ab0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efef9e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a8cf30 0x559134aa3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efefa800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a97f20 0x559134ab0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59458/0 (socket says 172.21.15.103:59458) 2024-05-23T00:01:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efefa800640 1 -- 172.21.15.103:0/2786346580 learned_addr learned my addr 172.21.15.103:0/2786346580 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:50.318 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efef9400640 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134ab0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efefa800640 1 -- 172.21.15.103:0/2786346580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a8cf30 msgr2=0x559134aa3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efefa800640 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a8cf30 0x559134aa3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efefa800640 1 -- 172.21.15.103:0/2786346580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 msgr2=0x559134ab0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efefa800640 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134ab0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efefa800640 1 -- 172.21.15.103:0/2786346580 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559134a8d680 con 0x559134a97f20 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efef9400640 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134ab0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.315+0000 7efef9e00640 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a8cf30 0x559134aa3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.316+0000 7efefa800640 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a97f20 0x559134ab0840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efef4005f00 tx=0x7efef4031d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ddd06855c3bec07 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.316+0000 7efee6600640 1 -- 172.21.15.103:0/2786346580 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4050030 con 0x559134a97f20 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.316+0000 7efee6600640 1 -- 172.21.15.103:0/2786346580 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef4045030 con 0x559134a97f20 2024-05-23T00:01:50.319 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.316+0000 7efee6600640 1 -- 172.21.15.103:0/2786346580 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef404c6a0 con 0x559134a97f20 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 -- 172.21.15.103:0/2786346580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a97f20 msgr2=0x559134ab0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a97f20 0x559134ab0840 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efef4005f00 tx=0x7efef4031d70 comp rx=0 tx=0).stop 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 -- 172.21.15.103:0/2786346580 shutdown_connections 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a97f20 0x559134ab0840 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134ab0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 --2- 172.21.15.103:0/2786346580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a8cf30 0x559134aa3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 -- 172.21.15.103:0/2786346580 >> 172.21.15.103:0/2786346580 conn(0x559134a89250 msgr2=0x559134a90060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 -- 172.21.15.103:0/2786346580 shutdown_connections 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.317+0000 7eff03844740 1 -- 172.21.15.103:0/2786346580 wait complete. 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.318+0000 7eff03844740 1 -- start start 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.318+0000 7eff03844740 1 Processor -- start 2024-05-23T00:01:50.321 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.318+0000 7eff03844740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134aa0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7eff03844740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a9b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7eff03844740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7eff03844740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1af510 con 0x559134a8d850 2024-05-23T00:01:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7eff03844740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1b0980 con 0x559134a97f20 2024-05-23T00:01:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7eff03844740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efefc1afbe0 con 0x559134a8cf30 2024-05-23T00:01:50.322 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7eff03844740 10 client.0 _ll_register_callbacks cb 0x7efef4009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7efef9e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134aa0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7efef9e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134aa0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59464/0 (socket says 172.21.15.103:59464) 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7efef9e00640 1 -- 172.21.15.103:0/3234623433 learned_addr learned my addr 172.21.15.103:0/3234623433 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7efefa800640 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.319+0000 7efef9400640 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a9b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efefa800640 1 -- 172.21.15.103:0/3234623433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 msgr2=0x559134aa0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efefa800640 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134aa0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efefa800640 1 -- 172.21.15.103:0/3234623433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 msgr2=0x559134a9b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efefa800640 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a9b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.324 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efefa800640 1 -- 172.21.15.103:0/3234623433 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559134a9c3a0 con 0x559134a97f20 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efef9400640 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a9b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efefa800640 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9bd00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x559134aaacb0 tx=0x7efef4009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab7b25cd15f39158 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efedb400640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efedb400640 1 -- 172.21.15.103:0/3234623433 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef4050040 con 0x559134a97f20 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7eff03844740 1 -- 172.21.15.103:0/3234623433 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efef4045030 con 0x559134a97f20 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.320+0000 7efedb400640 1 -- 172.21.15.103:0/3234623433 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7efef403f890 con 0x559134a97f20 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7efedb400640 1 -- 172.21.15.103:0/3234623433 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7efef4077050 con 0x559134a97f20 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7eff03844740 10 client.25946 fetch_fsmap learned FSMap version 27 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7eff03844740 1 -- 172.21.15.103:0/3234623433 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559134b17240 con 0x559134a97f20 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7efedb400640 1 -- 172.21.15.103:0/3234623433 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efef4045030 con 0x559134a97f20 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7efedb400640 1 -- 172.21.15.103:0/3234623433 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7efef405d040 con 0x559134a97f20 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7eff03844740 10 client.25946 fetch_fsmap finished waiting for FSMap version 27 2024-05-23T00:01:50.325 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7eff03844740 10 client.25946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:01:50.326 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7eff03844740 20 client.25946 populate_metadata read hostname 'smithi103' 2024-05-23T00:01:50.326 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7eff03844740 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x559134b176f0 0x559134b19b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:01:50.326 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7eff03844740 4 client.25946 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:01:50.326 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.321+0000 7eff03844740 1 -- 172.21.15.103:0/3234623433 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5591349fea10 con 0x559134b176f0 2024-05-23T00:01:50.326 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.322+0000 7efef9e00640 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x559134b176f0 0x559134b19b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:01:50.326 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.323+0000 7efedb400640 10 client.25946 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:50.326 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.323+0000 7efef9e00640 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x559134b176f0 0x559134b19b10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4e4eb292e85a15d server_cookie=aa07937c86ac1316 in_seq=0 out_seq=0 2024-05-23T00:01:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7efedb400640 1 -- 172.21.15.103:0/3234623433 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5591349fea10 con 0x559134b176f0 2024-05-23T00:01:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7efedb400640 10 client.25946 handle_command_reply: tid=0 2024-05-23T00:01:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 client.25946 shutdown 2024-05-23T00:01:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 -- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 msgr2=0x559134a9bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9bd00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x559134aaacb0 tx=0x7efef4009b90 comp rx=0 tx=0).stop 2024-05-23T00:01:50.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 -- 172.21.15.103:0/3234623433 shutdown_connections 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x559134b176f0 0x559134b19b10 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x559134a97f20 0x559134a9bd00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559134a8d850 0x559134a9b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 --2- 172.21.15.103:0/3234623433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559134a8cf30 0x559134aa0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7eff03844740 1 -- 172.21.15.103:0/3234623433 >> 172.21.15.103:0/3234623433 conn(0x559134a89250 msgr2=0x559134a8e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.330+0000 7efedb400640 0 client.25946 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.331+0000 7eff03844740 1 -- 172.21.15.103:0/3234623433 shutdown_connections 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.331+0000 7eff03844740 1 -- 172.21.15.103:0/3234623433 wait complete. 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.331+0000 7eff03844740 20 client.25946 trim_cache size 0 max 16384 2024-05-23T00:01:50.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:01:50.335 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:01:50.335 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:01:50.335 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:01:50.335 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 -- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7efee00a2f60 msgr2=0x7efee00a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7efee00a2f60 0x7efee00a5420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efef4006010 tx=0x7efef4009860 comp rx=0 tx=0).stop 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 -- 172.21.15.103:0/1123371225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 msgr2=0x7efefc19a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 0x7efefc19a630 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efee800e9f0 tx=0x7efee800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 -- 172.21.15.103:0/1123371225 shutdown_connections 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7efee00a2f60 0x7efee00a5420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efefc1098b0 0x7efefc19e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efefc104260 0x7efefc19a630 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 --2- 172.21.15.103:0/1123371225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efefc1038a0 0x7efefc19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 -- 172.21.15.103:0/1123371225 >> 172.21.15.103:0/1123371225 conn(0x7efefc0ff350 msgr2=0x7efefc0ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 -- 172.21.15.103:0/1123371225 shutdown_connections 2024-05-23T00:01:50.336 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:01:50.332+0000 7efefb200640 1 -- 172.21.15.103:0/1123371225 wait complete. 2024-05-23T00:01:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:50 smithi105 ceph-mon[36426]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-23T00:01:50.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:50 smithi103 ceph-mon[31370]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-23T00:01:50.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:50 smithi174 ceph-mon[36179]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-05-23T00:01:52.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:52 smithi105 ceph-mon[36426]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 6 op/s 2024-05-23T00:01:52.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:52 smithi103 ceph-mon[31370]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 6 op/s 2024-05-23T00:01:52.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:52 smithi174 ceph-mon[36179]: pgmap v44: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 6 op/s 2024-05-23T00:01:53.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[31366]: 2024-05-23T00:01:53.281+0000 7fe445c00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-23T00:01:53.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:53 smithi103 ceph-mon[31370]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-23T00:01:53.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:53 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:active 2024-05-23T00:01:53.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:53 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:53.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:53 smithi174 ceph-mon[36179]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-23T00:01:53.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:53 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:active 2024-05-23T00:01:53.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:53 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:53.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:53 smithi105 ceph-mon[36426]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-23T00:01:53.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:53 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] up:active 2024-05-23T00:01:53.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:53 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:01:54.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:54 smithi174 ceph-mon[36179]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 6 op/s 2024-05-23T00:01:54.709 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:54 smithi105 ceph-mon[36426]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 6 op/s 2024-05-23T00:01:54.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:54 smithi103 ceph-mon[31370]: pgmap v45: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 6 op/s 2024-05-23T00:01:56.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:56 smithi174 ceph-mon[36179]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-23T00:01:56.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:56 smithi105 ceph-mon[36426]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-23T00:01:56.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:56 smithi103 ceph-mon[31370]: pgmap v46: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-23T00:01:57.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:57 smithi174 ceph-mon[36179]: scrub summary: idle 2024-05-23T00:01:57.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:57 smithi174 ceph-mon[36179]: scrub summary: idle 2024-05-23T00:01:57.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:57 smithi105 ceph-mon[36426]: scrub summary: idle 2024-05-23T00:01:57.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:57 smithi105 ceph-mon[36426]: scrub summary: idle 2024-05-23T00:01:57.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:57 smithi103 ceph-mon[31370]: scrub summary: idle 2024-05-23T00:01:57.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:57 smithi103 ceph-mon[31370]: scrub summary: idle 2024-05-23T00:01:58.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:01:58 smithi174 ceph-mon[36179]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-23T00:01:58.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:01:58 smithi105 ceph-mon[36426]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-23T00:01:58.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:01:58 smithi103 ceph-mon[31370]: pgmap v47: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 682 B/s wr, 7 op/s 2024-05-23T00:02:00.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:00 smithi174 ceph-mon[36179]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-23T00:02:00.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:00 smithi105 ceph-mon[36426]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-23T00:02:00.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:00 smithi103 ceph-mon[31370]: pgmap v48: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-05-23T00:02:02.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:02 smithi174 ceph-mon[36179]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-23T00:02:02.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:02 smithi105 ceph-mon[36426]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-23T00:02:02.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:02 smithi103 ceph-mon[31370]: pgmap v49: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-23T00:02:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 -- 172.21.15.103:0/3034405305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1038a0 msgr2=0x7fc47c103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 --2- 172.21.15.103:0/3034405305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1038a0 0x7fc47c103d20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc468009930 tx=0x7fc46802a440 comp rx=0 tx=0).stop 2024-05-23T00:02:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 -- 172.21.15.103:0/3034405305 shutdown_connections 2024-05-23T00:02:03.527 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 --2- 172.21.15.103:0/3034405305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc47c104260 0x7fc47c110770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 --2- 172.21.15.103:0/3034405305 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1038a0 0x7fc47c103d20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 --2- 172.21.15.103:0/3034405305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc47c1098b0 0x7fc47c109c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 -- 172.21.15.103:0/3034405305 >> 172.21.15.103:0/3034405305 conn(0x7fc47c0ff350 msgr2=0x7fc47c101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 -- 172.21.15.103:0/3034405305 shutdown_connections 2024-05-23T00:02:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.524+0000 7fc482600640 1 -- 172.21.15.103:0/3034405305 wait complete. 2024-05-23T00:02:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.525+0000 7fc482600640 1 Processor -- start 2024-05-23T00:02:03.528 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.526+0000 7fc482600640 1 -- start start 2024-05-23T00:02:03.529 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.526+0000 7fc482600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc47c1038a0 0x7fc47c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.529 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.526+0000 7fc482600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc47c104260 0x7fc47c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.529 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.526+0000 7fc482600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1098b0 0x7fc47c19eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.529 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.526+0000 7fc482600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c113cc0 con 0x7fc47c104260 2024-05-23T00:02:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.526+0000 7fc482600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c113b40 con 0x7fc47c1038a0 2024-05-23T00:02:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.526+0000 7fc482600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c113e40 con 0x7fc47c1098b0 2024-05-23T00:02:03.530 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc47c1038a0 0x7fc47c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc47c1038a0 0x7fc47c19a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:41682/0 (socket says 172.21.15.103:41682) 2024-05-23T00:02:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481200640 1 -- 172.21.15.103:0/1285891411 learned_addr learned my addr 172.21.15.103:0/1285891411 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc47be00640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc47c104260 0x7fc47c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481c00640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1098b0 0x7fc47c19eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.531 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481c00640 1 -- 172.21.15.103:0/1285891411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc47c1038a0 msgr2=0x7fc47c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481c00640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc47c1038a0 0x7fc47c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481c00640 1 -- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc47c104260 msgr2=0x7fc47c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481c00640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc47c104260 0x7fc47c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481c00640 1 -- 172.21.15.103:0/1285891411 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47c19f1e0 con 0x7fc47c1098b0 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.527+0000 7fc481200640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc47c1038a0 0x7fc47c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.528+0000 7fc481c00640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1098b0 0x7fc47c19eab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc474009fd0 tx=0x7fc47400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.528+0000 7fc479600640 1 -- 172.21.15.103:0/1285891411 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47400ce20 con 0x7fc47c1098b0 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.528+0000 7fc479600640 1 -- 172.21.15.103:0/1285891411 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc474005020 con 0x7fc47c1098b0 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.528+0000 7fc479600640 1 -- 172.21.15.103:0/1285891411 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc474010720 con 0x7fc47c1098b0 2024-05-23T00:02:03.532 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.528+0000 7fc482600640 1 -- 172.21.15.103:0/1285891411 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc47c19f4d0 con 0x7fc47c1098b0 2024-05-23T00:02:03.533 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.528+0000 7fc482600640 1 -- 172.21.15.103:0/1285891411 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc47c1a6d60 con 0x7fc47c1098b0 2024-05-23T00:02:03.535 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.532+0000 7fc486a16740 1 Processor -- start 2024-05-23T00:02:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.533+0000 7fc479600640 1 -- 172.21.15.103:0/1285891411 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 26) ==== 138067+0+0 (secure 0 0 0) 0x7fc474005540 con 0x7fc47c1098b0 2024-05-23T00:02:03.536 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.533+0000 7fc486a16740 1 -- start start 2024-05-23T00:02:03.537 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc486a16740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d91310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.537 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc486a16740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9d9b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.537 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc486a16740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9d9e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.537 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc486a16740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5580b9cf52e0 con 0x5580b9d91850 2024-05-23T00:02:03.537 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc486a16740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5580b9c9ce50 con 0x5580b9d9bf20 2024-05-23T00:02:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc486a16740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580b9cf12a0 con 0x5580b9d90f30 2024-05-23T00:02:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc47be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d91310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc47be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d91310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:45040/0 (socket says 172.21.15.103:45040) 2024-05-23T00:02:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.534+0000 7fc47be00640 1 -- 172.21.15.103:0/1997805563 learned_addr learned my addr 172.21.15.103:0/1997805563 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:03.538 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc47be00640 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9d9e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc481200640 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9d9b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc47be00640 1 -- 172.21.15.103:0/1997805563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 msgr2=0x5580b9d9e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc47be00640 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9d9e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc47be00640 1 -- 172.21.15.103:0/1997805563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 msgr2=0x5580b9d9b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc47be00640 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9d9b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc47be00640 1 -- 172.21.15.103:0/1997805563 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b9a09210 con 0x5580b9d90f30 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc481200640 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9d9b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc47be00640 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d91310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc468000c00 tx=0x7fc46802a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=def9ae50b8634b9f server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:03.539 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.535+0000 7fc479600640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4640a2e90 0x7fc4640a5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.536+0000 7fc479600640 1 -- 172.21.15.103:0/1285891411 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc4740d0110 con 0x7fc47c1098b0 2024-05-23T00:02:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.536+0000 7fc46e000640 1 -- 172.21.15.103:0/1997805563 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468030dd0 con 0x5580b9d90f30 2024-05-23T00:02:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.536+0000 7fc481200640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4640a2e90 0x7fc4640a5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.536+0000 7fc46e000640 1 -- 172.21.15.103:0/1997805563 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4680303a0 con 0x5580b9d90f30 2024-05-23T00:02:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.536+0000 7fc46e000640 1 -- 172.21.15.103:0/1997805563 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4680038e0 con 0x5580b9d90f30 2024-05-23T00:02:03.540 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.537+0000 7fc486a16740 1 -- 172.21.15.103:0/1997805563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 msgr2=0x5580b9d91310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.537+0000 7fc486a16740 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d91310 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc468000c00 tx=0x7fc46802a440 comp rx=0 tx=0).stop 2024-05-23T00:02:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.537+0000 7fc486a16740 1 -- 172.21.15.103:0/1997805563 shutdown_connections 2024-05-23T00:02:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.537+0000 7fc486a16740 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9d9e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.537+0000 7fc486a16740 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9d9b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.537+0000 7fc486a16740 1 --2- 172.21.15.103:0/1997805563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d91310 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.541 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.537+0000 7fc486a16740 1 -- 172.21.15.103:0/1997805563 >> 172.21.15.103:0/1997805563 conn(0x5580b9d8d250 msgr2=0x5580b9d8d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.538+0000 7fc481200640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4640a2e90 0x7fc4640a5350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x5580b9d924f0 tx=0x7fc470005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:03.543 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.538+0000 7fc486a16740 1 -- 172.21.15.103:0/1997805563 shutdown_connections 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.538+0000 7fc486a16740 1 -- 172.21.15.103:0/1997805563 wait complete. 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.538+0000 7fc486a16740 1 -- start start 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.539+0000 7fc486a16740 1 Processor -- start 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.539+0000 7fc486a16740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9d98fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.539+0000 7fc486a16740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9d99500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.539+0000 7fc486a16740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9d99a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.539+0000 7fc486a16740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5580b9da73d0 con 0x5580b9d9bf20 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.539+0000 7fc486a16740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5580b9cb0210 con 0x5580b9d90f30 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.539+0000 7fc486a16740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5580b9da7550 con 0x5580b9d91850 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.539+0000 7fc486a16740 10 client.0 _ll_register_callbacks cb 0x5580b9c93140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc47be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9d99500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc481200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9d98fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.544 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc47be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9d99500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:45056/0 (socket says 172.21.15.103:45056) 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc47be00640 1 -- 172.21.15.103:0/1051343503 learned_addr learned my addr 172.21.15.103:0/1051343503 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc481c00640 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9d99a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc47be00640 1 -- 172.21.15.103:0/1051343503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 msgr2=0x5580b9d98fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc47be00640 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9d98fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc47be00640 1 -- 172.21.15.103:0/1051343503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 msgr2=0x5580b9d99a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc47be00640 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9d99a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc47be00640 1 -- 172.21.15.103:0/1051343503 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b9ad5620 con 0x5580b9d91850 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.540+0000 7fc481200640 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9d98fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:03.545 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.541+0000 7fc47be00640 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9d99500 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc468031000 tx=0x7fc4680058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7128bc8d8e44a5b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.541+0000 7fc462a00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:02:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.541+0000 7fc462a00640 1 -- 172.21.15.103:0/1051343503 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46804f020 con 0x5580b9d91850 2024-05-23T00:02:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.541+0000 7fc486a16740 1 -- 172.21.15.103:0/1051343503 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4680303a0 con 0x5580b9d91850 2024-05-23T00:02:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.541+0000 7fc462a00640 1 -- 172.21.15.103:0/1051343503 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc468005440 con 0x5580b9d91850 2024-05-23T00:02:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.542+0000 7fc462a00640 1 -- 172.21.15.103:0/1051343503 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc468057050 con 0x5580b9d91850 2024-05-23T00:02:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.542+0000 7fc486a16740 10 client.25966 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:03.550 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.542+0000 7fc486a16740 1 -- 172.21.15.103:0/1051343503 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5580b9da73d0 con 0x5580b9d91850 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.542+0000 7fc462a00640 1 -- 172.21.15.103:0/1051343503 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4680303a0 con 0x5580b9d91850 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.543+0000 7fc462a00640 1 -- 172.21.15.103:0/1051343503 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc46802f870 con 0x5580b9d91850 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.543+0000 7fc486a16740 10 client.25966 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.543+0000 7fc486a16740 10 client.25966 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.543+0000 7fc486a16740 20 client.25966 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.543+0000 7fc486a16740 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580b9db4930 0x5580b9db6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.543+0000 7fc486a16740 4 client.25966 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.543+0000 7fc486a16740 1 -- 172.21.15.103:0/1051343503 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580b9d08ec0 con 0x5580b9db4930 2024-05-23T00:02:03.551 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.543+0000 7fc481200640 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580b9db4930 0x5580b9db6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.552 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.544+0000 7fc481200640 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580b9db4930 0x5580b9db6d20 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef5dce6231bdd7b server_cookie=d6b7f85a35478666 in_seq=0 out_seq=0 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.544+0000 7fc462a00640 10 client.25966 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.545+0000 7fc462a00640 1 -- 172.21.15.103:0/1051343503 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5580b9d08ec0 con 0x5580b9db4930 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.545+0000 7fc462a00640 10 client.25966 handle_command_reply: tid=0 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.545+0000 7fc486a16740 1 client.25966 shutdown 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 -- 172.21.15.103:0/1051343503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 msgr2=0x5580b9d99500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9d99500 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc468031000 tx=0x7fc4680058d0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 -- 172.21.15.103:0/1051343503 shutdown_connections 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580b9db4930 0x5580b9db6d20 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9d99a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.553 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9d99500 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 --2- 172.21.15.103:0/1051343503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9d98fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc462a00640 0 client.25966 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 -- 172.21.15.103:0/1051343503 >> 172.21.15.103:0/1051343503 conn(0x5580b9d8d250 msgr2=0x5580b9d92f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 -- 172.21.15.103:0/1051343503 shutdown_connections 2024-05-23T00:02:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 1 -- 172.21.15.103:0/1051343503 wait complete. 2024-05-23T00:02:03.554 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.546+0000 7fc486a16740 20 client.25966 trim_cache size 0 max 16384 2024-05-23T00:02:03.569 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.566+0000 7fc486a16740 1 Processor -- start 2024-05-23T00:02:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc486a16740 1 -- start start 2024-05-23T00:02:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc486a16740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d9af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.570 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc486a16740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9dbc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc486a16740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9dbc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc486a16740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1af4c0 con 0x5580b9d91850 2024-05-23T00:02:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc486a16740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1acd90 con 0x5580b9d9bf20 2024-05-23T00:02:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc486a16740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1aabe0 con 0x5580b9d90f30 2024-05-23T00:02:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc47be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9dbc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.571 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc481200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d9af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc481200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d9af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:45064/0 (socket says 172.21.15.103:45064) 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc481200640 1 -- 172.21.15.103:0/3433843896 learned_addr learned my addr 172.21.15.103:0/3433843896 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.567+0000 7fc481c00640 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9dbc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.568+0000 7fc481200640 1 -- 172.21.15.103:0/3433843896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 msgr2=0x5580b9dbc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.568+0000 7fc481200640 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9dbc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.568+0000 7fc481200640 1 -- 172.21.15.103:0/3433843896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 msgr2=0x5580b9dbc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.568+0000 7fc481200640 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9dbc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.568+0000 7fc481200640 1 -- 172.21.15.103:0/3433843896 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580b9db1950 con 0x5580b9d90f30 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.568+0000 7fc481c00640 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9dbc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:02:03.572 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.568+0000 7fc481200640 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d9af00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x5580b9db7450 tx=0x7fc470016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4de72799ce40a56 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:03.573 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.568+0000 7fc46d600640 1 -- 172.21.15.103:0/3433843896 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47002a630 con 0x5580b9d90f30 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.569+0000 7fc46d600640 1 -- 172.21.15.103:0/3433843896 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47002f070 con 0x5580b9d90f30 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.569+0000 7fc46d600640 1 -- 172.21.15.103:0/3433843896 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc47001d940 con 0x5580b9d90f30 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.569+0000 7fc486a16740 1 -- 172.21.15.103:0/3433843896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 msgr2=0x5580b9d9af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.569+0000 7fc486a16740 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d9af00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x5580b9db7450 tx=0x7fc470016000 comp rx=0 tx=0).stop 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.569+0000 7fc486a16740 1 -- 172.21.15.103:0/3433843896 shutdown_connections 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.569+0000 7fc486a16740 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d9bf20 0x5580b9dbc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.570+0000 7fc486a16740 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d91850 0x5580b9dbc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.570+0000 7fc486a16740 1 --2- 172.21.15.103:0/3433843896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d90f30 0x5580b9d9af00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.570+0000 7fc486a16740 1 -- 172.21.15.103:0/3433843896 >> 172.21.15.103:0/3433843896 conn(0x5580b9d8d250 msgr2=0x5580b9d9a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:03.574 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.570+0000 7fc486a16740 1 -- 172.21.15.103:0/3433843896 shutdown_connections 2024-05-23T00:02:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.570+0000 7fc486a16740 1 -- 172.21.15.103:0/3433843896 wait complete. 2024-05-23T00:02:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.570+0000 7fc486a16740 1 -- start start 2024-05-23T00:02:03.575 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.570+0000 7fc486a16740 1 Processor -- start 2024-05-23T00:02:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc486a16740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9dac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc486a16740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9dace20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc486a16740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9da98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc486a16740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1ab250 con 0x5580b9d9bf20 2024-05-23T00:02:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc486a16740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1ac6c0 con 0x5580b9d90f30 2024-05-23T00:02:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc486a16740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc47c1ab920 con 0x5580b9d91850 2024-05-23T00:02:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc486a16740 10 client.0 _ll_register_callbacks cb 0x7fc468030860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:03.577 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc481200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9dac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc481c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9da98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc481200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9dac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:41706/0 (socket says 172.21.15.103:41706) 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.571+0000 7fc481200640 1 -- 172.21.15.103:0/295734715 learned_addr learned my addr 172.21.15.103:0/295734715 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc47be00640 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9dace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc481200640 1 -- 172.21.15.103:0/295734715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 msgr2=0x5580b9dace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc481200640 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9dace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc481200640 1 -- 172.21.15.103:0/295734715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 msgr2=0x5580b9da98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc481200640 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9da98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc481200640 1 -- 172.21.15.103:0/295734715 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580b9da9ef0 con 0x5580b9d90f30 2024-05-23T00:02:03.578 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc481c00640 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9da98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc481200640 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9dac8e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc470011ca0 tx=0x7fc470011ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e6dad6f6b769ec8 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc462a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.572+0000 7fc462a00640 1 -- 172.21.15.103:0/295734715 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc470039070 con 0x5580b9d90f30 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.573+0000 7fc462a00640 1 -- 172.21.15.103:0/295734715 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fc47002a770 con 0x5580b9d90f30 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.573+0000 7fc486a16740 1 -- 172.21.15.103:0/295734715 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc47002f070 con 0x5580b9d90f30 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.573+0000 7fc462a00640 1 -- 172.21.15.103:0/295734715 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc470042050 con 0x5580b9d90f30 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.573+0000 7fc486a16740 10 client.25964 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.573+0000 7fc486a16740 1 -- 172.21.15.103:0/295734715 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5580b9e1b660 con 0x5580b9d90f30 2024-05-23T00:02:03.579 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.573+0000 7fc462a00640 1 -- 172.21.15.103:0/295734715 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc47002f070 con 0x5580b9d90f30 2024-05-23T00:02:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.573+0000 7fc462a00640 1 -- 172.21.15.103:0/295734715 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fc47003c020 con 0x5580b9d90f30 2024-05-23T00:02:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.574+0000 7fc486a16740 10 client.25964 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.574+0000 7fc486a16740 10 client.25964 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.574+0000 7fc486a16740 20 client.25964 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.574+0000 7fc486a16740 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580b9e1be30 0x5580b9e1e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.574+0000 7fc486a16740 4 client.25964 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:02:03.580 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.574+0000 7fc486a16740 1 -- 172.21.15.103:0/295734715 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5580b9d02a10 con 0x5580b9e1be30 2024-05-23T00:02:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.575+0000 7fc47be00640 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580b9e1be30 0x5580b9e1e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.576+0000 7fc47be00640 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580b9e1be30 0x5580b9e1e250 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41d6257d1d171743 server_cookie=1b544a3408566cb7 in_seq=0 out_seq=0 2024-05-23T00:02:03.581 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.576+0000 7fc462a00640 10 client.25964 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:03.586 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc462a00640 1 -- 172.21.15.103:0/295734715 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5580b9d02a10 con 0x5580b9e1be30 2024-05-23T00:02:03.586 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc462a00640 10 client.25964 handle_command_reply: tid=0 2024-05-23T00:02:03.586 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 client.25964 shutdown 2024-05-23T00:02:03.586 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 -- 172.21.15.103:0/295734715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 msgr2=0x5580b9dac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.586 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9dac8e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc470011ca0 tx=0x7fc470011ed0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.586 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 -- 172.21.15.103:0/295734715 shutdown_connections 2024-05-23T00:02:03.586 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5580b9e1be30 0x5580b9e1e250 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.586 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580b9d9bf20 0x5580b9da98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc462a00640 0 client.25964 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5580b9d91850 0x5580b9dace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 --2- 172.21.15.103:0/295734715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5580b9d90f30 0x5580b9dac8e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.582+0000 7fc486a16740 1 -- 172.21.15.103:0/295734715 >> 172.21.15.103:0/295734715 conn(0x5580b9d8d250 msgr2=0x5580b9d9b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.583+0000 7fc486a16740 1 -- 172.21.15.103:0/295734715 shutdown_connections 2024-05-23T00:02:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.583+0000 7fc486a16740 1 -- 172.21.15.103:0/295734715 wait complete. 2024-05-23T00:02:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.583+0000 7fc486a16740 20 client.25964 trim_cache size 0 max 16384 2024-05-23T00:02:03.587 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:02:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:02:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:02:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:02:03.588 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.583+0000 7fc482600640 1 -- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4640a2e90 msgr2=0x7fc4640a5350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.583+0000 7fc482600640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4640a2e90 0x7fc4640a5350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x5580b9d924f0 tx=0x7fc470005f90 comp rx=0 tx=0).stop 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.583+0000 7fc482600640 1 -- 172.21.15.103:0/1285891411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1098b0 msgr2=0x7fc47c19eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.583+0000 7fc482600640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1098b0 0x7fc47c19eab0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc474009fd0 tx=0x7fc47400ef10 comp rx=0 tx=0).stop 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.585+0000 7fc482600640 1 -- 172.21.15.103:0/1285891411 shutdown_connections 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.585+0000 7fc482600640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4640a2e90 0x7fc4640a5350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.585+0000 7fc482600640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc47c1098b0 0x7fc47c19eab0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.585+0000 7fc482600640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc47c104260 0x7fc47c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.585+0000 7fc482600640 1 --2- 172.21.15.103:0/1285891411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc47c1038a0 0x7fc47c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.585+0000 7fc482600640 1 -- 172.21.15.103:0/1285891411 >> 172.21.15.103:0/1285891411 conn(0x7fc47c0ff350 msgr2=0x7fc47c0ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.585+0000 7fc482600640 1 -- 172.21.15.103:0/1285891411 shutdown_connections 2024-05-23T00:02:03.589 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:03.586+0000 7fc482600640 1 -- 172.21.15.103:0/1285891411 wait complete. 2024-05-23T00:02:04.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:04 smithi174 ceph-mon[36179]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-23T00:02:04.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:04 smithi105 ceph-mon[36426]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-23T00:02:04.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:04 smithi103 ceph-mon[31370]: pgmap v50: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2024-05-23T00:02:06.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:06 smithi105 ceph-mon[36426]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:02:06.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:06 smithi105 ceph-mon[36426]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-23T00:02:06.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:06 smithi103 ceph-mon[31370]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:02:06.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:06 smithi103 ceph-mon[31370]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-23T00:02:06.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:06 smithi174 ceph-mon[36179]: pgmap v51: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:02:06.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:06 smithi174 ceph-mon[36179]: mgrmap e27: x(active, since 92s), standbys: z, y 2024-05-23T00:02:08.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:08 smithi105 ceph-mon[36426]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:08.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:08 smithi103 ceph-mon[31370]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:08.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:08 smithi174 ceph-mon[36179]: pgmap v52: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:09.374 INFO:tasks.workunit.client.0.smithi103.stderr:Updating files: 64% (8044/12554) Updating files: 65% (8161/12554) Updating files: 66% (8286/12554) Updating files: 67% (8412/12554) Updating files: 68% (8537/12554) Updating files: 69% (8663/12554) Updating files: 70% (8788/12554) Updating files: 71% (8914/12554) Updating files: 72% (9039/12554) Updating files: 73% (9165/12554) Updating files: 74% (9290/12554) Updating files: 75% (9416/12554) Updating files: 76% (9542/12554) Updating files: 77% (9667/12554) Updating files: 78% (9793/12554) Updating files: 79% (9918/12554) Updating files: 80% (10044/12554) Updating files: 81% (10169/12554) Updating files: 82% (10295/12554) Updating files: 83% (10420/12554) Updating files: 84% (10546/12554) Updating files: 85% (10671/12554) Updating files: 86% (10797/12554) Updating files: 87% (10922/12554) Updating files: 88% (11048/12554) Updating files: 89% (11174/12554) Updating files: 90% (11299/12554) Updating files: 91% (11425/12554) Updating files: 92% (11550/12554) Updating files: 93% (11676/12554) Updating files: 94% (11801/12554) Updating files: 95% (11927/12554) Updating files: 96% (12052/12554) Updating files: 97% (12178/12554) Updating files: 98% (12303/12554) Updating files: 99% (12429/12554) Updating files: 100% (12554/12554) Updating files: 100% (12554/12554), done. 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr:Note: switching to '935217f1cb4dc722d5303f5ad364c883e9133376'. 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr:state without impacting any branches by switching back to a branch. 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:09.976 INFO:tasks.workunit.client.0.smithi103.stderr: git switch -c 2024-05-23T00:02:09.977 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:09.977 INFO:tasks.workunit.client.0.smithi103.stderr:Or undo this operation with: 2024-05-23T00:02:09.977 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:09.977 INFO:tasks.workunit.client.0.smithi103.stderr: git switch - 2024-05-23T00:02:09.977 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:09.977 INFO:tasks.workunit.client.0.smithi103.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-23T00:02:09.978 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:09.978 INFO:tasks.workunit.client.0.smithi103.stderr:HEAD is now at 935217f1cb4 Merge PR #57340 into squid 2024-05-23T00:02:09.986 DEBUG:teuthology.orchestra.run.smithi103:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-23T00:02:10.087 INFO:tasks.workunit.client.0.smithi103.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-23T00:02:10.094 INFO:tasks.workunit.client.0.smithi103.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-23T00:02:10.094 INFO:tasks.workunit.client.0.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-23T00:02:10.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:10 smithi105 ceph-mon[36426]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:02:10.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:10 smithi103 ceph-mon[31370]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:02:10.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:10 smithi174 ceph-mon[36179]: pgmap v53: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 24 op/s 2024-05-23T00:02:10.921 INFO:tasks.workunit.client.0.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-23T00:02:11.034 INFO:tasks.workunit.client.0.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-23T00:02:11.122 INFO:tasks.workunit.client.0.smithi103.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-23T00:02:11.128 INFO:tasks.workunit.client.0.smithi103.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-23T00:02:11.128 INFO:tasks.workunit.client.0.smithi103.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-23T00:02:11.207 INFO:tasks.workunit.client.0.smithi103.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-23T00:02:11.216 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-23T00:02:11.217 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-23T00:02:11.264 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-05-23T00:02:11.266 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-05-23T00:02:11.266 DEBUG:teuthology.orchestra.run.smithi103:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=935217f1cb4dc722d5303f5ad364c883e9133376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-05-23T00:02:11.380 INFO:tasks.workunit.client.0.smithi103.stderr:+ set -e 2024-05-23T00:02:11.381 INFO:tasks.workunit.client.0.smithi103.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-05-23T00:02:11.387 INFO:tasks.workunit.client.0.smithi103.stderr:Cloning into 'xfstests-dev'... 2024-05-23T00:02:12.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:12 smithi105 ceph-mon[36426]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:12.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:12 smithi103 ceph-mon[31370]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:12.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:12 smithi174 ceph-mon[36179]: pgmap v54: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:14.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:14 smithi105 ceph-mon[36426]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:14.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:14 smithi103 ceph-mon[31370]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:14.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:14 smithi174 ceph-mon[36179]: pgmap v55: 129 pgs: 129 active+clean; 1.4 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 21 op/s 2024-05-23T00:02:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.733+0000 7f9a76800640 1 -- 172.21.15.103:0/3942652384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a70109a80 msgr2=0x7f9a70109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.733+0000 7f9a76800640 1 --2- 172.21.15.103:0/3942652384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a70109a80 0x7f9a70109e60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009930 tx=0x7f9a6002a440 comp rx=0 tx=0).stop 2024-05-23T00:02:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.733+0000 7f9a76800640 1 -- 172.21.15.103:0/3942652384 shutdown_connections 2024-05-23T00:02:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.733+0000 7f9a76800640 1 --2- 172.21.15.103:0/3942652384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a70104430 0x7f9a70110940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.733+0000 7f9a76800640 1 --2- 172.21.15.103:0/3942652384 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a70103a70 0x7f9a70103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.733+0000 7f9a76800640 1 --2- 172.21.15.103:0/3942652384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a70109a80 0x7f9a70109e60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.737 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.733+0000 7f9a76800640 1 -- 172.21.15.103:0/3942652384 >> 172.21.15.103:0/3942652384 conn(0x7f9a700ff5a0 msgr2=0x7f9a701019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.734+0000 7f9a76800640 1 -- 172.21.15.103:0/3942652384 shutdown_connections 2024-05-23T00:02:16.738 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.734+0000 7f9a76800640 1 -- 172.21.15.103:0/3942652384 wait complete. 2024-05-23T00:02:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.734+0000 7f9a76800640 1 Processor -- start 2024-05-23T00:02:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a76800640 1 -- start start 2024-05-23T00:02:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a76800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a70103a70 0x7f9a7019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a76800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a70104430 0x7f9a7019a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a76800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a70109a80 0x7f9a7019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.740 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a76800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70113e60 con 0x7f9a70104430 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a76800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70113ce0 con 0x7f9a70109a80 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a76800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70113fe0 con 0x7f9a70103a70 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a66a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a70104430 0x7f9a7019a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a6f400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a70103a70 0x7f9a7019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a6f400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a70103a70 0x7f9a7019a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:50338/0 (socket says 172.21.15.103:50338) 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a6f400640 1 -- 172.21.15.103:0/1869370109 learned_addr learned my addr 172.21.15.103:0/1869370109 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.735+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a70109a80 0x7f9a7019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.736+0000 7f9a6f400640 1 -- 172.21.15.103:0/1869370109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a70109a80 msgr2=0x7f9a7019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.736+0000 7f9a6f400640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a70109a80 0x7f9a7019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.736+0000 7f9a6f400640 1 -- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a70104430 msgr2=0x7f9a7019a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.741 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.736+0000 7f9a6f400640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a70104430 0x7f9a7019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.736+0000 7f9a6f400640 1 -- 172.21.15.103:0/1869370109 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7019f310 con 0x7f9a70103a70 2024-05-23T00:02:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.736+0000 7f9a6f400640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a70103a70 0x7f9a7019a320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009230 tx=0x7f9a60004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.736+0000 7f9a6cc00640 1 -- 172.21.15.103:0/1869370109 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6003f030 con 0x7f9a70103a70 2024-05-23T00:02:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.737+0000 7f9a6cc00640 1 -- 172.21.15.103:0/1869370109 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a60004440 con 0x7f9a70103a70 2024-05-23T00:02:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.737+0000 7f9a76800640 1 -- 172.21.15.103:0/1869370109 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7019f5a0 con 0x7f9a70103a70 2024-05-23T00:02:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.737+0000 7f9a6cc00640 1 -- 172.21.15.103:0/1869370109 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60035070 con 0x7f9a70103a70 2024-05-23T00:02:16.742 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.737+0000 7f9a76800640 1 -- 172.21.15.103:0/1869370109 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a701a6e90 con 0x7f9a70103a70 2024-05-23T00:02:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.740+0000 7f9a6cc00640 1 -- 172.21.15.103:0/1869370109 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f9a60042030 con 0x7f9a70103a70 2024-05-23T00:02:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.740+0000 7f9a79774740 1 Processor -- start 2024-05-23T00:02:16.743 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.740+0000 7f9a79774740 1 -- start start 2024-05-23T00:02:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a79774740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac30fd50 0x55c0ac310130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a79774740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac31a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a79774740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac31d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a79774740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac2742e0 con 0x55c0ac310670 2024-05-23T00:02:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a79774740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac21be50 con 0x55c0ac31ad40 2024-05-23T00:02:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a79774740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac2702a0 con 0x55c0ac30fd50 2024-05-23T00:02:16.745 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac31a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac30fd50 0x55c0ac310130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac30fd50 0x55c0ac310130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:50352/0 (socket says 172.21.15.103:50352) 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6fe00640 1 -- 172.21.15.103:0/2334081771 learned_addr learned my addr 172.21.15.103:0/2334081771 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6f400640 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac31a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41236/0 (socket says 172.21.15.103:41236) 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a66a00640 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac31d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6fe00640 1 -- 172.21.15.103:0/2334081771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 msgr2=0x55c0ac31d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac31d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6cc00640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9a540a2da0 0x7f9a540a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6fe00640 1 -- 172.21.15.103:0/2334081771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 msgr2=0x55c0ac31a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac31a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.741+0000 7f9a6fe00640 1 -- 172.21.15.103:0/2334081771 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0abf88210 con 0x55c0ac30fd50 2024-05-23T00:02:16.746 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.742+0000 7f9a6cc00640 1 -- 172.21.15.103:0/1869370109 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9a600efb60 con 0x7f9a70103a70 2024-05-23T00:02:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.742+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac30fd50 0x55c0ac310130 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009f50 tx=0x7f9a680118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf591f510acce597 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.742+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9a540a2da0 0x7f9a540a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.742+0000 7f9a64c00640 1 -- 172.21.15.103:0/2334081771 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6801e070 con 0x55c0ac30fd50 2024-05-23T00:02:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.742+0000 7f9a64c00640 1 -- 172.21.15.103:0/2334081771 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a68003d70 con 0x55c0ac30fd50 2024-05-23T00:02:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.742+0000 7f9a64c00640 1 -- 172.21.15.103:0/2334081771 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6800fca0 con 0x55c0ac30fd50 2024-05-23T00:02:16.750 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 -- 172.21.15.103:0/2334081771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac30fd50 msgr2=0x55c0ac310130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.750 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9a540a2da0 0x7f9a540a5260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a68010010 tx=0x7f9a6800ff20 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:16.750 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac30fd50 0x55c0ac310130 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009f50 tx=0x7f9a680118d0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.750 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 -- 172.21.15.103:0/2334081771 shutdown_connections 2024-05-23T00:02:16.750 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac31d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.750 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac31a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.750 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 --2- 172.21.15.103:0/2334081771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac30fd50 0x55c0ac310130 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 -- 172.21.15.103:0/2334081771 >> 172.21.15.103:0/2334081771 conn(0x55c0ac30c070 msgr2=0x55c0ac30c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 -- 172.21.15.103:0/2334081771 shutdown_connections 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.743+0000 7f9a79774740 1 -- 172.21.15.103:0/2334081771 wait complete. 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 1 -- start start 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 1 Processor -- start 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac317bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac318110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac31ad40 0x55c0ac318650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac31dbd0 con 0x55c0ac310670 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac22f210 con 0x55c0ac30fd50 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c0ac31dd50 con 0x55c0ac31ad40 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.744+0000 7f9a79774740 10 client.0 _ll_register_callbacks cb 0x55c0ac212140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:16.751 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac317bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac317bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:44240/0 (socket says 172.21.15.103:44240) 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 -- 172.21.15.103:0/2302911710 learned_addr learned my addr 172.21.15.103:0/2302911710 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a66a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac318110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac31ad40 0x55c0ac318650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 -- 172.21.15.103:0/2302911710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac31ad40 msgr2=0x55c0ac318650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac31ad40 0x55c0ac318650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 -- 172.21.15.103:0/2302911710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 msgr2=0x55c0ac318110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac318110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 -- 172.21.15.103:0/2302911710 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0ac054620 con 0x55c0ac30fd50 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a6f400640 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac317bd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9a600ecec0 tx=0x7f9a60005c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd39c624e31b98b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a5d600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.745+0000 7f9a5d600640 1 -- 172.21.15.103:0/2302911710 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60049070 con 0x55c0ac30fd50 2024-05-23T00:02:16.752 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a5d600640 1 -- 172.21.15.103:0/2302911710 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9a600ed0e0 con 0x55c0ac30fd50 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 1 -- 172.21.15.103:0/2302911710 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a68003d70 con 0x55c0ac30fd50 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a5d600640 1 -- 172.21.15.103:0/2302911710 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9a600bb820 con 0x55c0ac30fd50 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 10 client.25976 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 1 -- 172.21.15.103:0/2302911710 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c0ac329da0 con 0x55c0ac30fd50 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a5d600640 1 -- 172.21.15.103:0/2302911710 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a68003d70 con 0x55c0ac30fd50 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a5d600640 1 -- 172.21.15.103:0/2302911710 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9a60031c60 con 0x55c0ac30fd50 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 10 client.25976 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 10 client.25976 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 20 client.25976 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55c0ac32b2d0 0x55c0ac32d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 4 client.25976 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.746+0000 7f9a79774740 1 -- 172.21.15.103:0/2302911710 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0ac287ec0 con 0x55c0ac32b2d0 2024-05-23T00:02:16.753 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.747+0000 7f9a66a00640 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55c0ac32b2d0 0x55c0ac32d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.747+0000 7f9a66a00640 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55c0ac32b2d0 0x55c0ac32d790 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b759a6c1e0424357 server_cookie=dc7c1b8abdfb89d5 in_seq=0 out_seq=0 2024-05-23T00:02:16.754 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.747+0000 7f9a5d600640 10 client.25976 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a5d600640 1 -- 172.21.15.103:0/2302911710 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c0ac287ec0 con 0x55c0ac32b2d0 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a5d600640 10 client.25976 handle_command_reply: tid=0 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 client.25976 shutdown 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 -- 172.21.15.103:0/2302911710 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 msgr2=0x55c0ac317bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac317bd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9a600ecec0 tx=0x7f9a60005c00 comp rx=0 tx=0).stop 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 -- 172.21.15.103:0/2302911710 shutdown_connections 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55c0ac32b2d0 0x55c0ac32d790 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac31ad40 0x55c0ac318650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a5d600640 0 client.25976 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac310670 0x55c0ac318110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.755 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 --2- 172.21.15.103:0/2302911710 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac317bd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 -- 172.21.15.103:0/2302911710 >> 172.21.15.103:0/2302911710 conn(0x55c0ac30c070 msgr2=0x55c0ac30b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 -- 172.21.15.103:0/2302911710 shutdown_connections 2024-05-23T00:02:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.748+0000 7f9a79774740 1 -- 172.21.15.103:0/2302911710 wait complete. 2024-05-23T00:02:16.756 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.749+0000 7f9a79774740 20 client.25976 trim_cache size 0 max 16384 2024-05-23T00:02:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a79774740 1 Processor -- start 2024-05-23T00:02:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a79774740 1 -- start start 2024-05-23T00:02:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a79774740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac30fd50 0x55c0ac326ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a79774740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac333060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.769 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a79774740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac3335a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a79774740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a701b3910 con 0x55c0ac30fd50 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a79774740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a701b11e0 con 0x55c0ac31ad40 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a79774740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a701af030 con 0x55c0ac310670 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.765+0000 7f9a6fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac3335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac3335a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:44248/0 (socket says 172.21.15.103:44248) 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6fe00640 1 -- 172.21.15.103:0/2335674061 learned_addr learned my addr 172.21.15.103:0/2335674061 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6f400640 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac30fd50 0x55c0ac326ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a66a00640 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac333060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6fe00640 1 -- 172.21.15.103:0/2335674061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 msgr2=0x55c0ac333060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac333060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6fe00640 1 -- 172.21.15.103:0/2335674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac30fd50 msgr2=0x55c0ac326ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac30fd50 0x55c0ac326ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.770 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6fe00640 1 -- 172.21.15.103:0/2335674061 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0ac311200 con 0x55c0ac31ad40 2024-05-23T00:02:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6f400640 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac30fd50 0x55c0ac326ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:02:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a66a00640 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac333060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:02:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac3335a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9a6801c040 tx=0x7f9a68027000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=508fce4ee0256912 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.766+0000 7f9a5fe00640 1 -- 172.21.15.103:0/2335674061 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68038070 con 0x55c0ac31ad40 2024-05-23T00:02:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a5fe00640 1 -- 172.21.15.103:0/2335674061 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a6800f7d0 con 0x55c0ac31ad40 2024-05-23T00:02:16.771 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a5fe00640 1 -- 172.21.15.103:0/2335674061 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6801e040 con 0x55c0ac31ad40 2024-05-23T00:02:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 -- 172.21.15.103:0/2335674061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 msgr2=0x55c0ac3335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.774 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac3335a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9a6801c040 tx=0x7f9a68027000 comp rx=0 tx=0).stop 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 -- 172.21.15.103:0/2335674061 shutdown_connections 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac31ad40 0x55c0ac3335a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac333060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 --2- 172.21.15.103:0/2335674061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac30fd50 0x55c0ac326ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 -- 172.21.15.103:0/2335674061 >> 172.21.15.103:0/2335674061 conn(0x55c0ac30c070 msgr2=0x55c0ac30b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 -- 172.21.15.103:0/2335674061 shutdown_connections 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.767+0000 7f9a79774740 1 -- 172.21.15.103:0/2335674061 wait complete. 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 1 -- start start 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 1 Processor -- start 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac31eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac31f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.775 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac31ad40 0x55c0ac31f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9a701af6a0 con 0x55c0ac31ad40 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9a701b0b10 con 0x55c0ac30fd50 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a701afd70 con 0x55c0ac310670 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.768+0000 7f9a79774740 10 client.0 _ll_register_callbacks cb 0x7f9a60005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac31ad40 0x55c0ac31f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac31ad40 0x55c0ac31f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41258/0 (socket says 172.21.15.103:41258) 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 -- 172.21.15.103:0/1918295709 learned_addr learned my addr 172.21.15.103:0/1918295709 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6f400640 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac31eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a66a00640 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac31f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 -- 172.21.15.103:0/1918295709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 msgr2=0x55c0ac31f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac31f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.776 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 -- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 msgr2=0x55c0ac31eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac31eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 -- 172.21.15.103:0/1918295709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0ac399240 con 0x55c0ac31ad40 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6f400640 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac31eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a6fe00640 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac31ad40 0x55c0ac31f5d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009540 tx=0x7f9a6800fbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81553861febc8fd server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.769+0000 7f9a5d600640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.770+0000 7f9a5d600640 1 -- 172.21.15.103:0/1918295709 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a68043060 con 0x55c0ac31ad40 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.770+0000 7f9a79774740 1 -- 172.21.15.103:0/1918295709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a6800f7d0 con 0x55c0ac31ad40 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.770+0000 7f9a5d600640 1 -- 172.21.15.103:0/1918295709 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9a68003c40 con 0x55c0ac31ad40 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.770+0000 7f9a5d600640 1 -- 172.21.15.103:0/1918295709 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9a6804a050 con 0x55c0ac31ad40 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.770+0000 7f9a79774740 10 client.15882 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.770+0000 7f9a79774740 1 -- 172.21.15.103:0/1918295709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c0ac39a530 con 0x55c0ac31ad40 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.770+0000 7f9a5d600640 1 -- 172.21.15.103:0/1918295709 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a6800f7d0 con 0x55c0ac31ad40 2024-05-23T00:02:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a5d600640 1 -- 172.21.15.103:0/1918295709 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9a6800e640 con 0x55c0ac31ad40 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a79774740 10 client.15882 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a79774740 10 client.15882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a79774740 20 client.15882 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a79774740 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55c0ac39ab80 0x55c0ac39cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a79774740 4 client.15882 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a79774740 1 -- 172.21.15.103:0/1918295709 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c0ac281a10 con 0x55c0ac39ab80 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a6f400640 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55c0ac39ab80 0x55c0ac39cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a6f400640 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55c0ac39ab80 0x55c0ac39cfa0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca3503ef1459cffd server_cookie=624194e676286983 in_seq=0 out_seq=0 2024-05-23T00:02:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.771+0000 7f9a5d600640 10 client.15882 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a5d600640 1 -- 172.21.15.103:0/1918295709 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c0ac281a10 con 0x55c0ac39ab80 2024-05-23T00:02:16.779 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a5d600640 10 client.15882 handle_command_reply: tid=0 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 client.15882 shutdown 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 -- 172.21.15.103:0/1918295709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac31ad40 msgr2=0x55c0ac31f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac31ad40 0x55c0ac31f5d0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9a68009540 tx=0x7f9a6800fbb0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 -- 172.21.15.103:0/1918295709 shutdown_connections 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55c0ac39ab80 0x55c0ac39cfa0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c0ac31ad40 0x55c0ac31f5d0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c0ac310670 0x55c0ac31f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 --2- 172.21.15.103:0/1918295709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c0ac30fd50 0x55c0ac31eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a5d600640 0 client.15882 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.775+0000 7f9a79774740 1 -- 172.21.15.103:0/1918295709 >> 172.21.15.103:0/1918295709 conn(0x55c0ac30c070 msgr2=0x55c0ac3114a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.776+0000 7f9a79774740 1 -- 172.21.15.103:0/1918295709 shutdown_connections 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.776+0000 7f9a79774740 1 -- 172.21.15.103:0/1918295709 wait complete. 2024-05-23T00:02:16.780 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.776+0000 7f9a79774740 20 client.15882 trim_cache size 0 max 16384 2024-05-23T00:02:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:02:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:02:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:02:16.781 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.776+0000 7f9a76800640 1 -- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9a540a2da0 msgr2=0x7f9a540a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.776+0000 7f9a76800640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9a540a2da0 0x7f9a540a5260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9a68010010 tx=0x7f9a6800ff20 comp rx=0 tx=0).stop 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.776+0000 7f9a76800640 1 -- 172.21.15.103:0/1869370109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a70103a70 msgr2=0x7f9a7019a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.776+0000 7f9a76800640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a70103a70 0x7f9a7019a320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9a60009230 tx=0x7f9a60004040 comp rx=0 tx=0).stop 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.777+0000 7f9a76800640 1 -- 172.21.15.103:0/1869370109 shutdown_connections 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.777+0000 7f9a76800640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9a540a2da0 0x7f9a540a5260 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.777+0000 7f9a76800640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9a70109a80 0x7f9a7019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.777+0000 7f9a76800640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9a70104430 0x7f9a7019a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.777+0000 7f9a76800640 1 --2- 172.21.15.103:0/1869370109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a70103a70 0x7f9a7019a320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.777+0000 7f9a76800640 1 -- 172.21.15.103:0/1869370109 >> 172.21.15.103:0/1869370109 conn(0x7f9a700ff5a0 msgr2=0x7f9a700ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:16.782 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.777+0000 7f9a76800640 1 -- 172.21.15.103:0/1869370109 shutdown_connections 2024-05-23T00:02:16.783 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:16.777+0000 7f9a76800640 1 -- 172.21.15.103:0/1869370109 wait complete. 2024-05-23T00:02:16.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:16 smithi105 ceph-mon[36426]: pgmap v56: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 26 op/s 2024-05-23T00:02:16.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:16 smithi103 ceph-mon[31370]: pgmap v56: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 26 op/s 2024-05-23T00:02:16.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:16 smithi174 ceph-mon[36179]: pgmap v56: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 38 KiB/s wr, 26 op/s 2024-05-23T00:02:17.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-23T00:02:17.324 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-23T00:02:17.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 -- 172.21.15.103:0/603037027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea501126f0 msgr2=0x7fea50114ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 --2- 172.21.15.103:0/603037027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea501126f0 0x7fea50114ae0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fea48009ca0 tx=0x7fea4802a8c0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 -- 172.21.15.103:0/603037027 shutdown_connections 2024-05-23T00:02:17.542 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 --2- 172.21.15.103:0/603037027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea501126f0 0x7fea50114ae0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 --2- 172.21.15.103:0/603037027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea500766c0 0x7fea501121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 --2- 172.21.15.103:0/603037027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea50075da0 0x7fea50076180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 -- 172.21.15.103:0/603037027 >> 172.21.15.103:0/603037027 conn(0x7fea500ff520 msgr2=0x7fea50101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 -- 172.21.15.103:0/603037027 shutdown_connections 2024-05-23T00:02:17.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.539+0000 7fea55a00640 1 -- 172.21.15.103:0/603037027 wait complete. 2024-05-23T00:02:17.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.540+0000 7fea55a00640 1 Processor -- start 2024-05-23T00:02:17.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.540+0000 7fea55a00640 1 -- start start 2024-05-23T00:02:17.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.540+0000 7fea55a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea50075da0 0x7fea5019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.540+0000 7fea55a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500766c0 0x7fea5019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.540+0000 7fea55a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea501126f0 0x7fea501a3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.540+0000 7fea55a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea50118230 con 0x7fea500766c0 2024-05-23T00:02:17.545 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.540+0000 7fea55a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fea501180b0 con 0x7fea50075da0 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.540+0000 7fea55a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fea501183b0 con 0x7fea501126f0 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500766c0 0x7fea5019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500766c0 0x7fea5019ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41276/0 (socket says 172.21.15.103:41276) 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 -- 172.21.15.103:0/652498611 learned_addr learned my addr 172.21.15.103:0/652498611 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea4ea00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea50075da0 0x7fea5019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea4f400640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea501126f0 0x7fea501a3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 -- 172.21.15.103:0/652498611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea501126f0 msgr2=0x7fea501a3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea501126f0 0x7fea501a3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 -- 172.21.15.103:0/652498611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea50075da0 msgr2=0x7fea5019e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea50075da0 0x7fea5019e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 -- 172.21.15.103:0/652498611 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea501a3740 con 0x7fea500766c0 2024-05-23T00:02:17.546 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.541+0000 7fea47e00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500766c0 0x7fea5019ec90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fea3800e9f0 tx=0x7fea3800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:17.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.542+0000 7fea47400640 1 -- 172.21.15.103:0/652498611 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea3800cd80 con 0x7fea500766c0 2024-05-23T00:02:17.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.542+0000 7fea47400640 1 -- 172.21.15.103:0/652498611 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea38010910 con 0x7fea500766c0 2024-05-23T00:02:17.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.542+0000 7fea55a00640 1 -- 172.21.15.103:0/652498611 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea501a3a30 con 0x7fea500766c0 2024-05-23T00:02:17.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.542+0000 7fea47400640 1 -- 172.21.15.103:0/652498611 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea38018990 con 0x7fea500766c0 2024-05-23T00:02:17.547 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.542+0000 7fea55a00640 1 -- 172.21.15.103:0/652498611 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea501ab2c0 con 0x7fea500766c0 2024-05-23T00:02:17.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.545+0000 7fea47400640 1 -- 172.21.15.103:0/652498611 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fea38010ac0 con 0x7fea500766c0 2024-05-23T00:02:17.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.545+0000 7fea589d3740 1 Processor -- start 2024-05-23T00:02:17.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea589d3740 1 -- start start 2024-05-23T00:02:17.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea589d3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab85d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea589d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312ab90450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea589d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 0x56312ab92d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea589d3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56312ab49320 con 0x56312ab859a0 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea589d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56312aaeacb0 con 0x56312ab862c0 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea589d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56312aa91fc0 con 0x56312ab90990 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea4f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab85d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea4ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312ab90450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea4f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab85d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41288/0 (socket says 172.21.15.103:41288) 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea4f400640 1 -- 172.21.15.103:0/97440164 learned_addr learned my addr 172.21.15.103:0/97440164 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.546+0000 7fea47e00640 1 --2- 172.21.15.103:0/97440164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 0x56312ab92d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea4ea00640 1 -- 172.21.15.103:0/97440164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 msgr2=0x56312ab92d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea4ea00640 1 --2- 172.21.15.103:0/97440164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 0x56312ab92d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea4ea00640 1 -- 172.21.15.103:0/97440164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 msgr2=0x56312ab85d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea4ea00640 1 --2- 172.21.15.103:0/97440164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab85d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea4ea00640 1 -- 172.21.15.103:0/97440164 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56312a8c87d0 con 0x56312ab862c0 2024-05-23T00:02:17.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea4ea00640 1 --2- 172.21.15.103:0/97440164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312ab90450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fea40009c00 tx=0x7fea40011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2fcd6c39efee3cc server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:17.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea47400640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fea340a2d00 0x7fea340a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea3fe00640 1 -- 172.21.15.103:0/97440164 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40004230 con 0x56312ab862c0 2024-05-23T00:02:17.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea47400640 1 -- 172.21.15.103:0/652498611 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fea380cfb30 con 0x7fea500766c0 2024-05-23T00:02:17.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea4f400640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fea340a2d00 0x7fea340a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea3fe00640 1 -- 172.21.15.103:0/97440164 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea400043d0 con 0x56312ab862c0 2024-05-23T00:02:17.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.547+0000 7fea3fe00640 1 -- 172.21.15.103:0/97440164 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40004d70 con 0x56312ab862c0 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 -- 172.21.15.103:0/97440164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 msgr2=0x56312ab90450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea4f400640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fea340a2d00 0x7fea340a51c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fea48009430 tx=0x7fea48009360 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 --2- 172.21.15.103:0/97440164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312ab90450 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fea40009c00 tx=0x7fea40011810 comp rx=0 tx=0).stop 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 -- 172.21.15.103:0/97440164 shutdown_connections 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 --2- 172.21.15.103:0/97440164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 0x56312ab92d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 --2- 172.21.15.103:0/97440164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312ab90450 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 --2- 172.21.15.103:0/97440164 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab85d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 -- 172.21.15.103:0/97440164 >> 172.21.15.103:0/97440164 conn(0x56312ab81cc0 msgr2=0x56312ab820a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:17.555 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 -- 172.21.15.103:0/97440164 shutdown_connections 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.548+0000 7fea589d3740 1 -- 172.21.15.103:0/97440164 wait complete. 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.549+0000 7fea589d3740 1 -- start start 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.549+0000 7fea589d3740 1 Processor -- start 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.549+0000 7fea589d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.549+0000 7fea589d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 0x56312ab8ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.549+0000 7fea589d3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.549+0000 7fea589d3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56312aaa5410 con 0x56312ab90990 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.549+0000 7fea589d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x56312aae6c70 con 0x56312ab859a0 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.549+0000 7fea589d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x56312ab93780 con 0x56312ab862c0 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea589d3740 10 client.0 _ll_register_callbacks cb 0x56312aae10d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:17.556 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41300/0 (socket says 172.21.15.103:41300) 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4f400640 1 -- 172.21.15.103:0/2153381144 learned_addr learned my addr 172.21.15.103:0/2153381144 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea47e00640 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 0x56312ab8ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4f400640 1 -- 172.21.15.103:0/2153381144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 msgr2=0x56312ab8ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4f400640 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 0x56312ab8ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4f400640 1 -- 172.21.15.103:0/2153381144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 msgr2=0x56312ab8d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4f400640 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4f400640 1 -- 172.21.15.103:0/2153381144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56312a7fe3f0 con 0x56312ab90990 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.550+0000 7fea4ea00640 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.551+0000 7fea4f400640 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8e300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fea4802f540 tx=0x7fea4802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=befb3b19c3e0349b server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.551+0000 7fea3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:02:17.557 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.551+0000 7fea3cc00640 1 -- 172.21.15.103:0/2153381144 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48031720 con 0x56312ab90990 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.551+0000 7fea3cc00640 1 -- 172.21.15.103:0/2153381144 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fea480318c0 con 0x56312ab90990 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.551+0000 7fea589d3740 1 -- 172.21.15.103:0/2153381144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea400043d0 con 0x56312ab90990 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.551+0000 7fea3cc00640 1 -- 172.21.15.103:0/2153381144 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fea480305c0 con 0x56312ab90990 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.551+0000 7fea589d3740 10 client.15894 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.551+0000 7fea589d3740 1 -- 172.21.15.103:0/2153381144 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56312a7be0f0 con 0x56312ab90990 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea3cc00640 1 -- 172.21.15.103:0/2153381144 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea400043d0 con 0x56312ab90990 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea3cc00640 1 -- 172.21.15.103:0/2153381144 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fea48051020 con 0x56312ab90990 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea589d3740 10 client.15894 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea589d3740 10 client.15894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea589d3740 20 client.15894 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea589d3740 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56312aba0b10 0x56312aba2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea589d3740 4 client.15894 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:17.558 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea589d3740 1 -- 172.21.15.103:0/2153381144 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56312aaa9e00 con 0x56312aba0b10 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.552+0000 7fea4ea00640 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56312aba0b10 0x56312aba2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.553+0000 7fea4ea00640 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56312aba0b10 0x56312aba2f70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a64b939b3e63b392 server_cookie=c6151d2e900c969d in_seq=0 out_seq=0 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.553+0000 7fea3cc00640 10 client.15894 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea3cc00640 1 -- 172.21.15.103:0/2153381144 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56312aaa9e00 con 0x56312aba0b10 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea3cc00640 10 client.15894 handle_command_reply: tid=0 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 client.15894 shutdown 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 -- 172.21.15.103:0/2153381144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 msgr2=0x56312ab8e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8e300 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fea4802f540 tx=0x7fea4802a8c0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.560 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 -- 172.21.15.103:0/2153381144 shutdown_connections 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56312aba0b10 0x56312aba2f70 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8e300 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 0x56312ab8ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 --2- 172.21.15.103:0/2153381144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea3cc00640 0 client.15894 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 -- 172.21.15.103:0/2153381144 >> 172.21.15.103:0/2153381144 conn(0x56312ab81cc0 msgr2=0x56312ab879c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 -- 172.21.15.103:0/2153381144 shutdown_connections 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 1 -- 172.21.15.103:0/2153381144 wait complete. 2024-05-23T00:02:17.561 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.554+0000 7fea589d3740 20 client.15894 trim_cache size 0 max 16384 2024-05-23T00:02:17.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.573+0000 7fea589d3740 1 Processor -- start 2024-05-23T00:02:17.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea589d3740 1 -- start start 2024-05-23T00:02:17.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea589d3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab9c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea589d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312aba8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea589d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 0x56312aba8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea589d3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea501b7e20 con 0x56312ab859a0 2024-05-23T00:02:17.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea589d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fea501b56f0 con 0x56312ab862c0 2024-05-23T00:02:17.577 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea589d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fea501b3540 con 0x56312ab90990 2024-05-23T00:02:17.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea4ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab9c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea4ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab9c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41310/0 (socket says 172.21.15.103:41310) 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea4ea00640 1 -- 172.21.15.103:0/3408480280 learned_addr learned my addr 172.21.15.103:0/3408480280 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea47e00640 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312aba8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.574+0000 7fea4f400640 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 0x56312aba8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea4ea00640 1 -- 172.21.15.103:0/3408480280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 msgr2=0x56312aba8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea4ea00640 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 0x56312aba8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea4ea00640 1 -- 172.21.15.103:0/3408480280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 msgr2=0x56312aba8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea4ea00640 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312aba8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea4ea00640 1 -- 172.21.15.103:0/3408480280 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56312ab8bdf0 con 0x56312ab859a0 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea47e00640 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312aba8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea4ea00640 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab9c4f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fea40009c00 tx=0x7fea400048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e42a482b985e13 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea3f400640 1 -- 172.21.15.103:0/3408480280 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea40032070 con 0x56312ab859a0 2024-05-23T00:02:17.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea3f400640 1 -- 172.21.15.103:0/3408480280 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea4002a070 con 0x56312ab859a0 2024-05-23T00:02:17.580 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.575+0000 7fea3f400640 1 -- 172.21.15.103:0/3408480280 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4001e040 con 0x56312ab859a0 2024-05-23T00:02:17.582 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 -- 172.21.15.103:0/3408480280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 msgr2=0x56312ab9c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab9c4f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fea40009c00 tx=0x7fea400048d0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 -- 172.21.15.103:0/3408480280 shutdown_connections 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab90990 0x56312aba8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab862c0 0x56312aba8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 --2- 172.21.15.103:0/3408480280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab859a0 0x56312ab9c4f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 -- 172.21.15.103:0/3408480280 >> 172.21.15.103:0/3408480280 conn(0x56312ab81cc0 msgr2=0x56312ab8c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 -- 172.21.15.103:0/3408480280 shutdown_connections 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.576+0000 7fea589d3740 1 -- 172.21.15.103:0/3408480280 wait complete. 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 1 -- start start 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 1 Processor -- start 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 0x56312ab8a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fea501b3bb0 con 0x56312ab90990 2024-05-23T00:02:17.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fea501b5020 con 0x56312ab859a0 2024-05-23T00:02:17.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fea501b4280 con 0x56312ab862c0 2024-05-23T00:02:17.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.577+0000 7fea589d3740 10 client.0 _ll_register_callbacks cb 0x7fea48009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:17.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41320/0 (socket says 172.21.15.103:41320) 2024-05-23T00:02:17.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 -- 172.21.15.103:0/3931478288 learned_addr learned my addr 172.21.15.103:0/3931478288 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:17.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4ea00640 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea47e00640 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 0x56312ab8a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 -- 172.21.15.103:0/3931478288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 msgr2=0x56312ab8a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 0x56312ab8a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 -- 172.21.15.103:0/3931478288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 msgr2=0x56312ab8a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 -- 172.21.15.103:0/3931478288 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56312ac060f0 con 0x56312ab90990 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea4f400640 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8ae80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fea4802ade0 tx=0x7fea48009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed09e09092c9a73 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.578+0000 7fea3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea3cc00640 1 -- 172.21.15.103:0/3931478288 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea480043c0 con 0x56312ab90990 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea3cc00640 1 -- 172.21.15.103:0/3931478288 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fea4802f840 con 0x56312ab90990 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 1 -- 172.21.15.103:0/3931478288 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fea4002a070 con 0x56312ab90990 2024-05-23T00:02:17.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea3cc00640 1 -- 172.21.15.103:0/3931478288 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fea48009540 con 0x56312ab90990 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 10 client.15900 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 1 -- 172.21.15.103:0/3931478288 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56312ac072a0 con 0x56312ab90990 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea3cc00640 1 -- 172.21.15.103:0/3931478288 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea4002a070 con 0x56312ab90990 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea3cc00640 1 -- 172.21.15.103:0/3931478288 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fea48051070 con 0x56312ab90990 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 10 client.15900 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 10 client.15900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 20 client.15900 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56312ac078b0 0x56312ac09cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 4 client.15900 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.579+0000 7fea589d3740 1 -- 172.21.15.103:0/3931478288 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56312aaf83e0 con 0x56312ac078b0 2024-05-23T00:02:17.586 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.580+0000 7fea4ea00640 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56312ac078b0 0x56312ac09cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.580+0000 7fea4ea00640 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56312ac078b0 0x56312ac09cd0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed52168b9b5271a server_cookie=dcfc4c34fbf0997c in_seq=0 out_seq=0 2024-05-23T00:02:17.587 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.580+0000 7fea3cc00640 10 client.15900 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.580+0000 7fea3cc00640 1 -- 172.21.15.103:0/3931478288 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x56312aaf83e0 con 0x56312ac078b0 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.580+0000 7fea3cc00640 10 client.15900 handle_command_reply: tid=0 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 client.15900 shutdown 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 -- 172.21.15.103:0/3931478288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 msgr2=0x56312ab8ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8ae80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fea4802ade0 tx=0x7fea48009910 comp rx=0 tx=0).stop 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 -- 172.21.15.103:0/3931478288 shutdown_connections 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x56312ac078b0 0x56312ac09cd0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56312ab90990 0x56312ab8ae80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x56312ab862c0 0x56312ab8a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 --2- 172.21.15.103:0/3931478288 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56312ab859a0 0x56312ab8a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea3cc00640 0 client.15900 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 -- 172.21.15.103:0/3931478288 >> 172.21.15.103:0/3931478288 conn(0x56312ab81cc0 msgr2=0x56312ab87130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:17.588 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 -- 172.21.15.103:0/3931478288 shutdown_connections 2024-05-23T00:02:17.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 1 -- 172.21.15.103:0/3931478288 wait complete. 2024-05-23T00:02:17.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.581+0000 7fea589d3740 20 client.15900 trim_cache size 0 max 16384 2024-05-23T00:02:17.589 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:02:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "status": "no active scrubs running", 2024-05-23T00:02:17.589 INFO:teuthology.orchestra.run.smithi103.stdout: "scrubs": {} 2024-05-23T00:02:17.589 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 -- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fea340a2d00 msgr2=0x7fea340a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fea340a2d00 0x7fea340a51c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fea48009430 tx=0x7fea48009360 comp rx=0 tx=0).stop 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 -- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500766c0 msgr2=0x7fea5019ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500766c0 0x7fea5019ec90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fea3800e9f0 tx=0x7fea3800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 -- 172.21.15.103:0/652498611 shutdown_connections 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fea340a2d00 0x7fea340a51c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea501126f0 0x7fea501a3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fea500766c0 0x7fea5019ec90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 --2- 172.21.15.103:0/652498611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea50075da0 0x7fea5019e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 -- 172.21.15.103:0/652498611 >> 172.21.15.103:0/652498611 conn(0x7fea500ff520 msgr2=0x7fea50113d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:17.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 -- 172.21.15.103:0/652498611 shutdown_connections 2024-05-23T00:02:17.591 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.582+0000 7fea55a00640 1 -- 172.21.15.103:0/652498611 wait complete. 2024-05-23T00:02:17.601 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-23T00:02:17.602 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:02:17.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.783+0000 7f813d400640 1 -- 172.21.15.103:0/2497262591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8138075da0 msgr2=0x7f8138076180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.783+0000 7f813d400640 1 --2- 172.21.15.103:0/2497262591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8138075da0 0x7f8138076180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8128009930 tx=0x7f812802a420 comp rx=0 tx=0).stop 2024-05-23T00:02:17.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.784+0000 7f813d400640 1 -- 172.21.15.103:0/2497262591 shutdown_connections 2024-05-23T00:02:17.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.784+0000 7f813d400640 1 --2- 172.21.15.103:0/2497262591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81381126f0 0x7f8138114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.784+0000 7f813d400640 1 --2- 172.21.15.103:0/2497262591 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81380766c0 0x7f81381121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.787 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.784+0000 7f813d400640 1 --2- 172.21.15.103:0/2497262591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8138075da0 0x7f8138076180 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.784+0000 7f813d400640 1 -- 172.21.15.103:0/2497262591 >> 172.21.15.103:0/2497262591 conn(0x7f81380ff520 msgr2=0x7f8138101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:17.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.784+0000 7f813d400640 1 -- 172.21.15.103:0/2497262591 shutdown_connections 2024-05-23T00:02:17.788 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.784+0000 7f813d400640 1 -- 172.21.15.103:0/2497262591 wait complete. 2024-05-23T00:02:17.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.785+0000 7f813d400640 1 Processor -- start 2024-05-23T00:02:17.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.785+0000 7f813d400640 1 -- start start 2024-05-23T00:02:17.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.785+0000 7f813d400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8138075da0 0x7f81381a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.785+0000 7f813d400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81380766c0 0x7f81381a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.785+0000 7f813d400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81381126f0 0x7f813819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.785+0000 7f813d400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8138118290 con 0x7f81381126f0 2024-05-23T00:02:17.790 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.785+0000 7f813d400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8138118110 con 0x7f8138075da0 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.785+0000 7f813d400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8138118410 con 0x7f81380766c0 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f8137200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81381126f0 0x7f813819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f812fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81380766c0 0x7f81381a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f8136800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8138075da0 0x7f81381a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f812fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81380766c0 0x7f81381a4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:50468/0 (socket says 172.21.15.103:50468) 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f8137200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81381126f0 0x7f813819e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41330/0 (socket says 172.21.15.103:41330) 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f812fe00640 1 -- 172.21.15.103:0/883351340 learned_addr learned my addr 172.21.15.103:0/883351340 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f8137200640 1 -- 172.21.15.103:0/883351340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81380766c0 msgr2=0x7f81381a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f8137200640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81380766c0 0x7f81381a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f8137200640 1 -- 172.21.15.103:0/883351340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8138075da0 msgr2=0x7f81381a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f8137200640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8138075da0 0x7f81381a4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:17.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f8137200640 1 -- 172.21.15.103:0/883351340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f813819edd0 con 0x7f81381126f0 2024-05-23T00:02:17.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.786+0000 7f812fe00640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81380766c0 0x7f81381a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:17.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.787+0000 7f8137200640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81381126f0 0x7f813819e650 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8130005fd0 tx=0x7f813000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:17.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.787+0000 7f812f400640 1 -- 172.21.15.103:0/883351340 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f813000cdf0 con 0x7f81381126f0 2024-05-23T00:02:17.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.787+0000 7f812f400640 1 -- 172.21.15.103:0/883351340 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8130005020 con 0x7f81381126f0 2024-05-23T00:02:17.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.787+0000 7f812f400640 1 -- 172.21.15.103:0/883351340 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81300106d0 con 0x7f81381126f0 2024-05-23T00:02:17.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.787+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f813819f0c0 con 0x7f81381126f0 2024-05-23T00:02:17.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.787+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81381ab310 con 0x7f81381126f0 2024-05-23T00:02:17.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.788+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8138068690 con 0x7f81381126f0 2024-05-23T00:02:17.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.790+0000 7f812f400640 1 -- 172.21.15.103:0/883351340 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f81300054d0 con 0x7f81381126f0 2024-05-23T00:02:17.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.791+0000 7f812f400640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f811c0a2d00 0x7f811c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:17.794 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.792+0000 7f812f400640 1 -- 172.21.15.103:0/883351340 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f81300cff50 con 0x7f81381126f0 2024-05-23T00:02:17.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.792+0000 7f8136800640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f811c0a2d00 0x7f811c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:17.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.793+0000 7f8136800640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f811c0a2d00 0x7f811c0a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8128006010 tx=0x7f8128000bc0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:17.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:17.797+0000 7f812f400640 1 -- 172.21.15.103:0/883351340 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f813009c720 con 0x7f81381126f0 2024-05-23T00:02:18.043 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.040+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f813810d880 con 0x7f81381126f0 2024-05-23T00:02:18.044 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.041+0000 7f812f400640 1 -- 172.21.15.103:0/883351340 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11899 (secure 0 0 0) 0x7f813009c020 con 0x7f81381126f0 2024-05-23T00:02:18.045 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:02:18.045 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-23T00:01:52.287226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":71,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24575,24559,24563]},"id":1}]} 2024-05-23T00:02:18.045 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 28 2024-05-23T00:02:18.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f811c0a2d00 msgr2=0x7f811c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.047 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f811c0a2d00 0x7f811c0a51c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8128006010 tx=0x7f8128000bc0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81381126f0 msgr2=0x7f813819e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81381126f0 0x7f813819e650 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8130005fd0 tx=0x7f813000ef10 comp rx=0 tx=0).stop 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 shutdown_connections 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f811c0a2d00 0x7f811c0a51c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81381126f0 0x7f813819e650 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81380766c0 0x7f81381a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 --2- 172.21.15.103:0/883351340 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8138075da0 0x7f81381a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 >> 172.21.15.103:0/883351340 conn(0x7f81380ff520 msgr2=0x7f8138113db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 shutdown_connections 2024-05-23T00:02:18.048 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.044+0000 7f813d400640 1 -- 172.21.15.103:0/883351340 wait complete. 2024-05-23T00:02:18.068 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-23T00:02:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.277+0000 7f9d22000640 1 -- 172.21.15.103:0/3901403762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c109210 msgr2=0x7f9d1c1150c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.277+0000 7f9d22000640 1 --2- 172.21.15.103:0/3901403762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c109210 0x7f9d1c1150c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9d14009ca0 tx=0x7f9d1402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.278+0000 7f9d22000640 1 -- 172.21.15.103:0/3901403762 shutdown_connections 2024-05-23T00:02:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.278+0000 7f9d22000640 1 --2- 172.21.15.103:0/3901403762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c109210 0x7f9d1c1150c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.278+0000 7f9d22000640 1 --2- 172.21.15.103:0/3901403762 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d1c108870 0x7f9d1c108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.278+0000 7f9d22000640 1 --2- 172.21.15.103:0/3901403762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d1c10d250 0x7f9d1c10d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.278+0000 7f9d22000640 1 -- 172.21.15.103:0/3901403762 >> 172.21.15.103:0/3901403762 conn(0x7f9d1c06cde0 msgr2=0x7f9d1c06d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.278+0000 7f9d22000640 1 -- 172.21.15.103:0/3901403762 shutdown_connections 2024-05-23T00:02:18.282 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.278+0000 7f9d22000640 1 -- 172.21.15.103:0/3901403762 wait complete. 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.279+0000 7f9d22000640 1 Processor -- start 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.279+0000 7f9d22000640 1 -- start start 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.279+0000 7f9d22000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d1c108870 0x7f9d1c1a66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d22000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c109210 0x7f9d1c1a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d22000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d1c10d250 0x7f9d1c1a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d22000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c11a500 con 0x7f9d1c108870 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d22000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c11a380 con 0x7f9d1c10d250 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d22000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c11a680 con 0x7f9d1c109210 2024-05-23T00:02:18.284 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d1c108870 0x7f9d1c1a66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d1c108870 0x7f9d1c1a66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41344/0 (socket says 172.21.15.103:41344) 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1b400640 1 -- 172.21.15.103:0/1189882377 learned_addr learned my addr 172.21.15.103:0/1189882377 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1be00640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d1c10d250 0x7f9d1c1a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d13e00640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c109210 0x7f9d1c1a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1b400640 1 -- 172.21.15.103:0/1189882377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c109210 msgr2=0x7f9d1c1a6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1b400640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c109210 0x7f9d1c1a6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1b400640 1 -- 172.21.15.103:0/1189882377 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d1c10d250 msgr2=0x7f9d1c1a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1b400640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d1c10d250 0x7f9d1c1a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.280+0000 7f9d1b400640 1 -- 172.21.15.103:0/1189882377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1c1a1020 con 0x7f9d1c108870 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.281+0000 7f9d1b400640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d1c108870 0x7f9d1c1a66a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00e9f0 tx=0x7f9d0c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.281+0000 7f9d18c00640 1 -- 172.21.15.103:0/1189882377 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00cd80 con 0x7f9d1c108870 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.281+0000 7f9d22000640 1 -- 172.21.15.103:0/1189882377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1c1a1310 con 0x7f9d1c108870 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.281+0000 7f9d18c00640 1 -- 172.21.15.103:0/1189882377 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c010910 con 0x7f9d1c108870 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.281+0000 7f9d18c00640 1 -- 172.21.15.103:0/1189882377 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0188f0 con 0x7f9d1c108870 2024-05-23T00:02:18.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.281+0000 7f9d22000640 1 -- 172.21.15.103:0/1189882377 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1c1ad620 con 0x7f9d1c108870 2024-05-23T00:02:18.287 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.284+0000 7f9d24f91740 1 Processor -- start 2024-05-23T00:02:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.284+0000 7f9d24f91740 1 -- start start 2024-05-23T00:02:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.284+0000 7f9d18c00640 1 -- 172.21.15.103:0/1189882377 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f9d0c010ac0 con 0x7f9d1c108870 2024-05-23T00:02:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbef40 0x564636bbf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbf8f0 0x564636bc9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 0x564636bcc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d24f91740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564636b23b20 con 0x564636bc9f60 2024-05-23T00:02:18.288 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d24f91740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564636acb1b0 con 0x564636bbef40 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d24f91740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564636b1fae0 con 0x564636bbf8f0 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbef40 0x564636bbf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbef40 0x564636bbf320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:44342/0 (socket says 172.21.15.103:44342) 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1be00640 1 -- 172.21.15.103:0/3805739310 learned_addr learned my addr 172.21.15.103:0/3805739310 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1b400640 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbf8f0 0x564636bc9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d13e00640 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 0x564636bcc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1be00640 1 -- 172.21.15.103:0/3805739310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbf8f0 msgr2=0x564636bc9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1be00640 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbf8f0 0x564636bc9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1be00640 1 -- 172.21.15.103:0/3805739310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 msgr2=0x564636bcc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1be00640 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 0x564636bcc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.289 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.285+0000 7f9d1be00640 1 -- 172.21.15.103:0/3805739310 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564636b87610 con 0x564636bbef40 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.286+0000 7f9d1be00640 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbef40 0x564636bbf320 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9d14009ca0 tx=0x7f9d14005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1cbaaa61d361024 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.286+0000 7f9d0be00640 1 -- 172.21.15.103:0/3805739310 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14031db0 con 0x564636bbef40 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.286+0000 7f9d0be00640 1 -- 172.21.15.103:0/3805739310 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d14036040 con 0x564636bbef40 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.286+0000 7f9d0be00640 1 -- 172.21.15.103:0/3805739310 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14004330 con 0x564636bbef40 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.286+0000 7f9d18c00640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9cfc0a2d00 0x7f9cfc0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.286+0000 7f9d18c00640 1 -- 172.21.15.103:0/1189882377 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9d0c0cfc50 con 0x7f9d1c108870 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d1b400640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9cfc0a2d00 0x7f9cfc0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 -- 172.21.15.103:0/3805739310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbef40 msgr2=0x564636bbf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbef40 0x564636bbf320 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9d14009ca0 tx=0x7f9d14005f10 comp rx=0 tx=0).stop 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 -- 172.21.15.103:0/3805739310 shutdown_connections 2024-05-23T00:02:18.292 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 0x564636bcc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbf8f0 0x564636bc9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 --2- 172.21.15.103:0/3805739310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbef40 0x564636bbf320 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 -- 172.21.15.103:0/3805739310 >> 172.21.15.103:0/3805739310 conn(0x564636bbb200 msgr2=0x564636bbb5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 -- 172.21.15.103:0/3805739310 shutdown_connections 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d24f91740 1 -- 172.21.15.103:0/3805739310 wait complete. 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.287+0000 7f9d1b400640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9cfc0a2d00 0x7f9cfc0a51c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x564636bc0500 tx=0x7f9d0c0d3430 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 1 -- start start 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 1 Processor -- start 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbef40 0x564636bc6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bc7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 0x564636bc7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564636bcccf0 con 0x564636bc9f60 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x564636ade600 con 0x564636bbf8f0 2024-05-23T00:02:18.293 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x564636bcce70 con 0x564636bbef40 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.288+0000 7f9d24f91740 10 client.0 _ll_register_callbacks cb 0x564636b19160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d1be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbef40 0x564636bc6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d1be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 0x564636bc7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bc7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bc7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:44352/0 (socket says 172.21.15.103:44352) 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 -- 172.21.15.103:0/4241965860 learned_addr learned my addr 172.21.15.103:0/4241965860 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 -- 172.21.15.103:0/4241965860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbef40 msgr2=0x564636bc6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbef40 0x564636bc6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 -- 172.21.15.103:0/4241965860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 msgr2=0x564636bc7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 0x564636bc7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.294 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 -- 172.21.15.103:0/4241965860 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564636aeba90 con 0x564636bbf8f0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d13e00640 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bc7400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9d040126d0 tx=0x7f9d04012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8119613bce19ddaa server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.289+0000 7f9d08c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.290+0000 7f9d08c00640 1 -- 172.21.15.103:0/4241965860 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04004050 con 0x564636bbf8f0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.290+0000 7f9d24f91740 1 -- 172.21.15.103:0/4241965860 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d14036040 con 0x564636bbf8f0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.290+0000 7f9d08c00640 1 -- 172.21.15.103:0/4241965860 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9d040041f0 con 0x564636bbf8f0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.290+0000 7f9d08c00640 1 -- 172.21.15.103:0/4241965860 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9d04025050 con 0x564636bbf8f0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.290+0000 7f9d24f91740 10 client.26018 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.290+0000 7f9d24f91740 1 -- 172.21.15.103:0/4241965860 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564636ad30d0 con 0x564636bbf8f0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.290+0000 7f9d08c00640 1 -- 172.21.15.103:0/4241965860 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d14036040 con 0x564636bbf8f0 2024-05-23T00:02:18.297 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.290+0000 7f9d08c00640 1 -- 172.21.15.103:0/4241965860 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9d04003a30 con 0x564636bbf8f0 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.291+0000 7f9d24f91740 10 client.26018 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.291+0000 7f9d24f91740 10 client.26018 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.291+0000 7f9d24f91740 20 client.26018 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.291+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x564636bde400 0x564636be0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.291+0000 7f9d24f91740 4 client.26018 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.291+0000 7f9d24f91740 1 -- 172.21.15.103:0/4241965860 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564636b31250 con 0x564636bde400 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.291+0000 7f9d1be00640 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x564636bde400 0x564636be0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.292+0000 7f9d1be00640 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x564636bde400 0x564636be0820 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=593ad069e3a7dec4 server_cookie=12f6ac54357d6308 in_seq=0 out_seq=0 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.292+0000 7f9d08c00640 10 client.26018 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.292+0000 7f9d08c00640 1 -- 172.21.15.103:0/4241965860 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564636b31250 con 0x564636bde400 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.292+0000 7f9d08c00640 10 client.26018 handle_command_reply: tid=0 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.292+0000 7f9d24f91740 1 client.26018 shutdown 2024-05-23T00:02:18.298 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 -- 172.21.15.103:0/4241965860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 msgr2=0x564636bc7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bc7400 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9d040126d0 tx=0x7f9d04012b90 comp rx=0 tx=0).stop 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 -- 172.21.15.103:0/4241965860 shutdown_connections 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x564636bde400 0x564636be0820 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bc9f60 0x564636bc7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bc7400 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d08c00640 0 client.26018 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4241965860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bbef40 0x564636bc6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 -- 172.21.15.103:0/4241965860 >> 172.21.15.103:0/4241965860 conn(0x564636bbb200 msgr2=0x564636bc1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 -- 172.21.15.103:0/4241965860 shutdown_connections 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 1 -- 172.21.15.103:0/4241965860 wait complete. 2024-05-23T00:02:18.299 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.293+0000 7f9d24f91740 20 client.26018 trim_cache size 0 max 16384 2024-05-23T00:02:18.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.308+0000 7f9d24f91740 1 Processor -- start 2024-05-23T00:02:18.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.308+0000 7f9d24f91740 1 -- start start 2024-05-23T00:02:18.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.308+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bda040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.311 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.308+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bbb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.308+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 0x564636be6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.308+0000 7f9d24f91740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b7cb0 con 0x564636bbef40 2024-05-23T00:02:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.308+0000 7f9d24f91740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b5580 con 0x564636bbf8f0 2024-05-23T00:02:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.308+0000 7f9d24f91740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b33d0 con 0x564636bc9f60 2024-05-23T00:02:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bda040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bda040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41362/0 (socket says 172.21.15.103:41362) 2024-05-23T00:02:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 -- 172.21.15.103:0/4090227898 learned_addr learned my addr 172.21.15.103:0/4090227898 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.312 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d13e00640 1 --2- 172.21.15.103:0/4090227898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bbb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.313 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 -- 172.21.15.103:0/4090227898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 msgr2=0x564636be6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 --2- 172.21.15.103:0/4090227898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 0x564636be6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 -- 172.21.15.103:0/4090227898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 msgr2=0x564636bbb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 --2- 172.21.15.103:0/4090227898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bbb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 -- 172.21.15.103:0/4090227898 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564636bc5e70 con 0x564636bbef40 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d1be00640 1 --2- 172.21.15.103:0/4090227898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bda040 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9d14006010 tx=0x7f9d14004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91807138491304a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.309+0000 7f9d0b400640 1 -- 172.21.15.103:0/4090227898 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14031c70 con 0x564636bbef40 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d0b400640 1 -- 172.21.15.103:0/4090227898 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d14036070 con 0x564636bbef40 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d0b400640 1 -- 172.21.15.103:0/4090227898 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d14058550 con 0x564636bbef40 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 -- 172.21.15.103:0/4090227898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 msgr2=0x564636bda040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4090227898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bda040 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9d14006010 tx=0x7f9d14004a80 comp rx=0 tx=0).stop 2024-05-23T00:02:18.314 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 -- 172.21.15.103:0/4090227898 shutdown_connections 2024-05-23T00:02:18.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4090227898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 0x564636be6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4090227898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bbb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 --2- 172.21.15.103:0/4090227898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bda040 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 -- 172.21.15.103:0/4090227898 >> 172.21.15.103:0/4090227898 conn(0x564636bbb200 msgr2=0x564636bba4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 -- 172.21.15.103:0/4090227898 shutdown_connections 2024-05-23T00:02:18.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.310+0000 7f9d24f91740 1 -- 172.21.15.103:0/4090227898 wait complete. 2024-05-23T00:02:18.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 1 -- start start 2024-05-23T00:02:18.315 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 1 Processor -- start 2024-05-23T00:02:18.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bd6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bd6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 0x564636bd2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b3a40 con 0x564636bbef40 2024-05-23T00:02:18.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b4eb0 con 0x564636bbf8f0 2024-05-23T00:02:18.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c1b4110 con 0x564636bc9f60 2024-05-23T00:02:18.318 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.311+0000 7f9d24f91740 10 client.0 _ll_register_callbacks cb 0x7f9d040097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d1be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bd6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d13e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bd6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d1be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bd6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41364/0 (socket says 172.21.15.103:41364) 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d1be00640 1 -- 172.21.15.103:0/1400459212 learned_addr learned my addr 172.21.15.103:0/1400459212 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d1be00640 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 0x564636bd2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d13e00640 1 -- 172.21.15.103:0/1400459212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 msgr2=0x564636bd2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d13e00640 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 0x564636bd2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d13e00640 1 -- 172.21.15.103:0/1400459212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 msgr2=0x564636bd6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d13e00640 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bd6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d13e00640 1 -- 172.21.15.103:0/1400459212 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646368373f0 con 0x564636bbf8f0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d13e00640 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bd6a60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9d04000c00 tx=0x7f9d040106b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14bba6e604a64767 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d08c00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d08c00640 1 -- 172.21.15.103:0/1400459212 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d04009d30 con 0x564636bbf8f0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.312+0000 7f9d24f91740 1 -- 172.21.15.103:0/1400459212 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d14036070 con 0x564636bbf8f0 2024-05-23T00:02:18.319 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d08c00640 1 -- 172.21.15.103:0/1400459212 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9d04022030 con 0x564636bbf8f0 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d08c00640 1 -- 172.21.15.103:0/1400459212 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9d04010b30 con 0x564636bbf8f0 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d24f91740 10 client.26024 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d24f91740 1 -- 172.21.15.103:0/1400459212 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5646369017d0 con 0x564636bbf8f0 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d08c00640 1 -- 172.21.15.103:0/1400459212 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d14036070 con 0x564636bbf8f0 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d08c00640 1 -- 172.21.15.103:0/1400459212 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f9d040036f0 con 0x564636bbf8f0 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d24f91740 10 client.26024 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d24f91740 10 client.26024 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d24f91740 20 client.26024 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d24f91740 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x564636c459d0 0x564636c47df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d24f91740 4 client.26024 mds_command: new command op to 24559 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.313+0000 7f9d24f91740 1 -- 172.21.15.103:0/1400459212 --> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564636b2b5a0 con 0x564636c459d0 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.314+0000 7f9d1be00640 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x564636c459d0 0x564636c47df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.314+0000 7f9d1be00640 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x564636c459d0 0x564636c47df0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a4097257c040893e server_cookie=adf8678d66ce170b in_seq=0 out_seq=0 2024-05-23T00:02:18.320 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.314+0000 7f9d08c00640 10 client.26024 ms_handle_connect on v2:172.21.15.174:6838/3021454884 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.314+0000 7f9d08c00640 1 -- 172.21.15.103:0/1400459212 <== mds.2 v2:172.21.15.174:6838/3021454884 1 ==== command_reply(tid 0: 0 ) ==== 8+0+512 (crc 0 0 0) 0x564636b2b5a0 con 0x564636c459d0 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.314+0000 7f9d08c00640 10 client.26024 handle_command_reply: tid=0 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.314+0000 7f9d24f91740 1 client.26024 shutdown 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.314+0000 7f9d24f91740 1 -- 172.21.15.103:0/1400459212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 msgr2=0x564636bd6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.314+0000 7f9d24f91740 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bd6a60 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9d04000c00 tx=0x7f9d040106b0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 1 -- 172.21.15.103:0/1400459212 shutdown_connections 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.174:6838/3021454884,v1:172.21.15.174:6839/3021454884] conn(0x564636c459d0 0x564636c47df0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x564636bc9f60 0x564636bd2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564636bbf8f0 0x564636bd6a60 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 1 --2- 172.21.15.103:0/1400459212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564636bbef40 0x564636bd6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d08c00640 0 client.26024 ms_handle_reset on v2:172.21.15.174:6838/3021454884 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 1 -- 172.21.15.103:0/1400459212 >> 172.21.15.103:0/1400459212 conn(0x564636bbb200 msgr2=0x564636bba4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 1 -- 172.21.15.103:0/1400459212 shutdown_connections 2024-05-23T00:02:18.321 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 1 -- 172.21.15.103:0/1400459212 wait complete. 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d24f91740 20 client.26024 trim_cache size 0 max 16384 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stdout:[ 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: "damage_type": "backtrace", 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 1273094960, 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 1099511627781, 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: "path": "/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4/client.0" 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:02:18.322 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: "damage_type": "backtrace", 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 1969116952, 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 1099511628284, 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: "path": "/volumes/qa/sv_1/.meta" 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: "damage_type": "backtrace", 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: "id": 2049093189, 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 1099511628283, 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: "path": "/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4" 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:02:18.323 INFO:teuthology.orchestra.run.smithi103.stdout:] 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d22000640 1 -- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9cfc0a2d00 msgr2=0x7f9cfc0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d22000640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9cfc0a2d00 0x7f9cfc0a51c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x564636bc0500 tx=0x7f9d0c0d3430 comp rx=0 tx=0).stop 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d22000640 1 -- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d1c108870 msgr2=0x7f9d1c1a66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.315+0000 7f9d22000640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d1c108870 0x7f9d1c1a66a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00e9f0 tx=0x7f9d0c00eeb0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.316+0000 7f9d22000640 1 -- 172.21.15.103:0/1189882377 shutdown_connections 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.316+0000 7f9d22000640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9cfc0a2d00 0x7f9cfc0a51c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.316+0000 7f9d22000640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d1c10d250 0x7f9d1c1a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.316+0000 7f9d22000640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9d1c109210 0x7f9d1c1a6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.316+0000 7f9d22000640 1 --2- 172.21.15.103:0/1189882377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d1c108870 0x7f9d1c1a66a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.324 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.316+0000 7f9d22000640 1 -- 172.21.15.103:0/1189882377 >> 172.21.15.103:0/1189882377 conn(0x7f9d1c06cde0 msgr2=0x7f9d1c071110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.325 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.316+0000 7f9d22000640 1 -- 172.21.15.103:0/1189882377 shutdown_connections 2024-05-23T00:02:18.325 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.316+0000 7f9d22000640 1 -- 172.21.15.103:0/1189882377 wait complete. 2024-05-23T00:02:18.336 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-23T00:02:18.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 -- 172.21.15.103:0/4215902729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208100750 msgr2=0x7fe208100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 --2- 172.21.15.103:0/4215902729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208100750 0x7fe208100bd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0009890 tx=0x7fe1f002a370 comp rx=0 tx=0).stop 2024-05-23T00:02:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 -- 172.21.15.103:0/4215902729 shutdown_connections 2024-05-23T00:02:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 --2- 172.21.15.103:0/4215902729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe208101110 0x7fe20810d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 --2- 172.21.15.103:0/4215902729 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208100750 0x7fe208100bd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 --2- 172.21.15.103:0/4215902729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe208106760 0x7fe208106b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 -- 172.21.15.103:0/4215902729 >> 172.21.15.103:0/4215902729 conn(0x7fe2080777d0 msgr2=0x7fe208077be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 -- 172.21.15.103:0/4215902729 shutdown_connections 2024-05-23T00:02:18.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.528+0000 7fe20ce00640 1 -- 172.21.15.103:0/4215902729 wait complete. 2024-05-23T00:02:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.529+0000 7fe20ce00640 1 Processor -- start 2024-05-23T00:02:18.534 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.529+0000 7fe20ce00640 1 -- start start 2024-05-23T00:02:18.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe20ce00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe208100750 0x7fe20819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe20ce00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe208101110 0x7fe20819a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe20ce00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208106760 0x7fe20819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe20ce00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe208110b70 con 0x7fe208100750 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe20ce00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081109f0 con 0x7fe208101110 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe20ce00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe208110cf0 con 0x7fe208106760 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe205e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe208100750 0x7fe20819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe206800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208106760 0x7fe20819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe206800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208106760 0x7fe20819ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:50546/0 (socket says 172.21.15.103:50546) 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe206800640 1 -- 172.21.15.103:0/1812738785 learned_addr learned my addr 172.21.15.103:0/1812738785 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.530+0000 7fe205400640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe208101110 0x7fe20819a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe206800640 1 -- 172.21.15.103:0/1812738785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe208101110 msgr2=0x7fe20819a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe206800640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe208101110 0x7fe20819a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe206800640 1 -- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe208100750 msgr2=0x7fe20819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe206800640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe208100750 0x7fe20819a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe206800640 1 -- 172.21.15.103:0/1812738785 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe20819f3b0 con 0x7fe208106760 2024-05-23T00:02:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe206800640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208106760 0x7fe20819ec80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe20000efc0 tx=0x7fe20000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe1fea00640 1 -- 172.21.15.103:0/1812738785 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe200003970 con 0x7fe208106760 2024-05-23T00:02:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe20ce00640 1 -- 172.21.15.103:0/1812738785 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe20819f6a0 con 0x7fe208106760 2024-05-23T00:02:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe1fea00640 1 -- 172.21.15.103:0/1812738785 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe200004020 con 0x7fe208106760 2024-05-23T00:02:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe1fea00640 1 -- 172.21.15.103:0/1812738785 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe20000b9f0 con 0x7fe208106760 2024-05-23T00:02:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.531+0000 7fe20ce00640 1 -- 172.21.15.103:0/1812738785 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2081a6f30 con 0x7fe208106760 2024-05-23T00:02:18.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.536+0000 7fe20fbed740 1 Processor -- start 2024-05-23T00:02:18.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.536+0000 7fe1fea00640 1 -- 172.21.15.103:0/1812738785 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fe20001d020 con 0x7fe208106760 2024-05-23T00:02:18.539 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.536+0000 7fe20fbed740 1 -- start start 2024-05-23T00:02:18.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783cb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c5783d5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe20fbed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c57832fb20 con 0x55c5783d5f60 2024-05-23T00:02:18.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe20fbed740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c5782d71b0 con 0x55c5783cb8f0 2024-05-23T00:02:18.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe20fbed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c57832bae0 con 0x55c5783caf40 2024-05-23T00:02:18.540 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe205e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c5783d5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe205400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe205e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c5783d5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:44404/0 (socket says 172.21.15.103:44404) 2024-05-23T00:02:18.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe205400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41396/0 (socket says 172.21.15.103:41396) 2024-05-23T00:02:18.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe205e00640 1 -- 172.21.15.103:0/1515356219 learned_addr learned my addr 172.21.15.103:0/1515356219 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.541 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe205400640 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe205e00640 1 -- 172.21.15.103:0/1515356219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 msgr2=0x55c5783cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.537+0000 7fe205e00640 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783cb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe205e00640 1 -- 172.21.15.103:0/1515356219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 msgr2=0x55c5783d83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe205e00640 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe205e00640 1 -- 172.21.15.103:0/1515356219 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c578393610 con 0x55c5783cb8f0 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe205400640 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe205e00640 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c5783d5990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009ca0 tx=0x7fe1f8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c5bc0ca662f21d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe1f7400640 1 -- 172.21.15.103:0/1515356219 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f801e070 con 0x55c5783cb8f0 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe1f7400640 1 -- 172.21.15.103:0/1515356219 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f8019b10 con 0x55c5783cb8f0 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe1f7400640 1 -- 172.21.15.103:0/1515356219 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f800e9f0 con 0x55c5783cb8f0 2024-05-23T00:02:18.543 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.538+0000 7fe1fea00640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe1e80a2d00 0x7fe1e80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe1fea00640 1 -- 172.21.15.103:0/1812738785 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fe2000cf710 con 0x7fe208106760 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe205400640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe1e80a2d00 0x7fe1e80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 -- 172.21.15.103:0/1515356219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 msgr2=0x55c5783d5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c5783d5990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009ca0 tx=0x7fe1f8011810 comp rx=0 tx=0).stop 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 -- 172.21.15.103:0/1515356219 shutdown_connections 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c5783d5990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1515356219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783cb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 -- 172.21.15.103:0/1515356219 >> 172.21.15.103:0/1515356219 conn(0x55c5783c7200 msgr2=0x55c5783c75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 -- 172.21.15.103:0/1515356219 shutdown_connections 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe20fbed740 1 -- 172.21.15.103:0/1515356219 wait complete. 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.539+0000 7fe205400640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe1e80a2d00 0x7fe1e80a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00097b0 tx=0x7fe1f002c040 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 1 -- start start 2024-05-23T00:02:18.544 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 1 Processor -- start 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783d2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d33d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783d3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c5783d8ed0 con 0x55c5783cb8f0 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55c5782ea600 con 0x55c5783d5f60 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55c5783d9050 con 0x55c5783caf40 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.540+0000 7fe20fbed740 10 client.0 _ll_register_callbacks cb 0x55c578325160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783d2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.548 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783d2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:50554/0 (socket says 172.21.15.103:50554) 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 -- 172.21.15.103:0/1639952738 learned_addr learned my addr 172.21.15.103:0/1639952738 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe206800640 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 -- 172.21.15.103:0/1639952738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 msgr2=0x55c5783d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 -- 172.21.15.103:0/1639952738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 msgr2=0x55c5783d33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 -- 172.21.15.103:0/1639952738 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5782f7a90 con 0x55c5783caf40 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe206800640 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe205e00640 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783d2e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009f50 tx=0x7fe1f800e3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95e5ed2bfcd67805 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.541+0000 7fe1efe00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:02:18.549 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.542+0000 7fe1efe00640 1 -- 172.21.15.103:0/1639952738 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f8027070 con 0x55c5783caf40 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.542+0000 7fe20fbed740 1 -- 172.21.15.103:0/1639952738 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1f8019b10 con 0x55c5783caf40 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.542+0000 7fe1efe00640 1 -- 172.21.15.103:0/1639952738 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fe1f8019840 con 0x55c5783caf40 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.542+0000 7fe1efe00640 1 -- 172.21.15.103:0/1639952738 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe1f8032050 con 0x55c5783caf40 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.542+0000 7fe20fbed740 10 client.26041 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.542+0000 7fe20fbed740 1 -- 172.21.15.103:0/1639952738 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c5782df0d0 con 0x55c5783caf40 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.542+0000 7fe1efe00640 1 -- 172.21.15.103:0/1639952738 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f8019b10 con 0x55c5783caf40 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.542+0000 7fe1efe00640 1 -- 172.21.15.103:0/1639952738 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe1f80046c0 con 0x55c5783caf40 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe20fbed740 10 client.26041 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe20fbed740 10 client.26041 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe20fbed740 20 client.26041 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55c5783e6010 0x55c5783e8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe20fbed740 4 client.26041 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:18.550 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe20fbed740 1 -- 172.21.15.103:0/1639952738 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c57833d250 con 0x55c5783e6010 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe205400640 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55c5783e6010 0x55c5783e8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe1efe00640 10 client.26041 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.543+0000 7fe205400640 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55c5783e6010 0x55c5783e8400 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fd065deb65e9fb0 server_cookie=c91bf6cdf1642053 in_seq=0 out_seq=0 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.544+0000 7fe1efe00640 1 -- 172.21.15.103:0/1639952738 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c57833d250 con 0x55c5783e6010 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.544+0000 7fe1efe00640 10 client.26041 handle_command_reply: tid=0 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.544+0000 7fe20fbed740 1 client.26041 shutdown 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.544+0000 7fe20fbed740 1 -- 172.21.15.103:0/1639952738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 msgr2=0x55c5783d2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.544+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783d2e90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8009f50 tx=0x7fe1f800e3b0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 1 -- 172.21.15.103:0/1639952738 shutdown_connections 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55c5783e6010 0x55c5783e8400 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 1 --2- 172.21.15.103:0/1639952738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783d2e90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.551 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe1efe00640 0 client.26041 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-23T00:02:18.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 1 -- 172.21.15.103:0/1639952738 >> 172.21.15.103:0/1639952738 conn(0x55c5783c7200 msgr2=0x55c5783cd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 1 -- 172.21.15.103:0/1639952738 shutdown_connections 2024-05-23T00:02:18.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 1 -- 172.21.15.103:0/1639952738 wait complete. 2024-05-23T00:02:18.552 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.545+0000 7fe20fbed740 20 client.26041 trim_cache size 0 max 16384 2024-05-23T00:02:18.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe20fbed740 1 Processor -- start 2024-05-23T00:02:18.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe20fbed740 1 -- start start 2024-05-23T00:02:18.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe20fbed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081b3950 con 0x55c5783cb8f0 2024-05-23T00:02:18.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe20fbed740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081b1220 con 0x55c5783d5f60 2024-05-23T00:02:18.563 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe20fbed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081af070 con 0x55c5783caf40 2024-05-23T00:02:18.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe205400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe205e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.564 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe205400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41412/0 (socket says 172.21.15.103:41412) 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.560+0000 7fe205400640 1 -- 172.21.15.103:0/912817323 learned_addr learned my addr 172.21.15.103:0/912817323 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe206800640 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe205400640 1 -- 172.21.15.103:0/912817323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 msgr2=0x55c5783c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe205400640 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783c5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe205400640 1 -- 172.21.15.103:0/912817323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 msgr2=0x55c5783cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe205400640 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783cd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe205400640 1 -- 172.21.15.103:0/912817323 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5780433f0 con 0x55c5783cb8f0 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe205e00640 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe206800640 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe205400640 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d2640 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0005e00 tx=0x7fe1f0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9913abf350e6984 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe1f6a00640 1 -- 172.21.15.103:0/912817323 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0058070 con 0x55c5783cb8f0 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe1f6a00640 1 -- 172.21.15.103:0/912817323 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f0051030 con 0x55c5783cb8f0 2024-05-23T00:02:18.565 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.561+0000 7fe1f6a00640 1 -- 172.21.15.103:0/912817323 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0047040 con 0x55c5783cb8f0 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 -- 172.21.15.103:0/912817323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 msgr2=0x55c5783d2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d2640 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe1f0005e00 tx=0x7fe1f0005f90 comp rx=0 tx=0).stop 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 -- 172.21.15.103:0/912817323 shutdown_connections 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783d5f60 0x55c5783cd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783cb8f0 0x55c5783d2640 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 --2- 172.21.15.103:0/912817323 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c5783c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 -- 172.21.15.103:0/912817323 >> 172.21.15.103:0/912817323 conn(0x55c5783c7200 msgr2=0x55c5783d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 -- 172.21.15.103:0/912817323 shutdown_connections 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.562+0000 7fe20fbed740 1 -- 172.21.15.103:0/912817323 wait complete. 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 1 -- start start 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 1 Processor -- start 2024-05-23T00:02:18.569 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c578444690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c578444bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081af6e0 con 0x55c5783d5f60 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081b0b50 con 0x55c5783cb8f0 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe2081afdb0 con 0x55c5783caf40 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.563+0000 7fe20fbed740 10 client.0 _ll_register_callbacks cb 0x7fe1f801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41424/0 (socket says 172.21.15.103:41424) 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 -- 172.21.15.103:0/4164855709 learned_addr learned my addr 172.21.15.103:0/4164855709 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe205400640 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c578444bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe205e00640 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c578444690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 -- 172.21.15.103:0/4164855709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 msgr2=0x55c578444690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.570 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c578444690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 -- 172.21.15.103:0/4164855709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 msgr2=0x55c578444bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c578444bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 -- 172.21.15.103:0/4164855709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c57810d7d0 con 0x55c5783d5f60 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe206800640 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d94b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe20000cd00 tx=0x7fe20009b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a02011a4b259ff6 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.564+0000 7fe1efe00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe1efe00640 1 -- 172.21.15.103:0/4164855709 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2000cf040 con 0x55c5783d5f60 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe1efe00640 1 -- 172.21.15.103:0/4164855709 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fe200097e50 con 0x55c5783d5f60 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 1 -- 172.21.15.103:0/4164855709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1f0051030 con 0x55c5783d5f60 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe1efe00640 1 -- 172.21.15.103:0/4164855709 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe2000ba080 con 0x55c5783d5f60 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 10 client.15942 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 1 -- 172.21.15.103:0/4164855709 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c57844bc70 con 0x55c5783d5f60 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe1efe00640 1 -- 172.21.15.103:0/4164855709 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f0051030 con 0x55c5783d5f60 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe1efe00640 1 -- 172.21.15.103:0/4164855709 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fe20000b040 con 0x55c5783d5f60 2024-05-23T00:02:18.571 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 10 client.15942 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:18.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 10 client.15942 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-05-23T00:02:18.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 20 client.15942 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:18.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55c57844c3c0 0x55c57844e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 4 client.15942 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:02:18.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.565+0000 7fe20fbed740 1 -- 172.21.15.103:0/4164855709 --> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c5783375a0 con 0x55c57844c3c0 2024-05-23T00:02:18.572 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.566+0000 7fe205e00640 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55c57844c3c0 0x55c57844e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.566+0000 7fe205e00640 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55c57844c3c0 0x55c57844e7e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=70a29120bf439252 server_cookie=2b548b86c9d6d81f in_seq=0 out_seq=0 2024-05-23T00:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.566+0000 7fe1efe00640 10 client.15942 ms_handle_connect on v2:172.21.15.103:6840/4258722678 2024-05-23T00:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.566+0000 7fe1efe00640 1 -- 172.21.15.103:0/4164855709 <== mds.1 v2:172.21.15.103:6840/4258722678 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c5783375a0 con 0x55c57844c3c0 2024-05-23T00:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.566+0000 7fe1efe00640 10 client.15942 handle_command_reply: tid=0 2024-05-23T00:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.566+0000 7fe20fbed740 1 client.15942 shutdown 2024-05-23T00:02:18.573 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 -- 172.21.15.103:0/4164855709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 msgr2=0x55c5783d94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d94b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe20000cd00 tx=0x7fe20009b8a0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 -- 172.21.15.103:0/4164855709 shutdown_connections 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c5783d5f60 0x55c5783d94b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55c5783cb8f0 0x55c578444bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.103:6840/4258722678,v1:172.21.15.103:6841/4258722678] conn(0x55c57844c3c0 0x55c57844e7e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 --2- 172.21.15.103:0/4164855709 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55c5783caf40 0x55c578444690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 -- 172.21.15.103:0/4164855709 >> 172.21.15.103:0/4164855709 conn(0x55c5783c7200 msgr2=0x55c5783e0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe1efe00640 0 client.15942 ms_handle_reset on v2:172.21.15.103:6840/4258722678 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 -- 172.21.15.103:0/4164855709 shutdown_connections 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 1 -- 172.21.15.103:0/4164855709 wait complete. 2024-05-23T00:02:18.574 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.567+0000 7fe20fbed740 20 client.15942 trim_cache size 0 max 16384 2024-05-23T00:02:18.575 INFO:teuthology.orchestra.run.smithi103.stdout:[] 2024-05-23T00:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 -- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe1e80a2d00 msgr2=0x7fe1e80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe1e80a2d00 0x7fe1e80a51c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe1f00097b0 tx=0x7fe1f002c040 comp rx=0 tx=0).stop 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 -- 172.21.15.103:0/1812738785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208106760 msgr2=0x7fe20819ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208106760 0x7fe20819ec80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe20000efc0 tx=0x7fe20000c800 comp rx=0 tx=0).stop 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 -- 172.21.15.103:0/1812738785 shutdown_connections 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe1e80a2d00 0x7fe1e80a51c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe208106760 0x7fe20819ec80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe208101110 0x7fe20819a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 --2- 172.21.15.103:0/1812738785 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe208100750 0x7fe20819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 -- 172.21.15.103:0/1812738785 >> 172.21.15.103:0/1812738785 conn(0x7fe2080777d0 msgr2=0x7fe20810bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 -- 172.21.15.103:0/1812738785 shutdown_connections 2024-05-23T00:02:18.576 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.568+0000 7fe20ce00640 1 -- 172.21.15.103:0/1812738785 wait complete. 2024-05-23T00:02:18.586 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:18 smithi103 ceph-mon[31370]: pgmap v57: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 6 op/s 2024-05-23T00:02:18.586 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:18 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/883351340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:02:18.589 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-23T00:02:18.669 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:18 smithi174 ceph-mon[36179]: pgmap v57: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 6 op/s 2024-05-23T00:02:18.669 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:18 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/883351340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:02:18.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.792+0000 7f631ca00640 1 -- 172.21.15.103:0/1751822387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6318107e10 msgr2=0x7f6318108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.792+0000 7f631ca00640 1 --2- 172.21.15.103:0/1751822387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6318107e10 0x7f6318108270 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6300009900 tx=0x7f630002a480 comp rx=0 tx=0).stop 2024-05-23T00:02:18.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.793+0000 7f631ca00640 1 -- 172.21.15.103:0/1751822387 shutdown_connections 2024-05-23T00:02:18.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.793+0000 7f631ca00640 1 --2- 172.21.15.103:0/1751822387 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63181087b0 0x7f6318114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.793+0000 7f631ca00640 1 --2- 172.21.15.103:0/1751822387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6318107e10 0x7f6318108270 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.793+0000 7f631ca00640 1 --2- 172.21.15.103:0/1751822387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f631810de30 0x7f631810e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.793+0000 7f631ca00640 1 -- 172.21.15.103:0/1751822387 >> 172.21.15.103:0/1751822387 conn(0x7f6318077690 msgr2=0x7f6318106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.793+0000 7f631ca00640 1 -- 172.21.15.103:0/1751822387 shutdown_connections 2024-05-23T00:02:18.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.793+0000 7f631ca00640 1 -- 172.21.15.103:0/1751822387 wait complete. 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f631ca00640 1 Processor -- start 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f631ca00640 1 -- start start 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f631ca00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6318107e10 0x7f631819e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f631ca00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63181087b0 0x7f631819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f631ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631810de30 0x7f63181a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f631ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6318118110 con 0x7f631810de30 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f631ca00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6318117f90 con 0x7f6318107e10 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f631ca00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6318118290 con 0x7f63181087b0 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f6316800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6318107e10 0x7f631819e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f6317200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631810de30 0x7f63181a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f6316800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6318107e10 0x7f631819e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:44452/0 (socket says 172.21.15.103:44452) 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f6316800640 1 -- 172.21.15.103:0/3535609660 learned_addr learned my addr 172.21.15.103:0/3535609660 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.794+0000 7f630de00640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63181087b0 0x7f631819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f6316800640 1 -- 172.21.15.103:0/3535609660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63181087b0 msgr2=0x7f631819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f6316800640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63181087b0 0x7f631819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f6316800640 1 -- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631810de30 msgr2=0x7f63181a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f6316800640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631810de30 0x7f63181a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f6316800640 1 -- 172.21.15.103:0/3535609660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63181a3560 con 0x7f6318107e10 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f6317200640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631810de30 0x7f63181a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f6316800640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6318107e10 0x7f631819e570 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f630800e9f0 tx=0x7f630800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f630fe00640 1 -- 172.21.15.103:0/3535609660 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f630800cd80 con 0x7f6318107e10 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f630fe00640 1 -- 172.21.15.103:0/3535609660 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6308010910 con 0x7f6318107e10 2024-05-23T00:02:18.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f631ca00640 1 -- 172.21.15.103:0/3535609660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63181a3850 con 0x7f6318107e10 2024-05-23T00:02:18.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.795+0000 7f630fe00640 1 -- 172.21.15.103:0/3535609660 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6308018990 con 0x7f6318107e10 2024-05-23T00:02:18.801 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.796+0000 7f631ca00640 1 -- 172.21.15.103:0/3535609660 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63181ab0e0 con 0x7f6318107e10 2024-05-23T00:02:18.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f63201d3740 1 Processor -- start 2024-05-23T00:02:18.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f630fe00640 1 -- 172.21.15.103:0/3535609660 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f6308010ac0 con 0x7f6318107e10 2024-05-23T00:02:18.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f63201d3740 1 -- start start 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f63201d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8644f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f63201d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e86eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f63201d3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e871580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f63201d3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55892e7c8b20 con 0x55892e86f130 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f63201d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55892e7701b0 con 0x55892e864110 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.799+0000 7f63201d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55892e7c4ae0 con 0x55892e864ac0 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8644f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8644f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:44460/0 (socket says 172.21.15.103:44460) 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 -- 172.21.15.103:0/1133395141 learned_addr learned my addr 172.21.15.103:0/1133395141 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f630de00640 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e871580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6316800640 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e86eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 -- 172.21.15.103:0/1133395141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 msgr2=0x55892e86eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e86eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 -- 172.21.15.103:0/1133395141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 msgr2=0x55892e871580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e871580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 -- 172.21.15.103:0/1133395141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55892e82c610 con 0x55892e864110 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f6317200640 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8644f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63100117f0 tx=0x7f6310011cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e67a24bb203e9f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.800+0000 7f630fe00640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f62fc0a2d80 0x7f62fc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f6307400640 1 -- 172.21.15.103:0/1133395141 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63100112f0 con 0x55892e864110 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f6307400640 1 -- 172.21.15.103:0/1133395141 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6310003e50 con 0x55892e864110 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f6307400640 1 -- 172.21.15.103:0/1133395141 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63100045f0 con 0x55892e864110 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f630fe00640 1 -- 172.21.15.103:0/3535609660 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f63080cfb10 con 0x7f6318107e10 2024-05-23T00:02:18.805 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f6316800640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f62fc0a2d80 0x7f62fc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f63201d3740 1 -- 172.21.15.103:0/1133395141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 msgr2=0x55892e8644f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f63201d3740 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8644f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f63100117f0 tx=0x7f6310011cb0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f6316800640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f62fc0a2d80 0x7f62fc0a5240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55892e8656d0 tx=0x7f6308000bc0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f63201d3740 1 -- 172.21.15.103:0/1133395141 shutdown_connections 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f63201d3740 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e871580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f63201d3740 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e86eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f63201d3740 1 --2- 172.21.15.103:0/1133395141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8644f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.801+0000 7f63201d3740 1 -- 172.21.15.103:0/1133395141 >> 172.21.15.103:0/1133395141 conn(0x55892e8603d0 msgr2=0x55892e8607b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.802+0000 7f63201d3740 1 -- 172.21.15.103:0/1133395141 shutdown_connections 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.802+0000 7f63201d3740 1 -- 172.21.15.103:0/1133395141 wait complete. 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.802+0000 7f63201d3740 1 -- start start 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.802+0000 7f63201d3740 1 Processor -- start 2024-05-23T00:02:18.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f63201d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e86bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f63201d3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e86c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f63201d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 0x55892e86ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f63201d3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55892e8720a0 con 0x55892e864ac0 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f63201d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x55892e783600 con 0x55892e864110 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f63201d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55892e872220 con 0x55892e86f130 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f63201d3740 10 client.0 _ll_register_callbacks cb 0x55892e7be160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f6317200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e86bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f630de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e86c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f6317200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e86bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:44462/0 (socket says 172.21.15.103:44462) 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f6317200640 1 -- 172.21.15.103:0/238618942 learned_addr learned my addr 172.21.15.103:0/238618942 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f6317200640 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 0x55892e86ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f630de00640 1 -- 172.21.15.103:0/238618942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 msgr2=0x55892e86ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f630de00640 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 0x55892e86ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f630de00640 1 -- 172.21.15.103:0/238618942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 msgr2=0x55892e86bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.803+0000 7f630de00640 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e86bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f630de00640 1 -- 172.21.15.103:0/238618942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55892e790a90 con 0x55892e864ac0 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f630de00640 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e86c520 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6300006010 tx=0x7f63000317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f52de6235cf4d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f62fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f62fbe00640 1 -- 172.21.15.103:0/238618942 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300035040 con 0x55892e864ac0 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f62fbe00640 1 -- 172.21.15.103:0/238618942 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f6300004250 con 0x55892e864ac0 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f63201d3740 1 -- 172.21.15.103:0/238618942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6310003e50 con 0x55892e864ac0 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f62fbe00640 1 -- 172.21.15.103:0/238618942 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6300004450 con 0x55892e864ac0 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f63201d3740 10 client.15954 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.804+0000 7f63201d3740 1 -- 172.21.15.103:0/238618942 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55892e7780d0 con 0x55892e864ac0 2024-05-23T00:02:18.812 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f62fbe00640 1 -- 172.21.15.103:0/238618942 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6310003e50 con 0x55892e864ac0 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f62fbe00640 1 -- 172.21.15.103:0/238618942 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6300004cd0 con 0x55892e864ac0 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f63201d3740 10 client.15954 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f63201d3740 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f63201d3740 20 client.15954 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f63201d3740 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55892e87f250 0x55892e8816d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f63201d3740 4 client.15954 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f63201d3740 1 -- 172.21.15.103:0/238618942 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55892e7d6250 con 0x55892e87f250 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.805+0000 7f6317200640 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55892e87f250 0x55892e8816d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.806+0000 7f6317200640 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55892e87f250 0x55892e8816d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de4e1697b825c475 server_cookie=f983e96da09895e0 in_seq=0 out_seq=0 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.806+0000 7f62fbe00640 10 client.15954 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.806+0000 7f62fbe00640 1 -- 172.21.15.103:0/238618942 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55892e7d6250 con 0x55892e87f250 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f62fbe00640 10 client.15954 handle_command_reply: tid=0 2024-05-23T00:02:18.813 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 client.15954 shutdown 2024-05-23T00:02:18.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 -- 172.21.15.103:0/238618942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 msgr2=0x55892e86c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e86c520 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6300006010 tx=0x7f63000317e0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:18 smithi105 ceph-mon[36426]: pgmap v57: 129 pgs: 129 active+clean; 1.5 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 6 op/s 2024-05-23T00:02:18.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:18 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/883351340' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:02:18.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 -- 172.21.15.103:0/238618942 shutdown_connections 2024-05-23T00:02:18.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55892e87f250 0x55892e8816d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 0x55892e86ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e86c520 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 --2- 172.21.15.103:0/238618942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e86bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.815 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 -- 172.21.15.103:0/238618942 >> 172.21.15.103:0/238618942 conn(0x55892e8603d0 msgr2=0x55892e866240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f62fbe00640 0 client.15954 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:18.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 -- 172.21.15.103:0/238618942 shutdown_connections 2024-05-23T00:02:18.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 1 -- 172.21.15.103:0/238618942 wait complete. 2024-05-23T00:02:18.816 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.807+0000 7f63201d3740 20 client.15954 trim_cache size 0 max 16384 2024-05-23T00:02:18.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.822+0000 7f63201d3740 1 Processor -- start 2024-05-23T00:02:18.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.822+0000 7f63201d3740 1 -- start start 2024-05-23T00:02:18.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.822+0000 7f63201d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e87af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.822+0000 7f63201d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e866710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.825 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.822+0000 7f63201d3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e886fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.822+0000 7f63201d3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63181b7b30 con 0x55892e86f130 2024-05-23T00:02:18.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.822+0000 7f63201d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63181b5400 con 0x55892e864110 2024-05-23T00:02:18.826 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.822+0000 7f63201d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f63181b3250 con 0x55892e864ac0 2024-05-23T00:02:18.827 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e886fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e886fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41468/0 (socket says 172.21.15.103:41468) 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 -- 172.21.15.103:0/3857186917 learned_addr learned my addr 172.21.15.103:0/3857186917 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f630de00640 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e866710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e87af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 -- 172.21.15.103:0/3857186917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 msgr2=0x55892e866710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e866710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 -- 172.21.15.103:0/3857186917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 msgr2=0x55892e87af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e87af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 -- 172.21.15.103:0/3857186917 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55892e85bb20 con 0x55892e86f130 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f630de00640 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e866710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6317200640 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e886fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6310004260 tx=0x7f631001c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af89c84c3f558b37 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.828 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6306a00640 1 -- 172.21.15.103:0/3857186917 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6310022040 con 0x55892e86f130 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.823+0000 7f6306a00640 1 -- 172.21.15.103:0/3857186917 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f631000fba0 con 0x55892e86f130 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f6306a00640 1 -- 172.21.15.103:0/3857186917 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631002f0c0 con 0x55892e86f130 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 -- 172.21.15.103:0/3857186917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 msgr2=0x55892e886fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e886fe0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6310004260 tx=0x7f631001c040 comp rx=0 tx=0).stop 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 -- 172.21.15.103:0/3857186917 shutdown_connections 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e86f130 0x55892e886fe0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e864ac0 0x55892e866710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 --2- 172.21.15.103:0/3857186917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e87af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 -- 172.21.15.103:0/3857186917 >> 172.21.15.103:0/3857186917 conn(0x55892e8603d0 msgr2=0x55892e86c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 -- 172.21.15.103:0/3857186917 shutdown_connections 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.824+0000 7f63201d3740 1 -- 172.21.15.103:0/3857186917 wait complete. 2024-05-23T00:02:18.829 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 1 -- start start 2024-05-23T00:02:18.830 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 1 Processor -- start 2024-05-23T00:02:18.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e8737f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 0x55892e8e4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63181b38c0 con 0x55892e864ac0 2024-05-23T00:02:18.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f63181b4d30 con 0x55892e864110 2024-05-23T00:02:18.833 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f63181b3f90 con 0x55892e86f130 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.825+0000 7f63201d3740 10 client.0 _ll_register_callbacks cb 0x7f6300009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e8737f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f6317200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e8737f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41476/0 (socket says 172.21.15.103:41476) 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 -- 172.21.15.103:0/1763906796 learned_addr learned my addr 172.21.15.103:0/1763906796 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f6317200640 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 0x55892e8e4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 -- 172.21.15.103:0/1763906796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 msgr2=0x55892e8e4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 0x55892e8e4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 -- 172.21.15.103:0/1763906796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 msgr2=0x55892e8732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8732b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 -- 172.21.15.103:0/1763906796 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55892e4dc3f0 con 0x55892e864ac0 2024-05-23T00:02:18.834 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f630de00640 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e8737f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f630002a9a0 tx=0x7f6300003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65a0f07abbbe67e server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.826+0000 7f62fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.827+0000 7f62fbe00640 1 -- 172.21.15.103:0/1763906796 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6300056070 con 0x55892e864ac0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.827+0000 7f62fbe00640 1 -- 172.21.15.103:0/1763906796 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f6300005040 con 0x55892e864ac0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.827+0000 7f63201d3740 1 -- 172.21.15.103:0/1763906796 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f631000fba0 con 0x55892e864ac0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.827+0000 7f62fbe00640 1 -- 172.21.15.103:0/1763906796 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f630005f050 con 0x55892e864ac0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.827+0000 7f63201d3740 10 client.15960 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.827+0000 7f63201d3740 1 -- 172.21.15.103:0/1763906796 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55892e5a67d0 con 0x55892e864ac0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.827+0000 7f62fbe00640 1 -- 172.21.15.103:0/1763906796 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f631000fba0 con 0x55892e864ac0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.828+0000 7f62fbe00640 1 -- 172.21.15.103:0/1763906796 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6300059020 con 0x55892e864ac0 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.828+0000 7f63201d3740 10 client.15960 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.828+0000 7f63201d3740 10 client.15960 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.828+0000 7f63201d3740 20 client.15960 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:18.835 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.828+0000 7f63201d3740 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55892e8e63e0 0x55892e8e8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.828+0000 7f63201d3740 4 client.15960 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.828+0000 7f63201d3740 1 -- 172.21.15.103:0/1763906796 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55892e7d05a0 con 0x55892e8e63e0 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.828+0000 7f6317200640 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55892e8e63e0 0x55892e8e8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f62fbe00640 10 client.15960 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f6317200640 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55892e8e63e0 0x55892e8e8800 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38042fe7d5ba4dd2 server_cookie=fc3f786d64ff3d9 in_seq=0 out_seq=0 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f62fbe00640 1 -- 172.21.15.103:0/1763906796 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55892e7d05a0 con 0x55892e8e63e0 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f62fbe00640 10 client.15960 handle_command_reply: tid=0 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 client.15960 shutdown 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 -- 172.21.15.103:0/1763906796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 msgr2=0x55892e8737f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e8737f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f630002a9a0 tx=0x7f6300003f00 comp rx=0 tx=0).stop 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 -- 172.21.15.103:0/1763906796 shutdown_connections 2024-05-23T00:02:18.836 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x55892e8e63e0 0x55892e8e8800 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55892e86f130 0x55892e8e4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55892e864ac0 0x55892e8737f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 --2- 172.21.15.103:0/1763906796 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55892e864110 0x55892e8732b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 -- 172.21.15.103:0/1763906796 >> 172.21.15.103:0/1763906796 conn(0x55892e8603d0 msgr2=0x55892e86d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f62fbe00640 0 client.15960 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 -- 172.21.15.103:0/1763906796 shutdown_connections 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 1 -- 172.21.15.103:0/1763906796 wait complete. 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.829+0000 7f63201d3740 20 client.15960 trim_cache size 0 max 16384 2024-05-23T00:02:18.837 INFO:teuthology.orchestra.run.smithi103.stdout:[] 2024-05-23T00:02:18.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 -- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f62fc0a2d80 msgr2=0x7f62fc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f62fc0a2d80 0x7f62fc0a5240 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x55892e8656d0 tx=0x7f6308000bc0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 -- 172.21.15.103:0/3535609660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6318107e10 msgr2=0x7f631819e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:18.838 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6318107e10 0x7f631819e570 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f630800e9f0 tx=0x7f630800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 -- 172.21.15.103:0/3535609660 shutdown_connections 2024-05-23T00:02:18.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f62fc0a2d80 0x7f62fc0a5240 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f631810de30 0x7f63181a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f63181087b0 0x7f631819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 --2- 172.21.15.103:0/3535609660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6318107e10 0x7f631819e570 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:18.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 -- 172.21.15.103:0/3535609660 >> 172.21.15.103:0/3535609660 conn(0x7f6318077690 msgr2=0x7f63180786a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:18.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 -- 172.21.15.103:0/3535609660 shutdown_connections 2024-05-23T00:02:18.839 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:02:18.830+0000 7f631ca00640 1 -- 172.21.15.103:0/3535609660 wait complete. 2024-05-23T00:02:18.852 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 1273094960, 'ino': 1099511627781, 'path': '/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4/client.0'}, {'damage_type': 'backtrace', 'id': 1969116952, 'ino': 1099511628284, 'path': '/volumes/qa/sv_1/.meta'}, {'damage_type': 'backtrace', 'id': 2049093189, 'ino': 1099511628283, 'path': '/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4'}] 2024-05-23T00:02:18.852 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-05-23T00:02:20.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:20 smithi105 ceph-mon[36426]: pgmap v58: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-05-23T00:02:20.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:20 smithi103 ceph-mon[31370]: pgmap v58: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-05-23T00:02:20.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:20 smithi174 ceph-mon[36179]: pgmap v58: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-05-23T00:02:22.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:22 smithi105 ceph-mon[36426]: pgmap v59: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-05-23T00:02:22.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:22 smithi103 ceph-mon[31370]: pgmap v59: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-05-23T00:02:22.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:22 smithi174 ceph-mon[36179]: pgmap v59: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-05-23T00:02:24.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:24 smithi105 ceph-mon[36426]: pgmap v60: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-05-23T00:02:24.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:24 smithi103 ceph-mon[31370]: pgmap v60: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-05-23T00:02:24.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:24 smithi174 ceph-mon[36179]: pgmap v60: 129 pgs: 129 active+clean; 15 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2024-05-23T00:02:26.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:26 smithi105 ceph-mon[36426]: pgmap v61: 129 pgs: 129 active+clean; 16 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-05-23T00:02:26.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:26 smithi103 ceph-mon[31370]: pgmap v61: 129 pgs: 129 active+clean; 16 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-05-23T00:02:26.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:26 smithi174 ceph-mon[36179]: pgmap v61: 129 pgs: 129 active+clean; 16 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2024-05-23T00:02:28.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:28 smithi105 ceph-mon[36426]: pgmap v62: 129 pgs: 129 active+clean; 16 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-05-23T00:02:28.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:28 smithi103 ceph-mon[31370]: pgmap v62: 129 pgs: 129 active+clean; 16 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-05-23T00:02:28.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:28 smithi174 ceph-mon[36179]: pgmap v62: 129 pgs: 129 active+clean; 16 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2024-05-23T00:02:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.919+0000 7fa24a800640 1 -- 172.21.15.103:0/482939926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244074490 msgr2=0x7fa24410dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.919+0000 7fa24a800640 1 --2- 172.21.15.103:0/482939926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244074490 0x7fa24410dc20 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa230009900 tx=0x7fa23002a490 comp rx=0 tx=0).stop 2024-05-23T00:02:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.920+0000 7fa24a800640 1 -- 172.21.15.103:0/482939926 shutdown_connections 2024-05-23T00:02:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.920+0000 7fa24a800640 1 --2- 172.21.15.103:0/482939926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa24410e160 0x7fa244110550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.920+0000 7fa24a800640 1 --2- 172.21.15.103:0/482939926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244074490 0x7fa24410dc20 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.920+0000 7fa24a800640 1 --2- 172.21.15.103:0/482939926 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa244073b70 0x7fa244073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.920+0000 7fa24a800640 1 -- 172.21.15.103:0/482939926 >> 172.21.15.103:0/482939926 conn(0x7fa244078d00 msgr2=0x7fa244079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.920+0000 7fa24a800640 1 -- 172.21.15.103:0/482939926 shutdown_connections 2024-05-23T00:02:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.920+0000 7fa24a800640 1 -- 172.21.15.103:0/482939926 wait complete. 2024-05-23T00:02:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.921+0000 7fa24a800640 1 Processor -- start 2024-05-23T00:02:29.923 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.921+0000 7fa24a800640 1 -- start start 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa24a800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244073b70 0x7fa24419a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa24a800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa244074490 0x7fa24419a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa24a800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa24410e160 0x7fa24419ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa24a800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa244113e90 con 0x7fa244073b70 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa24a800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa244113d10 con 0x7fa24410e160 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa24a800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa244114010 con 0x7fa244074490 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa243400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244073b70 0x7fa24419a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa242a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa244074490 0x7fa24419a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa243400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244073b70 0x7fa24419a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45828/0 (socket says 172.21.15.103:45828) 2024-05-23T00:02:29.925 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa243400640 1 -- 172.21.15.103:0/1305775057 learned_addr learned my addr 172.21.15.103:0/1305775057 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.922+0000 7fa243e00640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa24410e160 0x7fa24419ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.923+0000 7fa242a00640 1 -- 172.21.15.103:0/1305775057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa24410e160 msgr2=0x7fa24419ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.923+0000 7fa242a00640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa24410e160 0x7fa24419ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.923+0000 7fa242a00640 1 -- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244073b70 msgr2=0x7fa24419a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.923+0000 7fa242a00640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244073b70 0x7fa24419a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.926 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.923+0000 7fa242a00640 1 -- 172.21.15.103:0/1305775057 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24419f250 con 0x7fa244074490 2024-05-23T00:02:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.923+0000 7fa242a00640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa244074490 0x7fa24419a750 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa230005940 tx=0x7fa2300040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.923+0000 7fa23be00640 1 -- 172.21.15.103:0/1305775057 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23003f030 con 0x7fa244074490 2024-05-23T00:02:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.923+0000 7fa23be00640 1 -- 172.21.15.103:0/1305775057 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2300044c0 con 0x7fa244074490 2024-05-23T00:02:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.924+0000 7fa24a800640 1 -- 172.21.15.103:0/1305775057 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa24419f4e0 con 0x7fa244074490 2024-05-23T00:02:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.924+0000 7fa23be00640 1 -- 172.21.15.103:0/1305775057 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230035070 con 0x7fa244074490 2024-05-23T00:02:29.927 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.924+0000 7fa24a800640 1 -- 172.21.15.103:0/1305775057 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2441a6d80 con 0x7fa244074490 2024-05-23T00:02:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa24d5cb740 1 Processor -- start 2024-05-23T00:02:29.932 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa24d5cb740 1 -- start start 2024-05-23T00:02:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa23be00640 1 -- 172.21.15.103:0/1305775057 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fa230042030 con 0x7fa244074490 2024-05-23T00:02:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26dc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dc670 0x5652d26e6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26e6d40 0x5652d26e9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5652d26402e0 con 0x5652d26e6d40 2024-05-23T00:02:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5652d25e7e50 con 0x5652d26dc670 2024-05-23T00:02:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.927+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652d263c2a0 con 0x5652d26dbd50 2024-05-23T00:02:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26dc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26dc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:36134/0 (socket says 172.21.15.103:36134) 2024-05-23T00:02:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa242a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26e6d40 0x5652d26e9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243e00640 1 -- 172.21.15.103:0/1858269678 learned_addr learned my addr 172.21.15.103:0/1858269678 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243400640 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dc670 0x5652d26e6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243e00640 1 -- 172.21.15.103:0/1858269678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dc670 msgr2=0x5652d26e6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243e00640 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dc670 0x5652d26e6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243e00640 1 -- 172.21.15.103:0/1858269678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26e6d40 msgr2=0x5652d26e9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243e00640 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26e6d40 0x5652d26e9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243e00640 1 -- 172.21.15.103:0/1858269678 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652d2354210 con 0x5652d26dbd50 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.928+0000 7fa243400640 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dc670 0x5652d26e6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa243e00640 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26dc130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009f40 tx=0x7fa23c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae1f572d5eae3b73 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa238c00640 1 -- 172.21.15.103:0/1858269678 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23c004d20 con 0x5652d26dbd50 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa238c00640 1 -- 172.21.15.103:0/1858269678 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa23c004050 con 0x5652d26dbd50 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa238c00640 1 -- 172.21.15.103:0/1858269678 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23c005280 con 0x5652d26dbd50 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa23be00640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa2280a2da0 0x7fa2280a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa23be00640 1 -- 172.21.15.103:0/1305775057 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa2300efb40 con 0x7fa244074490 2024-05-23T00:02:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa243400640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa2280a2da0 0x7fa2280a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1858269678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 msgr2=0x5652d26dc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.929+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26dc130 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa23c009f40 tx=0x7fa23c0118d0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1858269678 shutdown_connections 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26e6d40 0x5652d26e9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dc670 0x5652d26e6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1858269678 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26dc130 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1858269678 >> 172.21.15.103:0/1858269678 conn(0x5652d26d8070 msgr2=0x5652d26d8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1858269678 shutdown_connections 2024-05-23T00:02:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1858269678 wait complete. 2024-05-23T00:02:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa243400640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa2280a2da0 0x7fa2280a5260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x5652d26dd310 tx=0x7fa234005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.930+0000 7fa24d5cb740 1 -- start start 2024-05-23T00:02:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.931+0000 7fa24d5cb740 1 Processor -- start 2024-05-23T00:02:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.931+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26e3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.942 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.931+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26e4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.931+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 0x5652d26e4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.931+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5652d26e9bd0 con 0x5652d26dc670 2024-05-23T00:02:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.931+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5652d25fb210 con 0x5652d26dbd50 2024-05-23T00:02:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.931+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5652d26e9d50 con 0x5652d26e6d40 2024-05-23T00:02:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.931+0000 7fa24d5cb740 10 client.0 _ll_register_callbacks cb 0x5652d25de140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa242a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26e4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26e3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.943 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26e3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:51328/0 (socket says 172.21.15.103:51328) 2024-05-23T00:02:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 -- 172.21.15.103:0/1081547249 learned_addr learned my addr 172.21.15.103:0/1081547249 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243e00640 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 0x5652d26e4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 -- 172.21.15.103:0/1081547249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 msgr2=0x5652d26e4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 0x5652d26e4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 -- 172.21.15.103:0/1081547249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 msgr2=0x5652d26e4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26e4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 -- 172.21.15.103:0/1081547249 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652d2420620 con 0x5652d26dbd50 2024-05-23T00:02:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa242a00640 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26e4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa243400640 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26e3cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa234004610 tx=0x7fa234004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93bc205a78f20cbe server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.932+0000 7fa22d600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:02:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.933+0000 7fa22d600640 1 -- 172.21.15.103:0/1081547249 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23401e680 con 0x5652d26dbd50 2024-05-23T00:02:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.933+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1081547249 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa23c004050 con 0x5652d26dbd50 2024-05-23T00:02:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.933+0000 7fa22d600640 1 -- 172.21.15.103:0/1081547249 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa23401e820 con 0x5652d26dbd50 2024-05-23T00:02:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.933+0000 7fa22d600640 1 -- 172.21.15.103:0/1081547249 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa23401ea20 con 0x5652d26dbd50 2024-05-23T00:02:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.933+0000 7fa24d5cb740 10 client.26072 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.933+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1081547249 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5652d26f5f50 con 0x5652d26dbd50 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.933+0000 7fa22d600640 1 -- 172.21.15.103:0/1081547249 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa23c004050 con 0x5652d26dbd50 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.934+0000 7fa22d600640 1 -- 172.21.15.103:0/1081547249 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa234027070 con 0x5652d26dbd50 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.934+0000 7fa24d5cb740 10 client.26072 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.934+0000 7fa24d5cb740 10 client.26072 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.934+0000 7fa24d5cb740 20 client.26072 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.934+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5652d26f7620 0x5652d26f9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.934+0000 7fa24d5cb740 4 client.26072 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.934+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1081547249 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652d2653ec0 con 0x5652d26f7620 2024-05-23T00:02:29.946 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.934+0000 7fa242a00640 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5652d26f7620 0x5652d26f9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.935+0000 7fa242a00640 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5652d26f7620 0x5652d26f9ae0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eefb6db8bdd6e81a server_cookie=9531f4f230cb4a2e in_seq=0 out_seq=0 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.935+0000 7fa22d600640 10 client.26072 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.935+0000 7fa22d600640 1 -- 172.21.15.103:0/1081547249 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652d2653ec0 con 0x5652d26f7620 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.935+0000 7fa22d600640 10 client.26072 handle_command_reply: tid=0 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.935+0000 7fa24d5cb740 1 client.26072 shutdown 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1081547249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 msgr2=0x5652d26e3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26e3cd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa234004610 tx=0x7fa234004730 comp rx=0 tx=0).stop 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1081547249 shutdown_connections 2024-05-23T00:02:29.947 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5652d26f7620 0x5652d26f9ae0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 0x5652d26e4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26e4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1081547249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26e3cd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1081547249 >> 172.21.15.103:0/1081547249 conn(0x5652d26d8070 msgr2=0x5652d26dde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa22d600640 0 client.26072 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1081547249 shutdown_connections 2024-05-23T00:02:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1081547249 wait complete. 2024-05-23T00:02:29.948 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.936+0000 7fa24d5cb740 20 client.26072 trim_cache size 0 max 16384 2024-05-23T00:02:29.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.952+0000 7fa24d5cb740 1 Processor -- start 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.952+0000 7fa24d5cb740 1 -- start start 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.952+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.952+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26de5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.952+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 0x5652d26d7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.952+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441b38b0 con 0x5652d26dc670 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.952+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441b1180 con 0x5652d26dbd50 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.952+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441aefd0 con 0x5652d26e6d40 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa242a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26de5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa242a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26de5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:45864/0 (socket says 172.21.15.103:45864) 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa242a00640 1 -- 172.21.15.103:0/4219312665 learned_addr learned my addr 172.21.15.103:0/4219312665 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:29.962 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa243400640 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa243e00640 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 0x5652d26d7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa242a00640 1 -- 172.21.15.103:0/4219312665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 msgr2=0x5652d26d7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa242a00640 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 0x5652d26d7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa242a00640 1 -- 172.21.15.103:0/4219312665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 msgr2=0x5652d26f2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa242a00640 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26f2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa242a00640 1 -- 172.21.15.103:0/4219312665 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652d26dc4a0 con 0x5652d26dc670 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.953+0000 7fa243400640 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26f2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa242a00640 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26de5c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa23004a000 tx=0x7fa2300f35f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea954b2034af926 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa22fe00640 1 -- 172.21.15.103:0/4219312665 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230049040 con 0x5652d26dc670 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa22fe00640 1 -- 172.21.15.103:0/4219312665 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2300bbcc0 con 0x5652d26dc670 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa22fe00640 1 -- 172.21.15.103:0/4219312665 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2300b9df0 con 0x5652d26dc670 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa24d5cb740 1 -- 172.21.15.103:0/4219312665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 msgr2=0x5652d26de5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.963 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26de5c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa23004a000 tx=0x7fa2300f35f0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa24d5cb740 1 -- 172.21.15.103:0/4219312665 shutdown_connections 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26e6d40 0x5652d26d7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26de5c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/4219312665 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26dbd50 0x5652d26f2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.954+0000 7fa24d5cb740 1 -- 172.21.15.103:0/4219312665 >> 172.21.15.103:0/4219312665 conn(0x5652d26d8070 msgr2=0x5652d26d6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.955+0000 7fa24d5cb740 1 -- 172.21.15.103:0/4219312665 shutdown_connections 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.955+0000 7fa24d5cb740 1 -- 172.21.15.103:0/4219312665 wait complete. 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.955+0000 7fa24d5cb740 1 -- start start 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.955+0000 7fa24d5cb740 1 Processor -- start 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26ef510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa24d5cb740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26e6d40 0x5652d26e00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441af640 con 0x5652d26dc670 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441b0ab0 con 0x5652d26e6d40 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa24d5cb740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa2441afd10 con 0x5652d26dbd50 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa24d5cb740 10 client.0 _ll_register_callbacks cb 0x7fa2340043d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa242a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26eefd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:36154/0 (socket says 172.21.15.103:36154) 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 -- 172.21.15.103:0/1409797262 learned_addr learned my addr 172.21.15.103:0/1409797262 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243e00640 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26e6d40 0x5652d26e00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 -- 172.21.15.103:0/1409797262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26e6d40 msgr2=0x5652d26e00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26e6d40 0x5652d26e00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 -- 172.21.15.103:0/1409797262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 msgr2=0x5652d26ef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26ef510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 -- 172.21.15.103:0/1409797262 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652d26e08c0 con 0x5652d26dbd50 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa243400640 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26eefd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa2340045e0 tx=0x7fa234003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69a720a59005d4f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.956+0000 7fa22d600640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa22d600640 1 -- 172.21.15.103:0/1409797262 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23403c060 con 0x5652d26dbd50 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa22d600640 1 -- 172.21.15.103:0/1409797262 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fa23401edd0 con 0x5652d26dbd50 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1409797262 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2300bbcc0 con 0x5652d26dbd50 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa22d600640 1 -- 172.21.15.103:0/1409797262 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa234043050 con 0x5652d26dbd50 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 10 client.26083 fetch_fsmap learned FSMap version 28 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1409797262 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5652d2765e60 con 0x5652d26dbd50 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa22d600640 1 -- 172.21.15.103:0/1409797262 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2300bbcc0 con 0x5652d26dbd50 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa22d600640 1 -- 172.21.15.103:0/1409797262 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fa234027040 con 0x5652d26dbd50 2024-05-23T00:02:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 10 client.26083 fetch_fsmap finished waiting for FSMap version 28 2024-05-23T00:02:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 10 client.26083 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 20 client.26083 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5652d2766670 0x5652d2768a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 4 client.26083 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:02:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.957+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1409797262 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5652d264da10 con 0x5652d2766670 2024-05-23T00:02:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.958+0000 7fa242a00640 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5652d2766670 0x5652d2768a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.958+0000 7fa242a00640 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5652d2766670 0x5652d2768a90 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=675d90e0c23074aa server_cookie=40cbc13afec9df1 in_seq=0 out_seq=0 2024-05-23T00:02:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.958+0000 7fa22d600640 10 client.26083 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa22d600640 1 -- 172.21.15.103:0/1409797262 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5652d264da10 con 0x5652d2766670 2024-05-23T00:02:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa22d600640 10 client.26083 handle_command_reply: tid=0 2024-05-23T00:02:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 client.26083 shutdown 2024-05-23T00:02:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1409797262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 msgr2=0x5652d26eefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.968 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26eefd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa2340045e0 tx=0x7fa234003f90 comp rx=0 tx=0).stop 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1409797262 shutdown_connections 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5652d2766670 0x5652d2768a90 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5652d26e6d40 0x5652d26e00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5652d26dc670 0x5652d26ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa22d600640 0 client.26083 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 --2- 172.21.15.103:0/1409797262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5652d26dbd50 0x5652d26eefd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1409797262 >> 172.21.15.103:0/1409797262 conn(0x5652d26d8070 msgr2=0x5652d26d71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1409797262 shutdown_connections 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 1 -- 172.21.15.103:0/1409797262 wait complete. 2024-05-23T00:02:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.965+0000 7fa24d5cb740 20 client.26083 trim_cache size 0 max 16384 2024-05-23T00:02:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:02:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:02:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:02:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:02:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 -- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa2280a2da0 msgr2=0x7fa2280a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa2280a2da0 0x7fa2280a5260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x5652d26dd310 tx=0x7fa234005f90 comp rx=0 tx=0).stop 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 -- 172.21.15.103:0/1305775057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa244074490 msgr2=0x7fa24419a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa244074490 0x7fa24419a750 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa230005940 tx=0x7fa2300040c0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 -- 172.21.15.103:0/1305775057 shutdown_connections 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa2280a2da0 0x7fa2280a5260 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa24410e160 0x7fa24419ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa244074490 0x7fa24419a750 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 --2- 172.21.15.103:0/1305775057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa244073b70 0x7fa24419a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 -- 172.21.15.103:0/1305775057 >> 172.21.15.103:0/1305775057 conn(0x7fa244078d00 msgr2=0x7fa2440fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 -- 172.21.15.103:0/1305775057 shutdown_connections 2024-05-23T00:02:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:29.966+0000 7fa24a800640 1 -- 172.21.15.103:0/1305775057 wait complete. 2024-05-23T00:02:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:30 smithi105 ceph-mon[36426]: pgmap v63: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 27 op/s 2024-05-23T00:02:30.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:30 smithi103 ceph-mon[31370]: pgmap v63: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 27 op/s 2024-05-23T00:02:30.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:30 smithi174 ceph-mon[36179]: pgmap v63: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.6 MiB/s wr, 27 op/s 2024-05-23T00:02:32.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:32 smithi105 ceph-mon[36426]: pgmap v64: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 441 KiB/s wr, 20 op/s 2024-05-23T00:02:32.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:32 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:02:32.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:32 smithi103 ceph-mon[31370]: pgmap v64: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 441 KiB/s wr, 20 op/s 2024-05-23T00:02:32.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:32 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:02:32.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:32 smithi174 ceph-mon[36179]: pgmap v64: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 441 KiB/s wr, 20 op/s 2024-05-23T00:02:32.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:32 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:02:34.310 INFO:tasks.workunit.client.0.smithi103.stderr:Updating files: 7% (321/4295) Updating files: 8% (344/4295) Updating files: 9% (387/4295) Updating files: 10% (430/4295) Updating files: 11% (473/4295) Updating files: 11% (490/4295) Updating files: 12% (516/4295) Updating files: 13% (559/4295) Updating files: 14% (602/4295) Updating files: 15% (645/4295) Updating files: 16% (688/4295) Updating files: 17% (731/4295) Updating files: 17% (765/4295) Updating files: 18% (774/4295) Updating files: 19% (817/4295) Updating files: 20% (859/4295) Updating files: 21% (902/4295) Updating files: 22% (945/4295) Updating files: 23% (988/4295) Updating files: 24% (1031/4295) Updating files: 25% (1074/4295) Updating files: 26% (1117/4295) Updating files: 27% (1160/4295) Updating files: 28% (1203/4295) Updating files: 29% (1246/4295) Updating files: 29% (1288/4295) Updating files: 30% (1289/4295) Updating files: 31% (1332/4295) Updating files: 32% (1375/4295) Updating files: 33% (1418/4295) Updating files: 33% (1427/4295) Updating files: 34% (1461/4295) Updating files: 35% (1504/4295) Updating files: 36% (1547/4295) Updating files: 37% (1590/4295) Updating files: 38% (1633/4295) Updating files: 39% (1676/4295) Updating files: 40% (1718/4295) Updating files: 41% (1761/4295) Updating files: 42% (1804/4295) Updating files: 43% (1847/4295) Updating files: 44% (1890/4295) Updating files: 44% (1925/4295) Updating files: 45% (1933/4295) Updating files: 46% (1976/4295) Updating files: 47% (2019/4295) Updating files: 48% (2062/4295) Updating files: 49% (2105/4295) Updating files: 50% (2148/4295) Updating files: 51% (2191/4295) Updating files: 52% (2234/4295) Updating files: 53% (2277/4295) Updating files: 54% (2320/4295) Updating files: 54% (2336/4295) Updating files: 55% (2363/4295) Updating files: 56% (2406/4295) Updating files: 57% (2449/4295) Updating files: 58% (2492/4295) Updating files: 59% (2535/4295) Updating files: 60% (2577/4295) Updating files: 61% (2620/4295) Updating files: 62% (2663/4295) Updating files: 63% (2706/4295) Updating files: 64% (2749/4295) Updating files: 64% (2786/4295) Updating files: 65% (2792/4295) Updating files: 66% (2835/4295) Updating files: 67% (2878/4295) Updating files: 68% (2921/4295) Updating files: 69% (2964/4295) Updating files: 70% (3007/4295) Updating files: 70% (3024/4295) Updating files: 71% (3050/4295) Updating files: 72% (3093/4295) Updating files: 73% (3136/4295) Updating files: 74% (3179/4295) Updating files: 75% (3222/4295) Updating files: 76% (3265/4295) Updating files: 77% (3308/4295) Updating files: 78% (3351/4295) Updating files: 79% (3394/4295) Updating files: 80% (3436/4295) Updating files: 81% (3479/4295) Updating files: 82% (3522/4295) Updating files: 82% (3540/4295) Updating files: 83% (3565/4295) Updating files: 84% (3608/4295) Updating files: 85% (3651/4295) Updating files: 86% (3694/4295) Updating files: 87% (3737/4295) Updating files: 88% (3780/4295) Updating files: 89% (3823/4295) Updating files: 90% (3866/4295) Updating files: 91% (3909/4295) Updating files: 92% (3952/4295) Updating files: 93% (3995/4295) Updating files: 94% (4038/4295) Updating files: 95% (4081/4295) Updating files: 96% (4124/4295) Updating files: 97% (4167/4295) Updating files: 98% (4210/4295) Updating files: 99% (4253/4295) Updating files: 100% (4295/4295) Updating files: 100% (4295/4295), done. 2024-05-23T00:02:34.331 INFO:tasks.workunit.client.0.smithi103.stderr:+ cd xfstests-dev 2024-05-23T00:02:34.331 INFO:tasks.workunit.client.0.smithi103.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-05-23T00:02:34.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:34 smithi105 ceph-mon[36426]: pgmap v65: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 441 KiB/s wr, 20 op/s 2024-05-23T00:02:34.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:34 smithi103 ceph-mon[31370]: pgmap v65: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 441 KiB/s wr, 20 op/s 2024-05-23T00:02:34.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:34 smithi174 ceph-mon[36179]: pgmap v65: 129 pgs: 129 active+clean; 20 MiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 441 KiB/s wr, 20 op/s 2024-05-23T00:02:36.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:36 smithi105 ceph-mon[36426]: pgmap v66: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 833 KiB/s wr, 41 op/s 2024-05-23T00:02:36.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:36 smithi103 ceph-mon[31370]: pgmap v66: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 833 KiB/s wr, 41 op/s 2024-05-23T00:02:36.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:36 smithi174 ceph-mon[36179]: pgmap v66: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 833 KiB/s wr, 41 op/s 2024-05-23T00:02:38.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:38 smithi105 ceph-mon[36426]: pgmap v67: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 786 KiB/s wr, 34 op/s 2024-05-23T00:02:38.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:38 smithi103 ceph-mon[31370]: pgmap v67: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 786 KiB/s wr, 34 op/s 2024-05-23T00:02:38.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:38 smithi174 ceph-mon[36179]: pgmap v67: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 786 KiB/s wr, 34 op/s 2024-05-23T00:02:40.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:40 smithi105 ceph-mon[36426]: pgmap v68: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-05-23T00:02:40.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:40 smithi103 ceph-mon[31370]: pgmap v68: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-05-23T00:02:40.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:40 smithi174 ceph-mon[36179]: pgmap v68: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 178 op/s 2024-05-23T00:02:42.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:42 smithi105 ceph-mon[36426]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-05-23T00:02:42.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:42 smithi103 ceph-mon[31370]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-05-23T00:02:42.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:42 smithi174 ceph-mon[36179]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 164 op/s 2024-05-23T00:02:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 -- 172.21.15.103:0/615821718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070112fe0 msgr2=0x7f30701153d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 --2- 172.21.15.103:0/615821718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070112fe0 0x7f30701153d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3068009cd0 tx=0x7f306802a8e0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 -- 172.21.15.103:0/615821718 shutdown_connections 2024-05-23T00:02:43.198 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 --2- 172.21.15.103:0/615821718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070112fe0 0x7f30701153d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 --2- 172.21.15.103:0/615821718 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3070109400 0x7f307010d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 --2- 172.21.15.103:0/615821718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3070069230 0x7f307010cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 -- 172.21.15.103:0/615821718 >> 172.21.15.103:0/615821718 conn(0x7f307006cc00 msgr2=0x7f307006d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 -- 172.21.15.103:0/615821718 shutdown_connections 2024-05-23T00:02:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.196+0000 7f3075600640 1 -- 172.21.15.103:0/615821718 wait complete. 2024-05-23T00:02:43.199 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.197+0000 7f3075600640 1 Processor -- start 2024-05-23T00:02:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.198+0000 7f3075600640 1 -- start start 2024-05-23T00:02:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.198+0000 7f3075600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3070069230 0x7f307011d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.198+0000 7f3075600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070109400 0x7f3070119ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.198+0000 7f3075600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3070112fe0 0x7f307011a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.198+0000 7f3075600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3070117140 con 0x7f3070112fe0 2024-05-23T00:02:43.200 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.198+0000 7f3075600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3070116fc0 con 0x7f3070109400 2024-05-23T00:02:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.198+0000 7f3075600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30701172c0 con 0x7f3070069230 2024-05-23T00:02:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.199+0000 7f306f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070109400 0x7f3070119ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.199+0000 7f306f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070109400 0x7f3070119ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:47162/0 (socket says 172.21.15.103:47162) 2024-05-23T00:02:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.199+0000 7f306f400640 1 -- 172.21.15.103:0/2432837528 learned_addr learned my addr 172.21.15.103:0/2432837528 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:43.201 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.199+0000 7f306fe00640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3070069230 0x7f307011d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.200+0000 7f306f400640 1 -- 172.21.15.103:0/2432837528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3070069230 msgr2=0x7f307011d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.200+0000 7f306f400640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3070069230 0x7f307011d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.200+0000 7f306f400640 1 -- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3070112fe0 msgr2=0x7f307011a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.200+0000 7f306f400640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3070112fe0 0x7f307011a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.200+0000 7f306f400640 1 -- 172.21.15.103:0/2432837528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307011a7c0 con 0x7f3070109400 2024-05-23T00:02:43.202 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.200+0000 7f306f400640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070109400 0x7f3070119ba0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f305c00e9f0 tx=0x7f305c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.200+0000 7f306cc00640 1 -- 172.21.15.103:0/2432837528 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c00cd80 con 0x7f3070109400 2024-05-23T00:02:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.201+0000 7f306cc00640 1 -- 172.21.15.103:0/2432837528 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f305c005500 con 0x7f3070109400 2024-05-23T00:02:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.201+0000 7f306cc00640 1 -- 172.21.15.103:0/2432837528 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c00cd80 con 0x7f3070109400 2024-05-23T00:02:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.201+0000 7f3075600640 1 -- 172.21.15.103:0/2432837528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30701b3520 con 0x7f3070109400 2024-05-23T00:02:43.203 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.201+0000 7f3075600640 1 -- 172.21.15.103:0/2432837528 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30701b38d0 con 0x7f3070109400 2024-05-23T00:02:43.212 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.210+0000 7f307996c740 1 Processor -- start 2024-05-23T00:02:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.211+0000 7f306cc00640 1 -- 172.21.15.103:0/2432837528 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f305c0046c0 con 0x7f3070109400 2024-05-23T00:02:43.213 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.211+0000 7f307996c740 1 -- start start 2024-05-23T00:02:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.211+0000 7f307996c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68a9f30 0x5568f68aa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.211+0000 7f307996c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68aa850 0x5568f68b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.211+0000 7f307996c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68b4f20 0x5568f68b7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.211+0000 7f307996c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5568f680e2e0 con 0x5568f68b4f20 2024-05-23T00:02:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.211+0000 7f307996c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5568f67b5e50 con 0x5568f68a9f30 2024-05-23T00:02:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.211+0000 7f307996c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5568f680a2a0 con 0x5568f68aa850 2024-05-23T00:02:43.214 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.212+0000 7f3074c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68a9f30 0x5568f68aa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.215 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.212+0000 7f306fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68aa850 0x5568f68b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.212+0000 7f3074c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68a9f30 0x5568f68aa310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:47164/0 (socket says 172.21.15.103:47164) 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.212+0000 7f306fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68aa850 0x5568f68b49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:58498/0 (socket says 172.21.15.103:58498) 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.212+0000 7f3074c00640 1 -- 172.21.15.103:0/3119361451 learned_addr learned my addr 172.21.15.103:0/3119361451 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.213+0000 7f3074c00640 1 -- 172.21.15.103:0/3119361451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68aa850 msgr2=0x5568f68b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.213+0000 7f3074c00640 1 --2- 172.21.15.103:0/3119361451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68aa850 0x5568f68b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.213+0000 7f3074c00640 1 -- 172.21.15.103:0/3119361451 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68b4f20 msgr2=0x5568f68b7310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.213+0000 7f3074c00640 1 --2- 172.21.15.103:0/3119361451 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68b4f20 0x5568f68b7310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.213+0000 7f3074c00640 1 -- 172.21.15.103:0/3119361451 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f6522210 con 0x5568f68a9f30 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.213+0000 7f306cc00640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30540a2fb0 0x7f30540a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.213+0000 7f3074c00640 1 --2- 172.21.15.103:0/3119361451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68a9f30 0x5568f68aa310 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f306802ae50 tx=0x7f3068009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a477aa8d46e029a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:43.216 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.214+0000 7f306cc00640 1 -- 172.21.15.103:0/2432837528 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f305c0cf710 con 0x7f3070109400 2024-05-23T00:02:43.217 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.214+0000 7f3065200640 1 -- 172.21.15.103:0/3119361451 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068031630 con 0x5568f68a9f30 2024-05-23T00:02:43.217 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.214+0000 7f306fe00640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30540a2fb0 0x7f30540a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.217 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.214+0000 7f3065200640 1 -- 172.21.15.103:0/3119361451 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3068041070 con 0x5568f68a9f30 2024-05-23T00:02:43.217 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.215+0000 7f3065200640 1 -- 172.21.15.103:0/3119361451 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068031630 con 0x5568f68a9f30 2024-05-23T00:02:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.216+0000 7f307996c740 1 -- 172.21.15.103:0/3119361451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68a9f30 msgr2=0x5568f68aa310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.216+0000 7f307996c740 1 --2- 172.21.15.103:0/3119361451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68a9f30 0x5568f68aa310 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f306802ae50 tx=0x7f3068009a20 comp rx=0 tx=0).stop 2024-05-23T00:02:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.216+0000 7f307996c740 1 -- 172.21.15.103:0/3119361451 shutdown_connections 2024-05-23T00:02:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.216+0000 7f307996c740 1 --2- 172.21.15.103:0/3119361451 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68b4f20 0x5568f68b7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.218 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.216+0000 7f307996c740 1 --2- 172.21.15.103:0/3119361451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68aa850 0x5568f68b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.219 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.217+0000 7f307996c740 1 --2- 172.21.15.103:0/3119361451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68a9f30 0x5568f68aa310 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.219 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.217+0000 7f307996c740 1 -- 172.21.15.103:0/3119361451 >> 172.21.15.103:0/3119361451 conn(0x5568f68a6250 msgr2=0x5568f68a6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:43.219 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.217+0000 7f306fe00640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30540a2fb0 0x7f30540a5470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x5568f68ab4f0 tx=0x7f306000b410 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:43.219 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.217+0000 7f307996c740 1 -- 172.21.15.103:0/3119361451 shutdown_connections 2024-05-23T00:02:43.220 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.218+0000 7f307996c740 1 -- 172.21.15.103:0/3119361451 wait complete. 2024-05-23T00:02:43.221 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.219+0000 7f307996c740 1 -- start start 2024-05-23T00:02:43.221 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.219+0000 7f307996c740 1 Processor -- start 2024-05-23T00:02:43.222 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.220+0000 7f307996c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 0x5568f68b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.221+0000 7f307996c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68b2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.221+0000 7f307996c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68b2950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.223 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.221+0000 7f307996c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5568f68b7d50 con 0x5568f68a9f30 2024-05-23T00:02:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.221+0000 7f3074c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.221+0000 7f3074c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:58504/0 (socket says 172.21.15.103:58504) 2024-05-23T00:02:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.221+0000 7f3074c00640 1 -- 172.21.15.103:0/2010869551 learned_addr learned my addr 172.21.15.103:0/2010869551 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.221+0000 7f306fe00640 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 0x5568f68b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.221+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5568f67c9210 con 0x5568f68aa850 2024-05-23T00:02:43.224 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.222+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5568f68b7ed0 con 0x5568f68b4f20 2024-05-23T00:02:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.222+0000 7f306f400640 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.222+0000 7f3074c00640 1 -- 172.21.15.103:0/2010869551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 msgr2=0x5568f68b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.222+0000 7f3074c00640 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68b2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.222+0000 7f3074c00640 1 -- 172.21.15.103:0/2010869551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 msgr2=0x5568f68b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.222+0000 7f3074c00640 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 0x5568f68b1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.222+0000 7f3074c00640 1 -- 172.21.15.103:0/2010869551 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f65ee620 con 0x5568f68b4f20 2024-05-23T00:02:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.223+0000 7f307996c740 10 client.0 _ll_register_callbacks cb 0x5568f67ac140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:43.225 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.223+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3068041070 con 0x5568f68b4f20 2024-05-23T00:02:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.223+0000 7f3074c00640 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68b2950 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3068005e30 tx=0x7f3068003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6425b74dbfb553df server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:43.228 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.226+0000 7f3059600640 10 client.26095 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:02:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.227+0000 7f3059600640 1 -- 172.21.15.103:0/2010869551 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068031cd0 con 0x5568f68b4f20 2024-05-23T00:02:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.227+0000 7f3059600640 1 -- 172.21.15.103:0/2010869551 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3068004b40 con 0x5568f68b4f20 2024-05-23T00:02:43.229 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.227+0000 7f3059600640 1 -- 172.21.15.103:0/2010869551 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3068004d80 con 0x5568f68b4f20 2024-05-23T00:02:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.228+0000 7f307996c740 10 client.26095 fetch_fsmap learned FSMap version 29 2024-05-23T00:02:43.230 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.228+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({fsmap=29}) -- 0x5568f68c3d10 con 0x5568f68b4f20 2024-05-23T00:02:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.229+0000 7f3059600640 1 -- 172.21.15.103:0/2010869551 <== mon.1 v2:172.21.15.174:3300/0 4 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f3068030070 con 0x5568f68b4f20 2024-05-23T00:02:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.229+0000 7f307996c740 10 client.26095 fetch_fsmap finished waiting for FSMap version 29 2024-05-23T00:02:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.229+0000 7f307996c740 10 client.26095 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.229+0000 7f307996c740 20 client.26095 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.229+0000 7f307996c740 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5568f68c5060 0x5568f68c7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.229+0000 7f307996c740 4 client.26095 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.229+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5568f6821ec0 con 0x5568f68c5060 2024-05-23T00:02:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.229+0000 7f306fe00640 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5568f68c5060 0x5568f68c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.230+0000 7f306fe00640 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5568f68c5060 0x5568f68c7450 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eed2c339e2d6e3e0 server_cookie=f28439a843bdaf1b in_seq=0 out_seq=0 2024-05-23T00:02:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.230+0000 7f3059600640 10 client.26095 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.231+0000 7f3059600640 1 -- 172.21.15.103:0/2010869551 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5568f6821ec0 con 0x5568f68c5060 2024-05-23T00:02:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.232+0000 7f3059600640 10 client.26095 handle_command_reply: tid=0 2024-05-23T00:02:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.232+0000 7f307996c740 1 client.26095 shutdown 2024-05-23T00:02:43.234 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.232+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 msgr2=0x5568f68b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.232+0000 7f307996c740 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68b2950 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3068005e30 tx=0x7f3068003ba0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.233+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 shutdown_connections 2024-05-23T00:02:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.233+0000 7f307996c740 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5568f68c5060 0x5568f68c7450 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.233+0000 7f3059600640 0 client.26095 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:43.235 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.233+0000 7f307996c740 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68b2950 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.236 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.233+0000 7f307996c740 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.236 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.233+0000 7f307996c740 1 --2- 172.21.15.103:0/2010869551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 0x5568f68b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.236 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.233+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 >> 172.21.15.103:0/2010869551 conn(0x5568f68a6250 msgr2=0x5568f68ac050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:43.236 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.233+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 shutdown_connections 2024-05-23T00:02:43.236 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.234+0000 7f307996c740 1 -- 172.21.15.103:0/2010869551 wait complete. 2024-05-23T00:02:43.236 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.234+0000 7f307996c740 20 client.26095 trim_cache size 0 max 16384 2024-05-23T00:02:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.267+0000 7f307996c740 1 Processor -- start 2024-05-23T00:02:43.269 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.267+0000 7f307996c740 1 -- start start 2024-05-23T00:02:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.267+0000 7f307996c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 0x5568f68c0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.267+0000 7f307996c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f307996c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f307996c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30701c03f0 con 0x5568f68a9f30 2024-05-23T00:02:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f307996c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30701bdc90 con 0x5568f68aa850 2024-05-23T00:02:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f307996c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30701bbb30 con 0x5568f68b4f20 2024-05-23T00:02:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f306f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.270 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f306f400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:47188/0 (socket says 172.21.15.103:47188) 2024-05-23T00:02:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f306f400640 1 -- 172.21.15.103:0/1377518361 learned_addr learned my addr 172.21.15.103:0/1377518361 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f306f400640 1 -- 172.21.15.103:0/1377518361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 msgr2=0x5568f68cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:43.271 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f306f400640 1 --2- 172.21.15.103:0/1377518361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.268+0000 7f306f400640 1 -- 172.21.15.103:0/1377518361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 msgr2=0x5568f68c0c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-23T00:02:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.269+0000 7f306f400640 1 --2- 172.21.15.103:0/1377518361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 0x5568f68c0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.269+0000 7f306f400640 1 -- 172.21.15.103:0/1377518361 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568f68aada0 con 0x5568f68aa850 2024-05-23T00:02:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.269+0000 7f306f400640 1 --2- 172.21.15.103:0/1377518361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68ccb90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f305c097ca0 tx=0x7f305c09b6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1bf35d199d27629 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.269+0000 7f305be00640 1 -- 172.21.15.103:0/1377518361 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c09b840 con 0x5568f68aa850 2024-05-23T00:02:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.269+0000 7f305be00640 1 -- 172.21.15.103:0/1377518361 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f305c09fbf0 con 0x5568f68aa850 2024-05-23T00:02:43.272 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.269+0000 7f305be00640 1 -- 172.21.15.103:0/1377518361 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c086240 con 0x5568f68aa850 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.270+0000 7f307996c740 1 -- 172.21.15.103:0/1377518361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 msgr2=0x5568f68ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.270+0000 7f307996c740 1 --2- 172.21.15.103:0/1377518361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68ccb90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f305c097ca0 tx=0x7f305c09b6d0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.271+0000 7f307996c740 1 -- 172.21.15.103:0/1377518361 shutdown_connections 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.271+0000 7f307996c740 1 --2- 172.21.15.103:0/1377518361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68b4f20 0x5568f68cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.271+0000 7f307996c740 1 --2- 172.21.15.103:0/1377518361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68aa850 0x5568f68ccb90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.271+0000 7f307996c740 1 --2- 172.21.15.103:0/1377518361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68a9f30 0x5568f68c0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.271+0000 7f307996c740 1 -- 172.21.15.103:0/1377518361 >> 172.21.15.103:0/1377518361 conn(0x5568f68a6250 msgr2=0x5568f68b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.271+0000 7f307996c740 1 -- 172.21.15.103:0/1377518361 shutdown_connections 2024-05-23T00:02:43.273 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.271+0000 7f307996c740 1 -- 172.21.15.103:0/1377518361 wait complete. 2024-05-23T00:02:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.271+0000 7f307996c740 1 -- start start 2024-05-23T00:02:43.274 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.272+0000 7f307996c740 1 Processor -- start 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.272+0000 7f307996c740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68a9f30 0x5568f68aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.272+0000 7f307996c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68aa850 0x5568f68af500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.272+0000 7f307996c740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68b4f20 0x5568f692a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.272+0000 7f307996c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30701bc150 con 0x5568f68aa850 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.272+0000 7f307996c740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30701bd5c0 con 0x5568f68b4f20 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.272+0000 7f307996c740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30701bc820 con 0x5568f68a9f30 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.272+0000 7f307996c740 10 client.0 _ll_register_callbacks cb 0x7f3068009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f3074c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68b4f20 0x5568f692a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f306fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68a9f30 0x5568f68aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f306fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68a9f30 0x5568f68aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:58516/0 (socket says 172.21.15.103:58516) 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f306fe00640 1 -- 172.21.15.103:0/3037625476 learned_addr learned my addr 172.21.15.103:0/3037625476 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:43.275 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f3074c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68b4f20 0x5568f692a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:47196/0 (socket says 172.21.15.103:47196) 2024-05-23T00:02:43.276 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f3074c00640 1 -- 172.21.15.103:0/3037625476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68a9f30 msgr2=0x5568f68aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f3074c00640 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68a9f30 0x5568f68aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f3074c00640 1 -- 172.21.15.103:0/3037625476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68aa850 msgr2=0x5568f68af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f3074c00640 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68aa850 0x5568f68af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.273+0000 7f3074c00640 1 -- 172.21.15.103:0/3037625476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5568f692a940 con 0x5568f68b4f20 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.274+0000 7f3074c00640 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68b4f20 0x5568f692a400 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f306804ff70 tx=0x7f3068009a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5ef38554128487 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.274+0000 7f3059600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.274+0000 7f3059600640 1 -- 172.21.15.103:0/3037625476 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3068031ad0 con 0x5568f68b4f20 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.274+0000 7f3059600640 1 -- 172.21.15.103:0/3037625476 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f3068031c70 con 0x5568f68b4f20 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.274+0000 7f307996c740 1 -- 172.21.15.103:0/3037625476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f305c09fbf0 con 0x5568f68b4f20 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.275+0000 7f3059600640 1 -- 172.21.15.103:0/3037625476 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f30680503f0 con 0x5568f68b4f20 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.275+0000 7f307996c740 10 client.26096 fetch_fsmap learned FSMap version 29 2024-05-23T00:02:43.277 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.275+0000 7f307996c740 1 -- 172.21.15.103:0/3037625476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5568f692bb10 con 0x5568f68b4f20 2024-05-23T00:02:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.275+0000 7f3059600640 1 -- 172.21.15.103:0/3037625476 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f305c09fbf0 con 0x5568f68b4f20 2024-05-23T00:02:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.275+0000 7f3059600640 1 -- 172.21.15.103:0/3037625476 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 29) ==== 8358+0+0 (secure 0 0 0) 0x7f3068030070 con 0x5568f68b4f20 2024-05-23T00:02:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.276+0000 7f307996c740 10 client.26096 fetch_fsmap finished waiting for FSMap version 29 2024-05-23T00:02:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.276+0000 7f307996c740 10 client.26096 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.276+0000 7f307996c740 20 client.26096 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.276+0000 7f307996c740 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5568f692c1c0 0x5568f692e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:43.278 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.276+0000 7f307996c740 4 client.26096 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:02:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.276+0000 7f307996c740 1 -- 172.21.15.103:0/3037625476 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5568f681ba10 con 0x5568f692c1c0 2024-05-23T00:02:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.276+0000 7f306fe00640 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5568f692c1c0 0x5568f692e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.277+0000 7f306fe00640 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5568f692c1c0 0x5568f692e5b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8b48160e2656767 server_cookie=afe31d498fa5d55a in_seq=0 out_seq=0 2024-05-23T00:02:43.279 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.277+0000 7f3059600640 10 client.26096 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:43.806 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.804+0000 7f3059600640 1 -- 172.21.15.103:0/3037625476 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5568f681ba10 con 0x5568f692c1c0 2024-05-23T00:02:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.804+0000 7f3059600640 10 client.26096 handle_command_reply: tid=0 2024-05-23T00:02:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.804+0000 7f307996c740 1 client.26096 shutdown 2024-05-23T00:02:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 -- 172.21.15.103:0/3037625476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68b4f20 msgr2=0x5568f692a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68b4f20 0x5568f692a400 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f306804ff70 tx=0x7f3068009a20 comp rx=0 tx=0).stop 2024-05-23T00:02:43.807 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 -- 172.21.15.103:0/3037625476 shutdown_connections 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5568f692c1c0 0x5568f692e5b0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5568f68b4f20 0x5568f692a400 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5568f68aa850 0x5568f68af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 --2- 172.21.15.103:0/3037625476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5568f68a9f30 0x5568f68aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 -- 172.21.15.103:0/3037625476 >> 172.21.15.103:0/3037625476 conn(0x5568f68a6250 msgr2=0x5568f68b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f3059600640 0 client.26096 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 -- 172.21.15.103:0/3037625476 shutdown_connections 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 1 -- 172.21.15.103:0/3037625476 wait complete. 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.805+0000 7f307996c740 20 client.26096 trim_cache size 0 max 16384 2024-05-23T00:02:43.808 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:02:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:02:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:02:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:02:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 -- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30540a2fb0 msgr2=0x7f30540a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.809 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30540a2fb0 0x7f30540a5470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x5568f68ab4f0 tx=0x7f306000b410 comp rx=0 tx=0).stop 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 -- 172.21.15.103:0/2432837528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070109400 msgr2=0x7f3070119ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070109400 0x7f3070119ba0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f305c00e9f0 tx=0x7f305c00edf0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 -- 172.21.15.103:0/2432837528 shutdown_connections 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30540a2fb0 0x7f30540a5470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3070112fe0 0x7f307011a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3070109400 0x7f3070119ba0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 --2- 172.21.15.103:0/2432837528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3070069230 0x7f307011d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.806+0000 7f3075600640 1 -- 172.21.15.103:0/2432837528 >> 172.21.15.103:0/2432837528 conn(0x7f307006cc00 msgr2=0x7f3070113bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.807+0000 7f3075600640 1 -- 172.21.15.103:0/2432837528 shutdown_connections 2024-05-23T00:02:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:43.807+0000 7f3075600640 1 -- 172.21.15.103:0/2432837528 wait complete. 2024-05-23T00:02:44.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:44 smithi105 ceph-mon[36426]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-23T00:02:44.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:44 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:02:44.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:44 smithi103 ceph-mon[31370]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-23T00:02:44.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:44 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:02:44.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:44 smithi174 ceph-mon[36179]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 435 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.5 MiB/s wr, 165 op/s 2024-05-23T00:02:44.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:44 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:02:45.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:02:45.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:02:45.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:45 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:02:45.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:02:45.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:02:45.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:45 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:02:45.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:02:45.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:02:45.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:45 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:02:46.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:46 smithi105 ceph-mon[36426]: pgmap v71: 129 pgs: 129 active+clean; 46 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 560 op/s 2024-05-23T00:02:46.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:46 smithi103 ceph-mon[31370]: pgmap v71: 129 pgs: 129 active+clean; 46 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 560 op/s 2024-05-23T00:02:46.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:46 smithi174 ceph-mon[36179]: pgmap v71: 129 pgs: 129 active+clean; 46 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.9 MiB/s wr, 560 op/s 2024-05-23T00:02:47.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:47 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:02:47.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:47 smithi105 ceph-mon[36426]: pgmap v72: 129 pgs: 129 active+clean; 46 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 539 op/s 2024-05-23T00:02:47.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:47 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:02:47.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:47 smithi103 ceph-mon[31370]: pgmap v72: 129 pgs: 129 active+clean; 46 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 539 op/s 2024-05-23T00:02:47.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:47 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:02:47.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:47 smithi174 ceph-mon[36179]: pgmap v72: 129 pgs: 129 active+clean; 46 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 539 op/s 2024-05-23T00:02:50.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:49 smithi103 ceph-mon[31370]: pgmap v73: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 669 op/s 2024-05-23T00:02:50.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:49 smithi174 ceph-mon[36179]: pgmap v73: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 669 op/s 2024-05-23T00:02:50.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:49 smithi105 ceph-mon[36426]: pgmap v73: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 3.3 MiB/s wr, 669 op/s 2024-05-23T00:02:51.556 INFO:tasks.workunit.client.0.smithi103.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-05-23T00:02:51.556 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:51.556 INFO:tasks.workunit.client.0.smithi103.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-23T00:02:51.557 INFO:tasks.workunit.client.0.smithi103.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-23T00:02:51.557 INFO:tasks.workunit.client.0.smithi103.stderr:state without impacting any branches by switching back to a branch. 2024-05-23T00:02:51.557 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:51.557 INFO:tasks.workunit.client.0.smithi103.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-23T00:02:51.557 INFO:tasks.workunit.client.0.smithi103.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-23T00:02:51.558 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:51.558 INFO:tasks.workunit.client.0.smithi103.stderr: git switch -c 2024-05-23T00:02:51.558 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:51.558 INFO:tasks.workunit.client.0.smithi103.stderr:Or undo this operation with: 2024-05-23T00:02:51.558 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:51.559 INFO:tasks.workunit.client.0.smithi103.stderr: git switch - 2024-05-23T00:02:51.559 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:51.559 INFO:tasks.workunit.client.0.smithi103.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-23T00:02:51.559 INFO:tasks.workunit.client.0.smithi103.stderr: 2024-05-23T00:02:51.560 INFO:tasks.workunit.client.0.smithi103.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-05-23T00:02:51.577 INFO:tasks.workunit.client.0.smithi103.stderr:+ make -j4 2024-05-23T00:02:51.614 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize -cfi 2024-05-23T00:02:52.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:51 smithi103 ceph-mon[31370]: pgmap v74: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 525 op/s 2024-05-23T00:02:52.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:51 smithi174 ceph-mon[36179]: pgmap v74: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 525 op/s 2024-05-23T00:02:52.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:51 smithi105 ceph-mon[36426]: pgmap v74: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 525 op/s 2024-05-23T00:02:52.510 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: putting auxiliary files in '.'. 2024-05-23T00:02:52.510 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file './config.guess' 2024-05-23T00:02:52.526 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file './config.sub' 2024-05-23T00:02:52.545 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file './install-sh' 2024-05-23T00:02:52.565 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file './ltmain.sh' 2024-05-23T00:02:52.677 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-05-23T00:02:52.677 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-05-23T00:02:52.708 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-05-23T00:02:52.731 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-05-23T00:02:52.752 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-05-23T00:02:52.774 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-05-23T00:02:52.794 INFO:tasks.workunit.client.0.smithi103.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-05-23T00:02:52.798 INFO:tasks.workunit.client.0.smithi103.stdout:cp include/install-sh . 2024-05-23T00:02:52.808 INFO:tasks.workunit.client.0.smithi103.stdout:aclocal -I m4 2024-05-23T00:02:54.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:53 smithi103 ceph-mon[31370]: pgmap v75: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 525 op/s 2024-05-23T00:02:54.109 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:53 smithi105 ceph-mon[36426]: pgmap v75: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 525 op/s 2024-05-23T00:02:54.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:53 smithi174 ceph-mon[36179]: pgmap v75: 129 pgs: 129 active+clean; 53 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 525 op/s 2024-05-23T00:02:54.532 INFO:tasks.workunit.client.0.smithi103.stdout:autoheader 2024-05-23T00:02:54.961 INFO:tasks.workunit.client.0.smithi103.stdout:autoconf 2024-05-23T00:02:55.225 INFO:tasks.workunit.client.0.smithi103.stdout:./configure \ 2024-05-23T00:02:55.225 INFO:tasks.workunit.client.0.smithi103.stdout: --libexecdir=/usr/lib \ 2024-05-23T00:02:55.225 INFO:tasks.workunit.client.0.smithi103.stdout: --exec_prefix=/var/lib 2024-05-23T00:02:55.591 INFO:tasks.workunit.client.0.smithi103.stdout:checking build system type... x86_64-pc-linux-gnu 2024-05-23T00:02:55.591 INFO:tasks.workunit.client.0.smithi103.stdout:checking host system type... x86_64-pc-linux-gnu 2024-05-23T00:02:55.595 INFO:tasks.workunit.client.0.smithi103.stdout:checking how to print strings... printf 2024-05-23T00:02:55.595 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gcc... gcc 2024-05-23T00:02:55.778 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether the C compiler works... yes 2024-05-23T00:02:55.778 INFO:tasks.workunit.client.0.smithi103.stdout:checking for C compiler default output file name... a.out 2024-05-23T00:02:55.942 INFO:tasks.workunit.client.0.smithi103.stdout:checking for suffix of executables... 2024-05-23T00:02:56.034 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether we are cross compiling... no 2024-05-23T00:02:56.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:55 smithi103 ceph-mon[31370]: pgmap v76: 129 pgs: 129 active+clean; 57 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 598 op/s 2024-05-23T00:02:56.111 INFO:tasks.workunit.client.0.smithi103.stdout:checking for suffix of object files... o 2024-05-23T00:02:56.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:55 smithi174 ceph-mon[36179]: pgmap v76: 129 pgs: 129 active+clean; 57 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 598 op/s 2024-05-23T00:02:56.170 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether we are using the GNU C compiler... yes 2024-05-23T00:02:56.227 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether gcc accepts -g... yes 2024-05-23T00:02:56.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:55 smithi105 ceph-mon[36426]: pgmap v76: 129 pgs: 129 active+clean; 57 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 598 op/s 2024-05-23T00:02:56.364 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gcc option to accept ISO C89... none needed 2024-05-23T00:02:56.375 INFO:tasks.workunit.client.0.smithi103.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-05-23T00:02:56.386 INFO:tasks.workunit.client.0.smithi103.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-05-23T00:02:56.390 INFO:tasks.workunit.client.0.smithi103.stdout:checking for egrep... /usr/bin/grep -E 2024-05-23T00:02:56.394 INFO:tasks.workunit.client.0.smithi103.stdout:checking for fgrep... /usr/bin/grep -F 2024-05-23T00:02:56.407 INFO:tasks.workunit.client.0.smithi103.stdout:checking for ld used by gcc... /usr/bin/ld 2024-05-23T00:02:56.413 INFO:tasks.workunit.client.0.smithi103.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-05-23T00:02:56.422 INFO:tasks.workunit.client.0.smithi103.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-05-23T00:02:56.954 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.950+0000 7fcdae200640 1 -- 172.21.15.103:0/135997957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcda81080d0 msgr2=0x7fcda81084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:56.954 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.950+0000 7fcdae200640 1 --2- 172.21.15.103:0/135997957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcda81080d0 0x7fcda81084b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcd98009900 tx=0x7fcd9802a400 comp rx=0 tx=0).stop 2024-05-23T00:02:56.954 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.950+0000 7fcdae200640 1 -- 172.21.15.103:0/135997957 shutdown_connections 2024-05-23T00:02:56.954 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.950+0000 7fcdae200640 1 --2- 172.21.15.103:0/135997957 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcda810d1a0 0x7fcda8114d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.957 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.950+0000 7fcdae200640 1 --2- 172.21.15.103:0/135997957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcda8108a80 0x7fcda810cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.957 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.950+0000 7fcdae200640 1 --2- 172.21.15.103:0/135997957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcda81080d0 0x7fcda81084b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.957 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.950+0000 7fcdae200640 1 -- 172.21.15.103:0/135997957 >> 172.21.15.103:0/135997957 conn(0x7fcda8075e00 msgr2=0x7fcda8078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.951+0000 7fcdae200640 1 -- 172.21.15.103:0/135997957 shutdown_connections 2024-05-23T00:02:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.951+0000 7fcdae200640 1 -- 172.21.15.103:0/135997957 wait complete. 2024-05-23T00:02:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcdae200640 1 Processor -- start 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcdae200640 1 -- start start 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcdae200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcda81080d0 0x7fcda819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcdae200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcda810d1a0 0x7fcda819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcdae200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcda81a2f20 0x7fcda81a33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcdae200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcda8118280 con 0x7fcda81080d0 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcdae200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcda8118100 con 0x7fcda81a2f20 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcdae200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcda8118400 con 0x7fcda810d1a0 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.952+0000 7fcda7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcda81080d0 0x7fcda819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcda81080d0 0x7fcda819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37020/0 (socket says 172.21.15.103:37020) 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7400640 1 -- 172.21.15.103:0/3448086999 learned_addr learned my addr 172.21.15.103:0/3448086999 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7e00640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcda81a2f20 0x7fcda81a33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda6a00640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcda810d1a0 0x7fcda819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7400640 1 -- 172.21.15.103:0/3448086999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcda810d1a0 msgr2=0x7fcda819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7400640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcda810d1a0 0x7fcda819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7400640 1 -- 172.21.15.103:0/3448086999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcda81a2f20 msgr2=0x7fcda81a33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7400640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcda81a2f20 0x7fcda81a33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7400640 1 -- 172.21.15.103:0/3448086999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcda81a3a10 con 0x7fcda81080d0 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.953+0000 7fcda7400640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcda81080d0 0x7fcda819e660 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcd98004770 tx=0x7fcd98003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.954+0000 7fcd9fe00640 1 -- 172.21.15.103:0/3448086999 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd9803f070 con 0x7fcda81080d0 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.954+0000 7fcd9fe00640 1 -- 172.21.15.103:0/3448086999 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd98031ca0 con 0x7fcda81080d0 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.954+0000 7fcd9fe00640 1 -- 172.21.15.103:0/3448086999 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98035030 con 0x7fcda81080d0 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.954+0000 7fcdae200640 1 -- 172.21.15.103:0/3448086999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcda81a3ca0 con 0x7fcda81080d0 2024-05-23T00:02:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.954+0000 7fcdae200640 1 -- 172.21.15.103:0/3448086999 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcda81ab410 con 0x7fcda81080d0 2024-05-23T00:02:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.958+0000 7fcd9fe00640 1 -- 172.21.15.103:0/3448086999 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fcd98042070 con 0x7fcda81080d0 2024-05-23T00:02:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.958+0000 7fcdb1037740 1 Processor -- start 2024-05-23T00:02:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.958+0000 7fcdb1037740 1 -- start start 2024-05-23T00:02:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 0x5600dd07b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.964 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 0x5600dd085800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd088130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcdb1037740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5600dcfdf2e0 con 0x5600dd07b670 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcdb1037740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5600dcf86e50 con 0x5600dd07ad50 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcdb1037740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5600dcfdb2a0 con 0x5600dd085d40 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 0x5600dd085800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda6a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd088130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda6a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd088130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:56686/0 (socket says 172.21.15.103:56686) 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda6a00640 1 -- 172.21.15.103:0/1616143585 learned_addr learned my addr 172.21.15.103:0/1616143585 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda7e00640 1 --2- 172.21.15.103:0/1616143585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 0x5600dd07b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda6a00640 1 -- 172.21.15.103:0/1616143585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 msgr2=0x5600dd07b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:56.965 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda6a00640 1 --2- 172.21.15.103:0/1616143585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 0x5600dd07b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda6a00640 1 -- 172.21.15.103:0/1616143585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 msgr2=0x5600dd085800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda6a00640 1 --2- 172.21.15.103:0/1616143585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 0x5600dd085800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.959+0000 7fcda6a00640 1 -- 172.21.15.103:0/1616143585 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600dccf3210 con 0x5600dd085d40 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.960+0000 7fcd9fe00640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd8c0a2da0 0x7fcd8c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.960+0000 7fcd9fe00640 1 -- 172.21.15.103:0/3448086999 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fcd980efe80 con 0x7fcda81080d0 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.960+0000 7fcda6a00640 1 --2- 172.21.15.103:0/1616143585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd088130 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcd94009c60 tx=0x7fcd94011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e4cc2d789e53b2c server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.960+0000 7fcda7e00640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd8c0a2da0 0x7fcd8c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.960+0000 7fcd93e00640 1 -- 172.21.15.103:0/1616143585 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94004230 con 0x5600dd085d40 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.960+0000 7fcd93e00640 1 -- 172.21.15.103:0/1616143585 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd940043d0 con 0x5600dd085d40 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.960+0000 7fcd93e00640 1 -- 172.21.15.103:0/1616143585 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd94004dc0 con 0x5600dd085d40 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 -- 172.21.15.103:0/1616143585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 msgr2=0x5600dd088130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1616143585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd088130 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcd94009c60 tx=0x7fcd94011810 comp rx=0 tx=0).stop 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcda7e00640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd8c0a2da0 0x7fcd8c0a5260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcda0005fd0 tx=0x7fcda0009500 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:56.966 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 -- 172.21.15.103:0/1616143585 shutdown_connections 2024-05-23T00:02:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1616143585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd088130 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1616143585 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 0x5600dd085800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1616143585 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 0x5600dd07b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 -- 172.21.15.103:0/1616143585 >> 172.21.15.103:0/1616143585 conn(0x5600dd077070 msgr2=0x5600dd077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 -- 172.21.15.103:0/1616143585 shutdown_connections 2024-05-23T00:02:56.967 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.961+0000 7fcdb1037740 1 -- 172.21.15.103:0/1616143585 wait complete. 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.962+0000 7fcdb1037740 1 -- start start 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.962+0000 7fcdb1037740 1 Processor -- start 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.962+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 0x5600dd082ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.962+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 0x5600dd0831e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.962+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd083720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.962+0000 7fcdb1037740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5600dd088bd0 con 0x5600dd07b670 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.962+0000 7fcdb1037740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5600dcf9a210 con 0x5600dd07ad50 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.962+0000 7fcdb1037740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5600dd088d50 con 0x5600dd085d40 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcdb1037740 10 client.0 _ll_register_callbacks cb 0x5600dcf7d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd083720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda6a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 0x5600dd0831e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd083720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:56694/0 (socket says 172.21.15.103:56694) 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7e00640 1 -- 172.21.15.103:0/3128697213 learned_addr learned my addr 172.21.15.103:0/3128697213 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7400640 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 0x5600dd082ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7e00640 1 -- 172.21.15.103:0/3128697213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 msgr2=0x5600dd082ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7e00640 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 0x5600dd082ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7e00640 1 -- 172.21.15.103:0/3128697213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 msgr2=0x5600dd0831e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7e00640 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 0x5600dd0831e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.963+0000 7fcda7e00640 1 -- 172.21.15.103:0/3128697213 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600dcdbf620 con 0x5600dd085d40 2024-05-23T00:02:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcda7e00640 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd083720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcda00099c0 tx=0x7fcda0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a4b7bd93a3275ff server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcd90c00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcd90c00640 1 -- 172.21.15.103:0/3128697213 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcda0004b20 con 0x5600dd085d40 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcd90c00640 1 -- 172.21.15.103:0/3128697213 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fcda0004cc0 con 0x5600dd085d40 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcdb1037740 1 -- 172.21.15.103:0/3128697213 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd940043d0 con 0x5600dd085d40 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcd90c00640 1 -- 172.21.15.103:0/3128697213 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcda000f7d0 con 0x5600dd085d40 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcdb1037740 10 client.26110 fetch_fsmap learned FSMap version 30 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcdb1037740 1 -- 172.21.15.103:0/3128697213 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5600dd094ec0 con 0x5600dd085d40 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.964+0000 7fcd90c00640 1 -- 172.21.15.103:0/3128697213 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd940043d0 con 0x5600dd085d40 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.965+0000 7fcd90c00640 1 -- 172.21.15.103:0/3128697213 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcda0026070 con 0x5600dd085d40 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.965+0000 7fcdb1037740 10 client.26110 fetch_fsmap finished waiting for FSMap version 30 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.965+0000 7fcdb1037740 10 client.26110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.965+0000 7fcdb1037740 20 client.26110 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.965+0000 7fcdb1037740 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5600dd096440 0x5600dd098850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.965+0000 7fcdb1037740 4 client.26110 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.965+0000 7fcdb1037740 1 -- 172.21.15.103:0/3128697213 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5600dcff2ec0 con 0x5600dd096440 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.965+0000 7fcda7400640 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5600dd096440 0x5600dd098850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.966+0000 7fcda7400640 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5600dd096440 0x5600dd098850 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45f44fb43d697462 server_cookie=7ad1c083d626f862 in_seq=0 out_seq=0 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.966+0000 7fcd90c00640 10 client.26110 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.966+0000 7fcd90c00640 1 -- 172.21.15.103:0/3128697213 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5600dcff2ec0 con 0x5600dd096440 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.966+0000 7fcd90c00640 10 client.26110 handle_command_reply: tid=0 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.966+0000 7fcdb1037740 1 client.26110 shutdown 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.966+0000 7fcdb1037740 1 -- 172.21.15.103:0/3128697213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 msgr2=0x5600dd083720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:56.974 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.966+0000 7fcdb1037740 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd083720 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcda00099c0 tx=0x7fcda0004ab0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 1 -- 172.21.15.103:0/3128697213 shutdown_connections 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5600dd096440 0x5600dd098850 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd083720 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcd90c00640 0 client.26110 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07b670 0x5600dd0831e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 1 --2- 172.21.15.103:0/3128697213 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07ad50 0x5600dd082ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 1 -- 172.21.15.103:0/3128697213 >> 172.21.15.103:0/3128697213 conn(0x5600dd077070 msgr2=0x5600dd0761f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 1 -- 172.21.15.103:0/3128697213 shutdown_connections 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 1 -- 172.21.15.103:0/3128697213 wait complete. 2024-05-23T00:02:56.975 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.967+0000 7fcdb1037740 20 client.26110 trim_cache size 0 max 16384 2024-05-23T00:02:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.984+0000 7fcdb1037740 1 Processor -- start 2024-05-23T00:02:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcdb1037740 1 -- start start 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd091ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07b670 0x5600dd09e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd09e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcdb1037740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81b7cb0 con 0x5600dd07ad50 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcdb1037740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81b5520 con 0x5600dd07b670 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcdb1037740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81b3370 con 0x5600dd085d40 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcda7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd091ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcda7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd091ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37040/0 (socket says 172.21.15.103:37040) 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcda7400640 1 -- 172.21.15.103:0/838227824 learned_addr learned my addr 172.21.15.103:0/838227824 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcda6a00640 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07b670 0x5600dd09e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.989 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcda7e00640 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd09e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcda7400640 1 -- 172.21.15.103:0/838227824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 msgr2=0x5600dd09e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.985+0000 7fcda7400640 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd09e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.986+0000 7fcda7400640 1 -- 172.21.15.103:0/838227824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07b670 msgr2=0x5600dd09e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.986+0000 7fcda7400640 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07b670 0x5600dd09e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.986+0000 7fcda7400640 1 -- 172.21.15.103:0/838227824 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600dd092f70 con 0x5600dd07ad50 2024-05-23T00:02:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.986+0000 7fcda6a00640 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07b670 0x5600dd09e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:02:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.986+0000 7fcda7400640 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd091ac0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcd980ed1e0 tx=0x7fcd980f3930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c854729e9f139b4b server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.986+0000 7fcd93400640 1 -- 172.21.15.103:0/838227824 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98049040 con 0x5600dd07ad50 2024-05-23T00:02:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.986+0000 7fcd93400640 1 -- 172.21.15.103:0/838227824 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd980bc0c0 con 0x5600dd07ad50 2024-05-23T00:02:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.986+0000 7fcd93400640 1 -- 172.21.15.103:0/838227824 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd980b8730 con 0x5600dd07ad50 2024-05-23T00:02:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 -- 172.21.15.103:0/838227824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 msgr2=0x5600dd091ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd091ac0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fcd980ed1e0 tx=0x7fcd980f3930 comp rx=0 tx=0).stop 2024-05-23T00:02:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 -- 172.21.15.103:0/838227824 shutdown_connections 2024-05-23T00:02:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd085d40 0x5600dd09e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd07b670 0x5600dd09e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 --2- 172.21.15.103:0/838227824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd091ac0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 -- 172.21.15.103:0/838227824 >> 172.21.15.103:0/838227824 conn(0x5600dd077070 msgr2=0x5600dd081570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 -- 172.21.15.103:0/838227824 shutdown_connections 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 -- 172.21.15.103:0/838227824 wait complete. 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.987+0000 7fcdb1037740 1 -- start start 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcdb1037740 1 Processor -- start 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd08e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd07b670 0x5600dd0895f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcdb1037740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd085d40 0x5600dd089b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcdb1037740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81b39e0 con 0x5600dd07ad50 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcdb1037740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81b4e50 con 0x5600dd085d40 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcdb1037740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcda81b40b0 con 0x5600dd07b670 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcdb1037740 10 client.0 _ll_register_callbacks cb 0x7fcda0003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcda7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd08e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcda7400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd08e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:37054/0 (socket says 172.21.15.103:37054) 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.988+0000 7fcda7400640 1 -- 172.21.15.103:0/1971155001 learned_addr learned my addr 172.21.15.103:0/1971155001 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda6a00640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd07b670 0x5600dd0895f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda7e00640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd085d40 0x5600dd089b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda7400640 1 -- 172.21.15.103:0/1971155001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd07b670 msgr2=0x5600dd0895f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda7400640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd07b670 0x5600dd0895f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda7400640 1 -- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd085d40 msgr2=0x5600dd089b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:02:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda7400640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd085d40 0x5600dd089b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda7400640 1 -- 172.21.15.103:0/1971155001 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600dd08a220 con 0x5600dd07ad50 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda6a00640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd07b670 0x5600dd0895f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda7e00640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd085d40 0x5600dd089b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcda7400640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd08e280 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcd9803b8d0 tx=0x7fcd98005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94afe294fbf2db1 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcd90c00640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcd90c00640 1 -- 172.21.15.103:0/1971155001 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd98049070 con 0x5600dd07ad50 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcd90c00640 1 -- 172.21.15.103:0/1971155001 <== mon.0 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7fcd980b8730 con 0x5600dd07ad50 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcdb1037740 1 -- 172.21.15.103:0/1971155001 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd980bc0c0 con 0x5600dd07ad50 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcd90c00640 1 -- 172.21.15.103:0/1971155001 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fcd980c4c80 con 0x5600dd07ad50 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcdb1037740 10 client.16002 fetch_fsmap learned FSMap version 30 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.989+0000 7fcdb1037740 1 -- 172.21.15.103:0/1971155001 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5600dd0fc8e0 con 0x5600dd07ad50 2024-05-23T00:02:56.997 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcd90c00640 1 -- 172.21.15.103:0/1971155001 <== mon.0 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd980bc0c0 con 0x5600dd07ad50 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcd90c00640 1 -- 172.21.15.103:0/1971155001 <== mon.0 v2:172.21.15.103:3300/0 5 ==== fsmap(e 30) ==== 8362+0+0 (secure 0 0 0) 0x7fcd980bc320 con 0x5600dd07ad50 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcdb1037740 10 client.16002 fetch_fsmap finished waiting for FSMap version 30 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcdb1037740 10 client.16002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcdb1037740 20 client.16002 populate_metadata read hostname 'smithi103' 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5600dd0fce80 0x5600dd0ff2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcdb1037740 4 client.16002 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcdb1037740 1 -- 172.21.15.103:0/1971155001 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5600dcfeca10 con 0x5600dd0fce80 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.990+0000 7fcda6a00640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5600dd0fce80 0x5600dd0ff2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.991+0000 7fcd90c00640 10 client.16002 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:56.998 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:56.991+0000 7fcda6a00640 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5600dd0fce80 0x5600dd0ff2a0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88f361f8a17ed35a server_cookie=76199e1fbf53d6a3 in_seq=0 out_seq=0 2024-05-23T00:02:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.323+0000 7fcd90c00640 1 -- 172.21.15.103:0/1971155001 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5600dcfeca10 con 0x5600dd0fce80 2024-05-23T00:02:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.323+0000 7fcd90c00640 10 client.16002 handle_command_reply: tid=0 2024-05-23T00:02:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.323+0000 7fcdb1037740 1 client.16002 shutdown 2024-05-23T00:02:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.323+0000 7fcdb1037740 1 -- 172.21.15.103:0/1971155001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 msgr2=0x5600dd08e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.323+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd08e280 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcd9803b8d0 tx=0x7fcd98005d50 comp rx=0 tx=0).stop 2024-05-23T00:02:57.328 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 1 -- 172.21.15.103:0/1971155001 shutdown_connections 2024-05-23T00:02:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5600dd0fce80 0x5600dd0ff2a0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5600dd085d40 0x5600dd089b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5600dd07b670 0x5600dd0895f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 1 --2- 172.21.15.103:0/1971155001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5600dd07ad50 0x5600dd08e280 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcd90c00640 0 client.16002 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:02:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 1 -- 172.21.15.103:0/1971155001 >> 172.21.15.103:0/1971155001 conn(0x5600dd077070 msgr2=0x5600dd075960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:57.329 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 1 -- 172.21.15.103:0/1971155001 shutdown_connections 2024-05-23T00:02:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 1 -- 172.21.15.103:0/1971155001 wait complete. 2024-05-23T00:02:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.324+0000 7fcdb1037740 20 client.16002 trim_cache size 0 max 16384 2024-05-23T00:02:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:02:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:02:57.330 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:02:57.331 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:02:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 -- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd8c0a2da0 msgr2=0x7fcd8c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd8c0a2da0 0x7fcd8c0a5260 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcda0005fd0 tx=0x7fcda0009500 comp rx=0 tx=0).stop 2024-05-23T00:02:57.332 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 -- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcda81080d0 msgr2=0x7fcda819e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:02:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcda81080d0 0x7fcda819e660 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcd98004770 tx=0x7fcd98003c10 comp rx=0 tx=0).stop 2024-05-23T00:02:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 -- 172.21.15.103:0/3448086999 shutdown_connections 2024-05-23T00:02:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fcd8c0a2da0 0x7fcd8c0a5260 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcda81a2f20 0x7fcda81a33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcda810d1a0 0x7fcda819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:57.333 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 --2- 172.21.15.103:0/3448086999 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcda81080d0 0x7fcda819e660 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:02:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 -- 172.21.15.103:0/3448086999 >> 172.21.15.103:0/3448086999 conn(0x7fcda8075e00 msgr2=0x7fcda8076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:02:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 -- 172.21.15.103:0/3448086999 shutdown_connections 2024-05-23T00:02:57.334 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:02:57.325+0000 7fcdae200640 1 -- 172.21.15.103:0/3448086999 wait complete. 2024-05-23T00:02:57.977 INFO:tasks.workunit.client.0.smithi103.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-05-23T00:02:57.977 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether ln -s works... yes 2024-05-23T00:02:58.036 INFO:tasks.workunit.client.0.smithi103.stdout:checking the maximum length of command line arguments... 1572864 2024-05-23T00:02:58.046 INFO:tasks.workunit.client.0.smithi103.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-05-23T00:02:58.047 INFO:tasks.workunit.client.0.smithi103.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-05-23T00:02:58.047 INFO:tasks.workunit.client.0.smithi103.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-05-23T00:02:58.048 INFO:tasks.workunit.client.0.smithi103.stdout:checking for objdump... objdump 2024-05-23T00:02:58.049 INFO:tasks.workunit.client.0.smithi103.stdout:checking how to recognize dependent libraries... pass_all 2024-05-23T00:02:58.051 INFO:tasks.workunit.client.0.smithi103.stdout:checking for dlltool... no 2024-05-23T00:02:58.051 INFO:tasks.workunit.client.0.smithi103.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-05-23T00:02:58.053 INFO:tasks.workunit.client.0.smithi103.stdout:checking for ar... ar 2024-05-23T00:02:58.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:57 smithi103 ceph-mon[31370]: pgmap v77: 129 pgs: 129 active+clean; 57 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 202 op/s 2024-05-23T00:02:58.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:57 smithi174 ceph-mon[36179]: pgmap v77: 129 pgs: 129 active+clean; 57 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 202 op/s 2024-05-23T00:02:58.214 INFO:tasks.workunit.client.0.smithi103.stdout:checking for archiver @FILE support... @ 2024-05-23T00:02:58.215 INFO:tasks.workunit.client.0.smithi103.stdout:checking for strip... strip 2024-05-23T00:02:58.216 INFO:tasks.workunit.client.0.smithi103.stdout:checking for ranlib... ranlib 2024-05-23T00:02:58.217 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gawk... gawk 2024-05-23T00:02:58.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:57 smithi105 ceph-mon[36426]: pgmap v77: 129 pgs: 129 active+clean; 57 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 202 op/s 2024-05-23T00:02:58.455 INFO:tasks.workunit.client.0.smithi103.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-05-23T00:02:58.460 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sysroot... no 2024-05-23T00:02:58.483 INFO:tasks.workunit.client.0.smithi103.stdout:checking for a working dd... /usr/bin/dd 2024-05-23T00:02:58.506 INFO:tasks.workunit.client.0.smithi103.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-05-23T00:02:58.575 INFO:tasks.workunit.client.0.smithi103.stdout:checking for mt... no 2024-05-23T00:02:58.607 INFO:tasks.workunit.client.0.smithi103.stdout:checking if : is a manifest tool... no 2024-05-23T00:02:58.771 INFO:tasks.workunit.client.0.smithi103.stdout:checking how to run the C preprocessor... gcc -E 2024-05-23T00:02:59.165 INFO:tasks.workunit.client.0.smithi103.stdout:checking for ANSI C header files... yes 2024-05-23T00:02:59.243 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/types.h... yes 2024-05-23T00:02:59.329 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/stat.h... yes 2024-05-23T00:02:59.419 INFO:tasks.workunit.client.0.smithi103.stdout:checking for stdlib.h... yes 2024-05-23T00:02:59.509 INFO:tasks.workunit.client.0.smithi103.stdout:checking for string.h... yes 2024-05-23T00:02:59.606 INFO:tasks.workunit.client.0.smithi103.stdout:checking for memory.h... yes 2024-05-23T00:02:59.690 INFO:tasks.workunit.client.0.smithi103.stdout:checking for strings.h... yes 2024-05-23T00:02:59.781 INFO:tasks.workunit.client.0.smithi103.stdout:checking for inttypes.h... yes 2024-05-23T00:02:59.876 INFO:tasks.workunit.client.0.smithi103.stdout:checking for stdint.h... yes 2024-05-23T00:02:59.962 INFO:tasks.workunit.client.0.smithi103.stdout:checking for unistd.h... yes 2024-05-23T00:03:00.073 INFO:tasks.workunit.client.0.smithi103.stdout:checking for dlfcn.h... yes 2024-05-23T00:03:00.087 INFO:tasks.workunit.client.0.smithi103.stdout:checking for objdir... .libs 2024-05-23T00:03:00.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:02:59 smithi174 ceph-mon[36179]: pgmap v78: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 425 op/s 2024-05-23T00:03:00.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:02:59 smithi105 ceph-mon[36426]: pgmap v78: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 425 op/s 2024-05-23T00:03:00.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:02:59 smithi103 ceph-mon[31370]: pgmap v78: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 425 op/s 2024-05-23T00:03:00.338 INFO:tasks.workunit.client.0.smithi103.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-05-23T00:03:00.339 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-05-23T00:03:00.426 INFO:tasks.workunit.client.0.smithi103.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-05-23T00:03:00.516 INFO:tasks.workunit.client.0.smithi103.stdout:checking if gcc static flag -static works... no 2024-05-23T00:03:00.635 INFO:tasks.workunit.client.0.smithi103.stdout:checking if gcc supports -c -o file.o... yes 2024-05-23T00:03:00.635 INFO:tasks.workunit.client.0.smithi103.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-05-23T00:03:00.658 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-05-23T00:03:00.755 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether -lc should be explicitly linked in... no 2024-05-23T00:03:00.897 INFO:tasks.workunit.client.0.smithi103.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-05-23T00:03:00.898 INFO:tasks.workunit.client.0.smithi103.stdout:checking how to hardcode library paths into programs... immediate 2024-05-23T00:03:00.902 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether stripping libraries is possible... yes 2024-05-23T00:03:00.902 INFO:tasks.workunit.client.0.smithi103.stdout:checking if libtool supports shared libraries... yes 2024-05-23T00:03:00.902 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether to build shared libraries... yes 2024-05-23T00:03:00.902 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether to build static libraries... yes 2024-05-23T00:03:00.921 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gcc... (cached) gcc 2024-05-23T00:03:01.015 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-05-23T00:03:01.016 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether gcc accepts -g... (cached) yes 2024-05-23T00:03:01.018 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-05-23T00:03:01.019 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gmake... /usr/bin/gmake 2024-05-23T00:03:01.020 INFO:tasks.workunit.client.0.smithi103.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-05-23T00:03:01.020 INFO:tasks.workunit.client.0.smithi103.stdout:checking for tar... /bin/tar 2024-05-23T00:03:01.022 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gzip... /bin/gzip 2024-05-23T00:03:01.077 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether gcc -MM is supported... yes 2024-05-23T00:03:01.078 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sort... /bin/sort 2024-05-23T00:03:01.079 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether ln -s works... yes 2024-05-23T00:03:01.080 INFO:tasks.workunit.client.0.smithi103.stdout:checking for rpm... /bin/rpm 2024-05-23T00:03:01.093 INFO:tasks.workunit.client.0.smithi103.stdout:checking for rpmbuild... no 2024-05-23T00:03:01.093 INFO:tasks.workunit.client.0.smithi103.stdout:checking for ANSI C header files... (cached) yes 2024-05-23T00:03:01.197 INFO:tasks.workunit.client.0.smithi103.stdout:checking assert.h usability... yes 2024-05-23T00:03:01.235 INFO:tasks.workunit.client.0.smithi103.stdout:checking assert.h presence... yes 2024-05-23T00:03:01.236 INFO:tasks.workunit.client.0.smithi103.stdout:checking for assert.h... yes 2024-05-23T00:03:01.352 INFO:tasks.workunit.client.0.smithi103.stdout:checking bstring.h usability... no 2024-05-23T00:03:01.416 INFO:tasks.workunit.client.0.smithi103.stdout:checking bstring.h presence... no 2024-05-23T00:03:01.416 INFO:tasks.workunit.client.0.smithi103.stdout:checking for bstring.h... no 2024-05-23T00:03:01.502 INFO:tasks.workunit.client.0.smithi103.stdout:checking libgen.h usability... yes 2024-05-23T00:03:01.539 INFO:tasks.workunit.client.0.smithi103.stdout:checking libgen.h presence... yes 2024-05-23T00:03:01.539 INFO:tasks.workunit.client.0.smithi103.stdout:checking for libgen.h... yes 2024-05-23T00:03:01.647 INFO:tasks.workunit.client.0.smithi103.stdout:checking dirent.h usability... yes 2024-05-23T00:03:01.686 INFO:tasks.workunit.client.0.smithi103.stdout:checking dirent.h presence... yes 2024-05-23T00:03:01.687 INFO:tasks.workunit.client.0.smithi103.stdout:checking for dirent.h... yes 2024-05-23T00:03:01.787 INFO:tasks.workunit.client.0.smithi103.stdout:checking errno.h usability... yes 2024-05-23T00:03:01.826 INFO:tasks.workunit.client.0.smithi103.stdout:checking errno.h presence... yes 2024-05-23T00:03:01.826 INFO:tasks.workunit.client.0.smithi103.stdout:checking for errno.h... yes 2024-05-23T00:03:01.948 INFO:tasks.workunit.client.0.smithi103.stdout:checking malloc.h usability... yes 2024-05-23T00:03:01.990 INFO:tasks.workunit.client.0.smithi103.stdout:checking malloc.h presence... yes 2024-05-23T00:03:01.990 INFO:tasks.workunit.client.0.smithi103.stdout:checking for malloc.h... yes 2024-05-23T00:03:02.104 INFO:tasks.workunit.client.0.smithi103.stdout:checking uuid.h usability... no 2024-05-23T00:03:02.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:01 smithi174 ceph-mon[36179]: pgmap v79: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 641 KiB/s wr, 296 op/s 2024-05-23T00:03:02.170 INFO:tasks.workunit.client.0.smithi103.stdout:checking uuid.h presence... no 2024-05-23T00:03:02.170 INFO:tasks.workunit.client.0.smithi103.stdout:checking for uuid.h... no 2024-05-23T00:03:02.270 INFO:tasks.workunit.client.0.smithi103.stdout:checking uuid/uuid.h usability... yes 2024-05-23T00:03:02.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:01 smithi105 ceph-mon[36426]: pgmap v79: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 641 KiB/s wr, 296 op/s 2024-05-23T00:03:02.311 INFO:tasks.workunit.client.0.smithi103.stdout:checking uuid/uuid.h presence... yes 2024-05-23T00:03:02.312 INFO:tasks.workunit.client.0.smithi103.stdout:checking for uuid/uuid.h... yes 2024-05-23T00:03:02.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:01 smithi103 ceph-mon[31370]: pgmap v79: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 641 KiB/s wr, 296 op/s 2024-05-23T00:03:02.419 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/uuid.h usability... no 2024-05-23T00:03:02.484 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/uuid.h presence... no 2024-05-23T00:03:02.484 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/uuid.h... no 2024-05-23T00:03:02.588 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/file.h usability... yes 2024-05-23T00:03:02.627 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/file.h presence... yes 2024-05-23T00:03:02.627 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/file.h... yes 2024-05-23T00:03:02.729 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/fcntl.h usability... yes 2024-05-23T00:03:02.768 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/fcntl.h presence... yes 2024-05-23T00:03:02.768 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/fcntl.h... yes 2024-05-23T00:03:02.878 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/syssgi.h usability... no 2024-05-23T00:03:02.943 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/syssgi.h presence... no 2024-05-23T00:03:02.943 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/syssgi.h... no 2024-05-23T00:03:03.172 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/param.h usability... yes 2024-05-23T00:03:03.219 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/param.h presence... yes 2024-05-23T00:03:03.220 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/param.h... yes 2024-05-23T00:03:03.253 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/stat.h... (cached) yes 2024-05-23T00:03:03.345 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/statvfs.h usability... yes 2024-05-23T00:03:03.384 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/statvfs.h presence... yes 2024-05-23T00:03:03.385 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/statvfs.h... yes 2024-05-23T00:03:03.518 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/time.h usability... yes 2024-05-23T00:03:03.586 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/time.h presence... yes 2024-05-23T00:03:03.586 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/time.h... yes 2024-05-23T00:03:04.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:03 smithi174 ceph-mon[36179]: pgmap v80: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 641 KiB/s wr, 296 op/s 2024-05-23T00:03:04.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:03 smithi105 ceph-mon[36426]: pgmap v80: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 641 KiB/s wr, 296 op/s 2024-05-23T00:03:04.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:03 smithi103 ceph-mon[31370]: pgmap v80: 129 pgs: 129 active+clean; 58 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 641 KiB/s wr, 296 op/s 2024-05-23T00:03:04.539 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/ioctl.h usability... yes 2024-05-23T00:03:04.570 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/ioctl.h presence... yes 2024-05-23T00:03:04.570 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/ioctl.h... yes 2024-05-23T00:03:04.926 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/wait.h usability... yes 2024-05-23T00:03:04.969 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/wait.h presence... yes 2024-05-23T00:03:04.969 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/wait.h... yes 2024-05-23T00:03:05.002 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/types.h... (cached) yes 2024-05-23T00:03:05.020 INFO:tasks.workunit.client.0.smithi103.stdout:checking for strings.h... (cached) yes 2024-05-23T00:03:05.103 INFO:tasks.workunit.client.0.smithi103.stdout:checking err.h usability... yes 2024-05-23T00:03:05.139 INFO:tasks.workunit.client.0.smithi103.stdout:checking err.h presence... yes 2024-05-23T00:03:05.139 INFO:tasks.workunit.client.0.smithi103.stdout:checking for err.h... yes 2024-05-23T00:03:05.244 INFO:tasks.workunit.client.0.smithi103.stdout:checking linux/falloc.h usability... yes 2024-05-23T00:03:05.282 INFO:tasks.workunit.client.0.smithi103.stdout:checking linux/falloc.h presence... yes 2024-05-23T00:03:05.283 INFO:tasks.workunit.client.0.smithi103.stdout:checking for linux/falloc.h... yes 2024-05-23T00:03:05.394 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-05-23T00:03:05.458 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-05-23T00:03:05.458 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/fs/xfs_fsops.h... no 2024-05-23T00:03:05.555 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/fs/xfs_itable.h usability... no 2024-05-23T00:03:05.621 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/fs/xfs_itable.h presence... no 2024-05-23T00:03:05.621 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/fs/xfs_itable.h... no 2024-05-23T00:03:05.717 INFO:tasks.workunit.client.0.smithi103.stdout:checking xfs/platform_defs.h usability... no 2024-05-23T00:03:05.779 INFO:tasks.workunit.client.0.smithi103.stdout:checking xfs/platform_defs.h presence... no 2024-05-23T00:03:05.779 INFO:tasks.workunit.client.0.smithi103.stdout:checking for xfs/platform_defs.h... no 2024-05-23T00:03:05.875 INFO:tasks.workunit.client.0.smithi103.stdout:checking btrfs/ioctl.h usability... no 2024-05-23T00:03:05.939 INFO:tasks.workunit.client.0.smithi103.stdout:checking btrfs/ioctl.h presence... no 2024-05-23T00:03:05.939 INFO:tasks.workunit.client.0.smithi103.stdout:checking for btrfs/ioctl.h... no 2024-05-23T00:03:06.037 INFO:tasks.workunit.client.0.smithi103.stdout:checking cifs/ioctl.h usability... no 2024-05-23T00:03:06.098 INFO:tasks.workunit.client.0.smithi103.stdout:checking cifs/ioctl.h presence... no 2024-05-23T00:03:06.098 INFO:tasks.workunit.client.0.smithi103.stdout:checking for cifs/ioctl.h... no 2024-05-23T00:03:06.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:05 smithi174 ceph-mon[36179]: pgmap v81: 129 pgs: 129 active+clean; 64 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 414 op/s 2024-05-23T00:03:06.207 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/mman.h usability... yes 2024-05-23T00:03:06.252 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/mman.h presence... yes 2024-05-23T00:03:06.252 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/mman.h... yes 2024-05-23T00:03:06.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:05 smithi105 ceph-mon[36426]: pgmap v81: 129 pgs: 129 active+clean; 64 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 414 op/s 2024-05-23T00:03:06.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:05 smithi103 ceph-mon[31370]: pgmap v81: 129 pgs: 129 active+clean; 64 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 1.2 MiB/s wr, 414 op/s 2024-05-23T00:03:06.337 INFO:tasks.workunit.client.0.smithi103.stdout:checking for xfs/xfs_log_format.h... no 2024-05-23T00:03:06.470 INFO:tasks.workunit.client.0.smithi103.stdout:checking for uuid_compare... no 2024-05-23T00:03:06.567 INFO:tasks.workunit.client.0.smithi103.stdout:checking for uuid_compare in -luuid... yes 2024-05-23T00:03:06.699 INFO:tasks.workunit.client.0.smithi103.stdout:checking for xfs/xfs.h... yes 2024-05-23T00:03:06.774 INFO:tasks.workunit.client.0.smithi103.stdout:checking for xfs/libxfs.h... no 2024-05-23T00:03:06.852 INFO:tasks.workunit.client.0.smithi103.stdout:checking whether xlog_assign_lsn is declared... no 2024-05-23T00:03:06.937 INFO:tasks.workunit.client.0.smithi103.stdout:checking for xfs/xqm.h... yes 2024-05-23T00:03:07.057 INFO:tasks.workunit.client.0.smithi103.stdout:checking xfsctl from xfs/xfs.h... ok 2024-05-23T00:03:07.147 INFO:tasks.workunit.client.0.smithi103.stdout:checking xfs/handle.h usability... yes 2024-05-23T00:03:07.214 INFO:tasks.workunit.client.0.smithi103.stdout:checking xfs/handle.h presence... yes 2024-05-23T00:03:07.215 INFO:tasks.workunit.client.0.smithi103.stdout:checking for xfs/handle.h... yes 2024-05-23T00:03:07.324 INFO:tasks.workunit.client.0.smithi103.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-05-23T00:03:07.423 INFO:tasks.workunit.client.0.smithi103.stdout:checking attr/attributes.h usability... yes 2024-05-23T00:03:07.466 INFO:tasks.workunit.client.0.smithi103.stdout:checking attr/attributes.h presence... yes 2024-05-23T00:03:07.467 INFO:tasks.workunit.client.0.smithi103.stdout:checking for attr/attributes.h... yes 2024-05-23T00:03:07.576 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/attributes.h usability... no 2024-05-23T00:03:07.642 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/attributes.h presence... no 2024-05-23T00:03:07.642 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/attributes.h... no 2024-05-23T00:03:07.733 INFO:tasks.workunit.client.0.smithi103.stdout:checking for attr_list in -lattr... yes 2024-05-23T00:03:07.827 INFO:tasks.workunit.client.0.smithi103.stdout:checking for attr_set in -lattr... yes 2024-05-23T00:03:07.908 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/acl.h usability... yes 2024-05-23T00:03:07.948 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/acl.h presence... yes 2024-05-23T00:03:07.948 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/acl.h... yes 2024-05-23T00:03:08.028 INFO:tasks.workunit.client.0.smithi103.stdout:checking acl/libacl.h usability... yes 2024-05-23T00:03:08.069 INFO:tasks.workunit.client.0.smithi103.stdout:checking acl/libacl.h presence... yes 2024-05-23T00:03:08.069 INFO:tasks.workunit.client.0.smithi103.stdout:checking for acl/libacl.h... yes 2024-05-23T00:03:08.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:07 smithi174 ceph-mon[36179]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 685 KiB/s wr, 341 op/s 2024-05-23T00:03:08.166 INFO:tasks.workunit.client.0.smithi103.stdout:checking for acl_init in -lacl... yes 2024-05-23T00:03:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:07 smithi105 ceph-mon[36426]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 685 KiB/s wr, 341 op/s 2024-05-23T00:03:08.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:07 smithi103 ceph-mon[31370]: pgmap v82: 129 pgs: 129 active+clean; 64 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 685 KiB/s wr, 341 op/s 2024-05-23T00:03:08.338 INFO:tasks.workunit.client.0.smithi103.stdout:checking gdbm-ndbm.h usability... no 2024-05-23T00:03:08.404 INFO:tasks.workunit.client.0.smithi103.stdout:checking gdbm-ndbm.h presence... no 2024-05-23T00:03:08.404 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gdbm-ndbm.h... no 2024-05-23T00:03:08.491 INFO:tasks.workunit.client.0.smithi103.stdout:checking gdbm/ndbm.h usability... no 2024-05-23T00:03:08.551 INFO:tasks.workunit.client.0.smithi103.stdout:checking gdbm/ndbm.h presence... no 2024-05-23T00:03:08.552 INFO:tasks.workunit.client.0.smithi103.stdout:checking for gdbm/ndbm.h... no 2024-05-23T00:03:08.935 INFO:tasks.workunit.client.0.smithi103.stdout:checking ndbm.h usability... no 2024-05-23T00:03:09.058 INFO:tasks.workunit.client.0.smithi103.stdout:checking ndbm.h presence... no 2024-05-23T00:03:09.058 INFO:tasks.workunit.client.0.smithi103.stdout:checking for ndbm.h... no 2024-05-23T00:03:09.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:08 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:09.132 INFO:tasks.workunit.client.0.smithi103.stdout:checking libaio.h usability... yes 2024-05-23T00:03:09.175 INFO:tasks.workunit.client.0.smithi103.stdout:checking libaio.h presence... yes 2024-05-23T00:03:09.175 INFO:tasks.workunit.client.0.smithi103.stdout:checking for libaio.h... yes 2024-05-23T00:03:09.286 INFO:tasks.workunit.client.0.smithi103.stdout:checking for dm_make_handle in -ldm... no 2024-05-23T00:03:09.286 INFO:tasks.workunit.client.0.smithi103.stdout: 2024-05-23T00:03:09.286 INFO:tasks.workunit.client.0.smithi103.stdout:WARNING: could not find a valid DMAPI base library. 2024-05-23T00:03:09.286 INFO:tasks.workunit.client.0.smithi103.stdout:If you want DMAPI support please install the data migration 2024-05-23T00:03:09.286 INFO:tasks.workunit.client.0.smithi103.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-05-23T00:03:09.286 INFO:tasks.workunit.client.0.smithi103.stdout:from the dmapi source. 2024-05-23T00:03:09.286 INFO:tasks.workunit.client.0.smithi103.stdout: 2024-05-23T00:03:09.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:08 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:09.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:08 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:09.362 INFO:tasks.workunit.client.0.smithi103.stdout:checking linux/fiemap.h usability... yes 2024-05-23T00:03:09.396 INFO:tasks.workunit.client.0.smithi103.stdout:checking linux/fiemap.h presence... yes 2024-05-23T00:03:09.396 INFO:tasks.workunit.client.0.smithi103.stdout:checking for linux/fiemap.h... yes 2024-05-23T00:03:09.499 INFO:tasks.workunit.client.0.smithi103.stdout:checking for fallocate... yes 2024-05-23T00:03:09.630 INFO:tasks.workunit.client.0.smithi103.stdout:checking for open_by_handle_at... yes 2024-05-23T00:03:09.705 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/prctl.h usability... yes 2024-05-23T00:03:09.738 INFO:tasks.workunit.client.0.smithi103.stdout:checking sys/prctl.h presence... yes 2024-05-23T00:03:09.738 INFO:tasks.workunit.client.0.smithi103.stdout:checking for sys/prctl.h... yes 2024-05-23T00:03:09.810 INFO:tasks.workunit.client.0.smithi103.stdout:checking linux/fs.h usability... yes 2024-05-23T00:03:09.848 INFO:tasks.workunit.client.0.smithi103.stdout:checking linux/fs.h presence... yes 2024-05-23T00:03:09.848 INFO:tasks.workunit.client.0.smithi103.stdout:checking for linux/fs.h... yes 2024-05-23T00:03:09.958 INFO:tasks.workunit.client.0.smithi103.stdout:checking btrfsutil.h usability... no 2024-05-23T00:03:10.025 INFO:tasks.workunit.client.0.smithi103.stdout:checking btrfsutil.h presence... no 2024-05-23T00:03:10.025 INFO:tasks.workunit.client.0.smithi103.stdout:checking for btrfsutil.h... no 2024-05-23T00:03:10.123 INFO:tasks.workunit.client.0.smithi103.stdout:checking for copy_file_range... yes 2024-05-23T00:03:10.223 INFO:tasks.workunit.client.0.smithi103.stdout:checking for renameat2... yes 2024-05-23T00:03:10.267 INFO:tasks.workunit.client.0.smithi103.stdout:configure: creating ./config.status 2024-05-23T00:03:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:09 smithi105 ceph-mon[36426]: pgmap v83: 129 pgs: 129 active+clean; 73 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 1.5 MiB/s wr, 396 op/s 2024-05-23T00:03:10.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:09 smithi103 ceph-mon[31370]: pgmap v83: 129 pgs: 129 active+clean; 73 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 1.5 MiB/s wr, 396 op/s 2024-05-23T00:03:10.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:09 smithi174 ceph-mon[36179]: pgmap v83: 129 pgs: 129 active+clean; 73 MiB data, 517 MiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 1.5 MiB/s wr, 396 op/s 2024-05-23T00:03:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 -- 172.21.15.103:0/1155413195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0108a80 msgr2=0x7f9fe010cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.461 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 --2- 172.21.15.103:0/1155413195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0108a80 0x7f9fe010cac0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009860 tx=0x7f9fc802a340 comp rx=0 tx=0).stop 2024-05-23T00:03:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 -- 172.21.15.103:0/1155413195 shutdown_connections 2024-05-23T00:03:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 --2- 172.21.15.103:0/1155413195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe010d1a0 0x7f9fe0114d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 --2- 172.21.15.103:0/1155413195 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe0108a80 0x7f9fe010cac0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 --2- 172.21.15.103:0/1155413195 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe01080d0 0x7f9fe01084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 -- 172.21.15.103:0/1155413195 >> 172.21.15.103:0/1155413195 conn(0x7f9fe0075e00 msgr2=0x7f9fe0078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 -- 172.21.15.103:0/1155413195 shutdown_connections 2024-05-23T00:03:10.462 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.458+0000 7f9fe6800640 1 -- 172.21.15.103:0/1155413195 wait complete. 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.459+0000 7f9fe6800640 1 Processor -- start 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.459+0000 7f9fe6800640 1 -- start start 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fe6800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe01080d0 0x7f9fe019e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fe6800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe010d1a0 0x7f9fe019ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fe6800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe01a2fe0 0x7f9fe01a3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fe6800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0118210 con 0x7f9fe01a2fe0 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fe6800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0118090 con 0x7f9fe010d1a0 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fe6800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe0118390 con 0x7f9fe01080d0 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe01080d0 0x7f9fe019e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe01080d0 0x7f9fe019e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:47274/0 (socket says 172.21.15.103:47274) 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fdf400640 1 -- 172.21.15.103:0/3599010484 learned_addr learned my addr 172.21.15.103:0/3599010484 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:10.464 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe010d1a0 0x7f9fe019ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.460+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe01a2fe0 0x7f9fe01a3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fd6a00640 1 -- 172.21.15.103:0/3599010484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe01080d0 msgr2=0x7f9fe019e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe01080d0 0x7f9fe019e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fd6a00640 1 -- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe01a2fe0 msgr2=0x7f9fe01a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe01a2fe0 0x7f9fe01a3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fd6a00640 1 -- 172.21.15.103:0/3599010484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fe01a3ad0 con 0x7f9fe010d1a0 2024-05-23T00:03:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fdf400640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe01080d0 0x7f9fe019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:03:10.465 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe010d1a0 0x7f9fe019ecb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9fc802a860 tx=0x7f9fc8033b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:10.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fdcc00640 1 -- 172.21.15.103:0/3599010484 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8037040 con 0x7f9fe010d1a0 2024-05-23T00:03:10.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fdcc00640 1 -- 172.21.15.103:0/3599010484 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fc8033d50 con 0x7f9fe010d1a0 2024-05-23T00:03:10.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fe6800640 1 -- 172.21.15.103:0/3599010484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fe01a3d60 con 0x7f9fe010d1a0 2024-05-23T00:03:10.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.461+0000 7f9fdcc00640 1 -- 172.21.15.103:0/3599010484 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc8003920 con 0x7f9fe010d1a0 2024-05-23T00:03:10.466 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.462+0000 7f9fe6800640 1 -- 172.21.15.103:0/3599010484 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fe01ab520 con 0x7f9fe010d1a0 2024-05-23T00:03:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.465+0000 7f9fdcc00640 1 -- 172.21.15.103:0/3599010484 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f9fc8042020 con 0x7f9fe010d1a0 2024-05-23T00:03:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.465+0000 7f9fe9674740 1 Processor -- start 2024-05-23T00:03:10.468 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.465+0000 7f9fe9674740 1 -- start start 2024-05-23T00:03:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75021d50 0x556a75022130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a7502c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fe9674740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556a74f862e0 con 0x556a7502cd40 2024-05-23T00:03:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fe9674740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556a74f2de50 con 0x556a75021d50 2024-05-23T00:03:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fe9674740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556a74f822a0 con 0x556a75022670 2024-05-23T00:03:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75021d50 0x556a75022130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.470 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75021d50 0x556a75022130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:47144/0 (socket says 172.21.15.103:47144) 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fdfe00640 1 -- 172.21.15.103:0/4017701047 learned_addr learned my addr 172.21.15.103:0/4017701047 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fdf400640 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a7502c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fdcc00640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2da0 0x7f9fc40a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fdfe00640 1 -- 172.21.15.103:0/4017701047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 msgr2=0x556a7502c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a7502c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.466+0000 7f9fdfe00640 1 -- 172.21.15.103:0/4017701047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 msgr2=0x556a7502f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.467+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.467+0000 7f9fdfe00640 1 -- 172.21.15.103:0/4017701047 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a74c9a210 con 0x556a75021d50 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.467+0000 7f9fdf400640 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a7502c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.467+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.467+0000 7f9fdcc00640 1 -- 172.21.15.103:0/3599010484 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9fc80eec10 con 0x7f9fe010d1a0 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.467+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2da0 0x7f9fc40a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.471 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.467+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75021d50 0x556a75022130 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009c20 tx=0x7f9fd8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4493e48b80e7449 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:10.472 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.467+0000 7f9fd4c00640 1 -- 172.21.15.103:0/4017701047 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd801b850 con 0x556a75021d50 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fd4c00640 1 -- 172.21.15.103:0/4017701047 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8019db0 con 0x556a75021d50 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fd4c00640 1 -- 172.21.15.103:0/4017701047 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd800eda0 con 0x556a75021d50 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fe9674740 1 -- 172.21.15.103:0/4017701047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75021d50 msgr2=0x556a75022130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fe9674740 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75021d50 0x556a75022130 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009c20 tx=0x7f9fd8011810 comp rx=0 tx=0).stop 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2da0 0x7f9fc40a5260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009fd0 tx=0x7f9fd8021000 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fe9674740 1 -- 172.21.15.103:0/4017701047 shutdown_connections 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fe9674740 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fe9674740 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a7502c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fe9674740 1 --2- 172.21.15.103:0/4017701047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75021d50 0x556a75022130 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.468+0000 7f9fe9674740 1 -- 172.21.15.103:0/4017701047 >> 172.21.15.103:0/4017701047 conn(0x556a7501e070 msgr2=0x556a7501e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.469+0000 7f9fe9674740 1 -- 172.21.15.103:0/4017701047 shutdown_connections 2024-05-23T00:03:10.473 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.469+0000 7f9fe9674740 1 -- 172.21.15.103:0/4017701047 wait complete. 2024-05-23T00:03:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.469+0000 7f9fe9674740 1 -- start start 2024-05-23T00:03:10.474 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.470+0000 7f9fe9674740 1 Processor -- start 2024-05-23T00:03:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.470+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75029cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.476 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.470+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a7502a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.470+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.470+0000 7f9fe9674740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556a7502fbd0 con 0x556a7502cd40 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.470+0000 7f9fe9674740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x556a74f41210 con 0x556a75022670 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.470+0000 7f9fe9674740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x556a7502fd50 con 0x556a75021d50 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.470+0000 7f9fe9674740 10 client.0 _ll_register_callbacks cb 0x556a74f24140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75029cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75029cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:47296/0 (socket says 172.21.15.103:47296) 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 -- 172.21.15.103:0/3892357242 learned_addr learned my addr 172.21.15.103:0/3892357242 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fd6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a7502a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.477 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 -- 172.21.15.103:0/3892357242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 msgr2=0x556a7502a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a7502a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 -- 172.21.15.103:0/3892357242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 msgr2=0x556a7502a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 -- 172.21.15.103:0/3892357242 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a74d66620 con 0x556a75021d50 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fdf400640 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75029cf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000b5f0 tx=0x7f9fd000bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d34dcd176473b887 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.471+0000 7f9fcd600640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.472+0000 7f9fcd600640 1 -- 172.21.15.103:0/3892357242 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd000c840 con 0x556a75021d50 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.472+0000 7f9fe9674740 1 -- 172.21.15.103:0/3892357242 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fd8019db0 con 0x556a75021d50 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.472+0000 7f9fcd600640 1 -- 172.21.15.103:0/3892357242 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9fd000c9e0 con 0x556a75021d50 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.472+0000 7f9fcd600640 1 -- 172.21.15.103:0/3892357242 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9fd0029050 con 0x556a75021d50 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.472+0000 7f9fe9674740 10 client.26128 fetch_fsmap learned FSMap version 31 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.472+0000 7f9fe9674740 1 -- 172.21.15.103:0/3892357242 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556a7503bf40 con 0x556a75021d50 2024-05-23T00:03:10.478 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.472+0000 7f9fcd600640 1 -- 172.21.15.103:0/3892357242 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd8019db0 con 0x556a75021d50 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.472+0000 7f9fcd600640 1 -- 172.21.15.103:0/3892357242 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 31) ==== 8358+0+0 (secure 0 0 0) 0x7f9fd000b040 con 0x556a75021d50 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.473+0000 7f9fe9674740 10 client.26128 fetch_fsmap finished waiting for FSMap version 31 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.473+0000 7f9fe9674740 10 client.26128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.473+0000 7f9fe9674740 20 client.26128 populate_metadata read hostname 'smithi103' 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.473+0000 7f9fe9674740 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556a7503d450 0x556a7503f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.473+0000 7f9fe9674740 4 client.26128 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.473+0000 7f9fe9674740 1 -- 172.21.15.103:0/3892357242 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a74f99ec0 con 0x556a7503d450 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.473+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556a7503d450 0x556a7503f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.474+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556a7503d450 0x556a7503f880 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dddfcb6827cab9c9 server_cookie=a5dc67eebae7cdc3 in_seq=0 out_seq=0 2024-05-23T00:03:10.479 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.474+0000 7f9fcd600640 10 client.26128 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:03:10.480 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.474+0000 7f9fcd600640 1 -- 172.21.15.103:0/3892357242 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a74f99ec0 con 0x556a7503d450 2024-05-23T00:03:10.480 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.474+0000 7f9fcd600640 10 client.26128 handle_command_reply: tid=0 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.474+0000 7f9fe9674740 1 client.26128 shutdown 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 -- 172.21.15.103:0/3892357242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 msgr2=0x556a75029cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75029cf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000b5f0 tx=0x7f9fd000bab0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 -- 172.21.15.103:0/3892357242 shutdown_connections 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556a7503d450 0x556a7503f880 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a7502a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a7502a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fcd600640 0 client.26128 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 --2- 172.21.15.103:0/3892357242 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75029cf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 -- 172.21.15.103:0/3892357242 >> 172.21.15.103:0/3892357242 conn(0x556a7501e070 msgr2=0x556a75023e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 -- 172.21.15.103:0/3892357242 shutdown_connections 2024-05-23T00:03:10.481 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 1 -- 172.21.15.103:0/3892357242 wait complete. 2024-05-23T00:03:10.482 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.475+0000 7f9fe9674740 20 client.26128 trim_cache size 0 max 16384 2024-05-23T00:03:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.491+0000 7f9fe9674740 1 Processor -- start 2024-05-23T00:03:10.494 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.491+0000 7f9fe9674740 1 -- start start 2024-05-23T00:03:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a75021d50 0x556a75038a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a75045190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.495 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a7502cd40 0x556a750456d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fe9674740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b7ca0 con 0x556a75021d50 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fe9674740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b5510 con 0x556a7502cd40 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fe9674740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b3360 con 0x556a75022670 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a75021d50 0x556a75038a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a75021d50 0x556a75038a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:34154/0 (socket says 172.21.15.103:34154) 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fdf400640 1 -- 172.21.15.103:0/1348453033 learned_addr learned my addr 172.21.15.103:0/1348453033 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a7502cd40 0x556a750456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.492+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a75045190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fdf400640 1 -- 172.21.15.103:0/1348453033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 msgr2=0x556a75045190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fdf400640 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a75045190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fdf400640 1 -- 172.21.15.103:0/1348453033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a7502cd40 msgr2=0x556a750456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fdf400640 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a7502cd40 0x556a750456d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fdf400640 1 -- 172.21.15.103:0/1348453033 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a750199d0 con 0x556a75021d50 2024-05-23T00:03:10.496 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a75045190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:03:10.497 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fdfe00640 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a7502cd40 0x556a750456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:03:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fdf400640 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a75021d50 0x556a75038a90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9fd001bf70 tx=0x7f9fd000fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f061d8eb8e4cff48 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fcfe00640 1 -- 172.21.15.103:0/1348453033 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd000c640 con 0x556a75021d50 2024-05-23T00:03:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fcfe00640 1 -- 172.21.15.103:0/1348453033 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd0020070 con 0x556a75021d50 2024-05-23T00:03:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.493+0000 7f9fcfe00640 1 -- 172.21.15.103:0/1348453033 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd001bb90 con 0x556a75021d50 2024-05-23T00:03:10.498 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 -- 172.21.15.103:0/1348453033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a75021d50 msgr2=0x556a75038a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a75021d50 0x556a75038a90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9fd001bf70 tx=0x7f9fd000fb90 comp rx=0 tx=0).stop 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 -- 172.21.15.103:0/1348453033 shutdown_connections 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a7502cd40 0x556a750456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75022670 0x556a75045190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 --2- 172.21.15.103:0/1348453033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a75021d50 0x556a75038a90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 -- 172.21.15.103:0/1348453033 >> 172.21.15.103:0/1348453033 conn(0x556a7501e070 msgr2=0x556a75024fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 -- 172.21.15.103:0/1348453033 shutdown_connections 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.494+0000 7f9fe9674740 1 -- 172.21.15.103:0/1348453033 wait complete. 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 1 -- start start 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 1 Processor -- start 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75026850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a75026d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a750272d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.499 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b39d0 con 0x556a7502cd40 2024-05-23T00:03:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b4e40 con 0x556a75022670 2024-05-23T00:03:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fe01b40a0 con 0x556a75021d50 2024-05-23T00:03:10.500 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.495+0000 7f9fe9674740 10 client.0 _ll_register_callbacks cb 0x7f9fd000fea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:03:10.501 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a75026d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fdfe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a750272d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a75026d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:47160/0 (socket says 172.21.15.103:47160) 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fdf400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75026850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 -- 172.21.15.103:0/2572725806 learned_addr learned my addr 172.21.15.103:0/2572725806 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 -- 172.21.15.103:0/2572725806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 msgr2=0x556a75026850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75026850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 -- 172.21.15.103:0/2572725806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 msgr2=0x556a750272d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a750272d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 -- 172.21.15.103:0/2572725806 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a750a2a00 con 0x556a75022670 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fdf400640 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75026850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fd6a00640 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a75026d90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8005e00 tx=0x7f9fc8005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a093bd9a84bd9010 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:10.502 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.496+0000 7f9fcd600640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fcd600640 1 -- 172.21.15.103:0/2572725806 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fc80bae50 con 0x556a75022670 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fcd600640 1 -- 172.21.15.103:0/2572725806 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f9fc80b74c0 con 0x556a75022670 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 1 -- 172.21.15.103:0/2572725806 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9fd0020070 con 0x556a75022670 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fcd600640 1 -- 172.21.15.103:0/2572725806 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f9fc80bf200 con 0x556a75022670 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 10 client.26132 fetch_fsmap learned FSMap version 31 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 1 -- 172.21.15.103:0/2572725806 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x556a750a3c10 con 0x556a75022670 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fcd600640 1 -- 172.21.15.103:0/2572725806 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9fd0020070 con 0x556a75022670 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fcd600640 1 -- 172.21.15.103:0/2572725806 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 31) ==== 8358+0+0 (secure 0 0 0) 0x7f9fc803a070 con 0x556a75022670 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 10 client.26132 fetch_fsmap finished waiting for FSMap version 31 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 10 client.26132 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 20 client.26132 populate_metadata read hostname 'smithi103' 2024-05-23T00:03:10.503 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556a750a4370 0x556a750a6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 4 client.26132 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:03:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.497+0000 7f9fe9674740 1 -- 172.21.15.103:0/2572725806 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556a74f93a10 con 0x556a750a4370 2024-05-23T00:03:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.498+0000 7f9fdf400640 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556a750a4370 0x556a750a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.498+0000 7f9fdf400640 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556a750a4370 0x556a750a6790 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4576fd2ef7f58a70 server_cookie=300326067d5f05ed in_seq=0 out_seq=0 2024-05-23T00:03:10.504 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.498+0000 7f9fcd600640 10 client.26132 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:03:10.877 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fcd600640 1 -- 172.21.15.103:0/2572725806 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556a74f93a10 con 0x556a750a4370 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fcd600640 10 client.26132 handle_command_reply: tid=0 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 client.26132 shutdown 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 -- 172.21.15.103:0/2572725806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 msgr2=0x556a75026d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a75026d90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8005e00 tx=0x7f9fc8005740 comp rx=0 tx=0).stop 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 -- 172.21.15.103:0/2572725806 shutdown_connections 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x556a750a4370 0x556a750a6790 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556a7502cd40 0x556a750272d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556a75022670 0x556a75026d90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 --2- 172.21.15.103:0/2572725806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x556a75021d50 0x556a75026850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.878 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fcd600640 0 client.26132 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:03:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 -- 172.21.15.103:0/2572725806 >> 172.21.15.103:0/2572725806 conn(0x556a7501e070 msgr2=0x556a7501c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 -- 172.21.15.103:0/2572725806 shutdown_connections 2024-05-23T00:03:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.874+0000 7f9fe9674740 1 -- 172.21.15.103:0/2572725806 wait complete. 2024-05-23T00:03:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe9674740 20 client.26132 trim_cache size 0 max 16384 2024-05-23T00:03:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:03:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:03:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:03:10.879 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:03:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 -- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2da0 msgr2=0x7f9fc40a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2da0 0x7f9fc40a5260 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009fd0 tx=0x7f9fd8021000 comp rx=0 tx=0).stop 2024-05-23T00:03:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 -- 172.21.15.103:0/3599010484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe010d1a0 msgr2=0x7f9fe019ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe010d1a0 0x7f9fe019ecb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9fc802a860 tx=0x7f9fc8033b60 comp rx=0 tx=0).stop 2024-05-23T00:03:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 -- 172.21.15.103:0/3599010484 shutdown_connections 2024-05-23T00:03:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f9fc40a2da0 0x7f9fc40a5260 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.880 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9fe01a2fe0 0x7f9fe01a3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.881 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9fe010d1a0 0x7f9fe019ecb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.881 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 --2- 172.21.15.103:0/3599010484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fe01080d0 0x7f9fe019e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:10.881 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.875+0000 7f9fe6800640 1 -- 172.21.15.103:0/3599010484 >> 172.21.15.103:0/3599010484 conn(0x7f9fe0075e00 msgr2=0x7f9fe00780f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:10.881 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.876+0000 7f9fe6800640 1 -- 172.21.15.103:0/3599010484 shutdown_connections 2024-05-23T00:03:10.881 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:10.876+0000 7f9fe6800640 1 -- 172.21.15.103:0/3599010484 wait complete. 2024-05-23T00:03:11.458 INFO:tasks.workunit.client.0.smithi103.stdout:config.status: creating include/builddefs 2024-05-23T00:03:12.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:11 smithi105 ceph-mon[36426]: pgmap v84: 129 pgs: 129 active+clean; 79 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-05-23T00:03:12.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:11 smithi103 ceph-mon[31370]: pgmap v84: 129 pgs: 129 active+clean; 79 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-05-23T00:03:12.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:11 smithi174 ceph-mon[36179]: pgmap v84: 129 pgs: 129 active+clean; 79 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-05-23T00:03:14.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:13 smithi105 ceph-mon[36426]: pgmap v85: 129 pgs: 129 active+clean; 79 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-05-23T00:03:14.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:13 smithi103 ceph-mon[31370]: pgmap v85: 129 pgs: 129 active+clean; 79 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-05-23T00:03:14.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:13 smithi174 ceph-mon[36179]: pgmap v85: 129 pgs: 129 active+clean; 79 MiB data, 521 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 1.8 MiB/s wr, 207 op/s 2024-05-23T00:03:15.991 INFO:tasks.workunit.client.0.smithi103.stdout:config.status: creating include/config.h 2024-05-23T00:03:16.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:15 smithi105 ceph-mon[36426]: pgmap v86: 129 pgs: 129 active+clean; 77 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 433 op/s 2024-05-23T00:03:16.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:15 smithi103 ceph-mon[31370]: pgmap v86: 129 pgs: 129 active+clean; 77 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 433 op/s 2024-05-23T00:03:16.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:15 smithi174 ceph-mon[36179]: pgmap v86: 129 pgs: 129 active+clean; 77 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 433 op/s 2024-05-23T00:03:18.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:17 smithi105 ceph-mon[36426]: pgmap v87: 129 pgs: 129 active+clean; 77 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 314 op/s 2024-05-23T00:03:18.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:17 smithi103 ceph-mon[31370]: pgmap v87: 129 pgs: 129 active+clean; 77 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 314 op/s 2024-05-23T00:03:18.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:17 smithi174 ceph-mon[36179]: pgmap v87: 129 pgs: 129 active+clean; 77 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 314 op/s 2024-05-23T00:03:20.168 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:19 smithi174 ceph-mon[36179]: pgmap v88: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 319 op/s 2024-05-23T00:03:20.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:19 smithi105 ceph-mon[36426]: pgmap v88: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 319 op/s 2024-05-23T00:03:20.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:19 smithi103 ceph-mon[31370]: pgmap v88: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 319 op/s 2024-05-23T00:03:20.992 INFO:tasks.workunit.client.0.smithi103.stdout:config.status: executing libtool commands 2024-05-23T00:03:21.180 INFO:tasks.workunit.client.0.smithi103.stdout:Building include 2024-05-23T00:03:21.181 INFO:tasks.workunit.client.0.smithi103.stdout:Building lib 2024-05-23T00:03:21.182 INFO:tasks.workunit.client.0.smithi103.stdout:Building tests 2024-05-23T00:03:21.196 INFO:tasks.workunit.client.0.smithi103.stdout: [LTDEP] 2024-05-23T00:03:21.688 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] dataascii.lo 2024-05-23T00:03:21.690 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] databin.lo 2024-05-23T00:03:21.690 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] datapid.lo 2024-05-23T00:03:21.691 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] file_lock.lo 2024-05-23T00:03:21.963 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] forker.lo 2024-05-23T00:03:22.092 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] pattern.lo 2024-05-23T00:03:22.272 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] open_flags.lo 2024-05-23T00:03:22.272 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] random_range.lo 2024-05-23T00:03:22.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:21 smithi105 ceph-mon[36426]: pgmap v89: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 891 KiB/s wr, 264 op/s 2024-05-23T00:03:22.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:21 smithi103 ceph-mon[31370]: pgmap v89: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 891 KiB/s wr, 264 op/s 2024-05-23T00:03:22.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:21 smithi174 ceph-mon[36179]: pgmap v89: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 891 KiB/s wr, 264 op/s 2024-05-23T00:03:22.421 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] string_to_tokens.lo 2024-05-23T00:03:22.508 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] str_to_bytes.lo 2024-05-23T00:03:22.602 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] tlibio.lo 2024-05-23T00:03:22.670 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] write_log.lo 2024-05-23T00:03:22.684 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] random.lo 2024-05-23T00:03:23.103 INFO:tasks.workunit.client.0.smithi103.stdout: [LD] 2024-05-23T00:03:23.336 INFO:tasks.workunit.client.0.smithi103.stdout:Building ltp 2024-05-23T00:03:23.337 INFO:tasks.workunit.client.0.smithi103.stdout:Building src 2024-05-23T00:03:23.337 INFO:tasks.workunit.client.0.smithi103.stdout:Building m4 2024-05-23T00:03:23.337 INFO:tasks.workunit.client.0.smithi103.stdout:Building common 2024-05-23T00:03:23.355 INFO:tasks.workunit.client.0.smithi103.stdout: [DEP] 2024-05-23T00:03:23.406 INFO:tasks.workunit.client.0.smithi103.stdout: [DEP] 2024-05-23T00:03:23.890 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] fsstress 2024-05-23T00:03:23.891 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] doio 2024-05-23T00:03:23.892 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] fsx 2024-05-23T00:03:24.036 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.034+0000 7f1a67a00640 1 -- 172.21.15.103:0/1437178735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 msgr2=0x7f1a60076180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.034+0000 7f1a67a00640 1 --2- 172.21.15.103:0/1437178735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 0x7f1a60076180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1a54009930 tx=0x7f1a5402a420 comp rx=0 tx=0).stop 2024-05-23T00:03:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.034+0000 7f1a67a00640 1 -- 172.21.15.103:0/1437178735 shutdown_connections 2024-05-23T00:03:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.034+0000 7f1a67a00640 1 --2- 172.21.15.103:0/1437178735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a60112700 0x7f1a60114af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.034+0000 7f1a67a00640 1 --2- 172.21.15.103:0/1437178735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a600766c0 0x7f1a601121c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.037 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.034+0000 7f1a67a00640 1 --2- 172.21.15.103:0/1437178735 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 0x7f1a60076180 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.034+0000 7f1a67a00640 1 -- 172.21.15.103:0/1437178735 >> 172.21.15.103:0/1437178735 conn(0x7f1a600ff570 msgr2=0x7f1a60101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.034+0000 7f1a67a00640 1 -- 172.21.15.103:0/1437178735 shutdown_connections 2024-05-23T00:03:24.038 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.035+0000 7f1a67a00640 1 -- 172.21.15.103:0/1437178735 wait complete. 2024-05-23T00:03:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.035+0000 7f1a67a00640 1 Processor -- start 2024-05-23T00:03:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.036+0000 7f1a67a00640 1 -- start start 2024-05-23T00:03:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.036+0000 7f1a67a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 0x7f1a6019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.036+0000 7f1a67a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a600766c0 0x7f1a6019ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.036+0000 7f1a67a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a60112700 0x7f1a601a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.036+0000 7f1a67a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a601182c0 con 0x7f1a600766c0 2024-05-23T00:03:24.039 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.036+0000 7f1a67a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1a60118140 con 0x7f1a60112700 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.036+0000 7f1a67a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a60118440 con 0x7f1a60075da0 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.036+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a600766c0 0x7f1a6019ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a65000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 0x7f1a6019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a600766c0 0x7f1a6019ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44932/0 (socket says 172.21.15.103:44932) 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a5fe00640 1 -- 172.21.15.103:0/3817035032 learned_addr learned my addr 172.21.15.103:0/3817035032 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a65a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a60112700 0x7f1a601a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a65000640 1 -- 172.21.15.103:0/3817035032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a60112700 msgr2=0x7f1a601a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a65000640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a60112700 0x7f1a601a2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a65000640 1 -- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a600766c0 msgr2=0x7f1a6019ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a65000640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a600766c0 0x7f1a6019ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.040 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.037+0000 7f1a65000640 1 -- 172.21.15.103:0/3817035032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a601a3710 con 0x7f1a60075da0 2024-05-23T00:03:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.038+0000 7f1a65000640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 0x7f1a6019e720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402a940 tx=0x7f1a54003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.038+0000 7f1a5d600640 1 -- 172.21.15.103:0/3817035032 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5403f070 con 0x7f1a60075da0 2024-05-23T00:03:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.038+0000 7f1a5d600640 1 -- 172.21.15.103:0/3817035032 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a54031c90 con 0x7f1a60075da0 2024-05-23T00:03:24.041 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.038+0000 7f1a5d600640 1 -- 172.21.15.103:0/3817035032 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a54035030 con 0x7f1a60075da0 2024-05-23T00:03:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.039+0000 7f1a67a00640 1 -- 172.21.15.103:0/3817035032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a601a39a0 con 0x7f1a60075da0 2024-05-23T00:03:24.042 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.039+0000 7f1a67a00640 1 -- 172.21.15.103:0/3817035032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a601ab290 con 0x7f1a60075da0 2024-05-23T00:03:24.045 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.043+0000 7f1a5d600640 1 -- 172.21.15.103:0/3817035032 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f1a54042070 con 0x7f1a60075da0 2024-05-23T00:03:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.043+0000 7f1a6a84a740 1 Processor -- start 2024-05-23T00:03:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.044+0000 7f1a6a84a740 1 -- start start 2024-05-23T00:03:24.046 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.044+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05a5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.044+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05af800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.044+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05b2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.044+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5653e05092e0 con 0x5653e05a5670 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.044+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5653e04b0e50 con 0x5653e05a4d50 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.044+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5653e05052a0 con 0x5653e05afd40 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05b2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:37752/0 (socket says 172.21.15.103:37752) 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a5fe00640 1 -- 172.21.15.103:0/1796846393 learned_addr learned my addr 172.21.15.103:0/1796846393 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a65a00640 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.047 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a65000640 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a5fe00640 1 -- 172.21.15.103:0/1796846393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 msgr2=0x5653e05a5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:24.048 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05a5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a5fe00640 1 -- 172.21.15.103:0/1796846393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 msgr2=0x5653e05af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a5fe00640 1 -- 172.21.15.103:0/1796846393 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653e021d210 con 0x5653e05afd40 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.045+0000 7f1a65000640 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.046+0000 7f1a5d600640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a480a2e20 0x7f1a480a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.046+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05b2130 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c004630 tx=0x7f1a4c00bc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6dccd3738da35d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.046+0000 7f1a5d600640 1 -- 172.21.15.103:0/3817035032 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f1a540efec0 con 0x7f1a60075da0 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.046+0000 7f1a65a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a480a2e20 0x7f1a480a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.046+0000 7f1a52000640 1 -- 172.21.15.103:0/1796846393 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c003dd0 con 0x5653e05afd40 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.046+0000 7f1a52000640 1 -- 172.21.15.103:0/1796846393 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a4c015070 con 0x5653e05afd40 2024-05-23T00:03:24.049 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.046+0000 7f1a52000640 1 -- 172.21.15.103:0/1796846393 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c011750 con 0x5653e05afd40 2024-05-23T00:03:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1796846393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 msgr2=0x5653e05b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a65a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a480a2e20 0x7f1a480a52e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a6006ad70 tx=0x7f1a58009600 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05b2130 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c004630 tx=0x7f1a4c00bc70 comp rx=0 tx=0).stop 2024-05-23T00:03:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1796846393 shutdown_connections 2024-05-23T00:03:24.050 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05b2130 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/1796846393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1796846393 >> 172.21.15.103:0/1796846393 conn(0x5653e05a1070 msgr2=0x5653e05a1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1796846393 shutdown_connections 2024-05-23T00:03:24.051 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.048+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1796846393 wait complete. 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.049+0000 7f1a6a84a740 1 -- start start 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.049+0000 7f1a6a84a740 1 Processor -- start 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a4d50 0x5653e05acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05ad240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05afd40 0x5653e05ad780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5653e05b2bd0 con 0x5653e05a4d50 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x5653e04c4210 con 0x5653e05afd40 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x5653e05b2d50 con 0x5653e05a5670 2024-05-23T00:03:24.052 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a6a84a740 10 client.0 _ll_register_callbacks cb 0x5653e04a7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:03:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a65a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05afd40 0x5653e05ad780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a65000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a4d50 0x5653e05acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a65a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05afd40 0x5653e05ad780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:56472/0 (socket says 172.21.15.103:56472) 2024-05-23T00:03:24.053 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a65a00640 1 -- 172.21.15.103:0/872071108 learned_addr learned my addr 172.21.15.103:0/872071108 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.050+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05ad240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.051+0000 7f1a65a00640 1 -- 172.21.15.103:0/872071108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 msgr2=0x5653e05ad240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.051+0000 7f1a65a00640 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05ad240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.051+0000 7f1a65a00640 1 -- 172.21.15.103:0/872071108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a4d50 msgr2=0x5653e05acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.051+0000 7f1a65a00640 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a4d50 0x5653e05acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.051+0000 7f1a65a00640 1 -- 172.21.15.103:0/872071108 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653e02e9620 con 0x5653e05afd40 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.051+0000 7f1a65a00640 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05afd40 0x5653e05ad780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1a58013f20 tx=0x7f1a58004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7bc0c8c76cae37b server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.051+0000 7f1a46a00640 10 client.0 ms_handle_connect on v2:172.21.15.105:3300/0 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.051+0000 7f1a46a00640 1 -- 172.21.15.103:0/872071108 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a58004d20 con 0x5653e05afd40 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.052+0000 7f1a46a00640 1 -- 172.21.15.103:0/872071108 <== mon.2 v2:172.21.15.105:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f1a58005020 con 0x5653e05afd40 2024-05-23T00:03:24.054 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.052+0000 7f1a6a84a740 1 -- 172.21.15.103:0/872071108 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a4c015070 con 0x5653e05afd40 2024-05-23T00:03:24.055 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.052+0000 7f1a46a00640 1 -- 172.21.15.103:0/872071108 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1a5801fe80 con 0x5653e05afd40 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.052+0000 7f1a6a84a740 10 client.26144 fetch_fsmap learned FSMap version 32 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.052+0000 7f1a6a84a740 1 -- 172.21.15.103:0/872071108 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5653e05bef30 con 0x5653e05afd40 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.052+0000 7f1a46a00640 1 -- 172.21.15.103:0/872071108 <== mon.2 v2:172.21.15.105:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a4c015070 con 0x5653e05afd40 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.052+0000 7f1a46a00640 1 -- 172.21.15.103:0/872071108 <== mon.2 v2:172.21.15.105:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1a5800f040 con 0x5653e05afd40 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.052+0000 7f1a6a84a740 10 client.26144 fetch_fsmap finished waiting for FSMap version 32 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.053+0000 7f1a6a84a740 10 client.26144 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.053+0000 7f1a6a84a740 20 client.26144 populate_metadata read hostname 'smithi103' 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.053+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5653e05c0440 0x5653e05c2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.053+0000 7f1a6a84a740 4 client.26144 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.053+0000 7f1a6a84a740 1 -- 172.21.15.103:0/872071108 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653e051cec0 con 0x5653e05c0440 2024-05-23T00:03:24.056 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.053+0000 7f1a65000640 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5653e05c0440 0x5653e05c2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.054+0000 7f1a65000640 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5653e05c0440 0x5653e05c2830 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba0868a8140111d server_cookie=f3dc083015ceba26 in_seq=0 out_seq=0 2024-05-23T00:03:24.057 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.054+0000 7f1a46a00640 10 client.26144 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:03:24.058 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.054+0000 7f1a46a00640 1 -- 172.21.15.103:0/872071108 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653e051cec0 con 0x5653e05c0440 2024-05-23T00:03:24.058 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.054+0000 7f1a46a00640 10 client.26144 handle_command_reply: tid=0 2024-05-23T00:03:24.058 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.054+0000 7f1a6a84a740 1 client.26144 shutdown 2024-05-23T00:03:24.058 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 -- 172.21.15.103:0/872071108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05afd40 msgr2=0x5653e05ad780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.058 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05afd40 0x5653e05ad780 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1a58013f20 tx=0x7f1a58004a40 comp rx=0 tx=0).stop 2024-05-23T00:03:24.058 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 -- 172.21.15.103:0/872071108 shutdown_connections 2024-05-23T00:03:24.058 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5653e05c0440 0x5653e05c2830 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.058 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05afd40 0x5653e05ad780 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05ad240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/872071108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a4d50 0x5653e05acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 -- 172.21.15.103:0/872071108 >> 172.21.15.103:0/872071108 conn(0x5653e05a1070 msgr2=0x5653e05a01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a46a00640 0 client.26144 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:03:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 -- 172.21.15.103:0/872071108 shutdown_connections 2024-05-23T00:03:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.055+0000 7f1a6a84a740 1 -- 172.21.15.103:0/872071108 wait complete. 2024-05-23T00:03:24.059 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.056+0000 7f1a6a84a740 20 client.26144 trim_cache size 0 max 16384 2024-05-23T00:03:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.078+0000 7f1a6a84a740 1 Processor -- start 2024-05-23T00:03:24.080 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.079+0000 7f1a6a84a740 1 -- start start 2024-05-23T00:03:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.079+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05bbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.079+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05c8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.079+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05c8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.079+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a601af640 con 0x5653e05a5670 2024-05-23T00:03:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.079+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1a601aceb0 con 0x5653e05a4d50 2024-05-23T00:03:24.081 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.079+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a60073340 con 0x5653e05afd40 2024-05-23T00:03:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a65000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05bbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a65a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05c8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a65a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05c8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:37776/0 (socket says 172.21.15.103:37776) 2024-05-23T00:03:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a65a00640 1 -- 172.21.15.103:0/1630569331 learned_addr learned my addr 172.21.15.103:0/1630569331 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:24.082 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/1630569331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a5fe00640 1 -- 172.21.15.103:0/1630569331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 msgr2=0x5653e05c8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/1630569331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05c8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a5fe00640 1 -- 172.21.15.103:0/1630569331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 msgr2=0x5653e05bbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.080+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/1630569331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05bbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.081+0000 7f1a5fe00640 1 -- 172.21.15.103:0/1630569331 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653e05a1c40 con 0x5653e05a5670 2024-05-23T00:03:24.083 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.081+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/1630569331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05c8140 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c006010 tx=0x7f1a4c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b848a354726ac2 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.082+0000 7f1a51600640 1 -- 172.21.15.103:0/1630569331 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c027070 con 0x5653e05a5670 2024-05-23T00:03:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.082+0000 7f1a51600640 1 -- 172.21.15.103:0/1630569331 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a4c015040 con 0x5653e05a5670 2024-05-23T00:03:24.084 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.082+0000 7f1a51600640 1 -- 172.21.15.103:0/1630569331 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c011730 con 0x5653e05a5670 2024-05-23T00:03:24.085 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1630569331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 msgr2=0x5653e05c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.085 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/1630569331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05c8140 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c006010 tx=0x7f1a4c005740 comp rx=0 tx=0).stop 2024-05-23T00:03:24.085 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1630569331 shutdown_connections 2024-05-23T00:03:24.085 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/1630569331 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05afd40 0x5653e05c8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.085 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/1630569331 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05a5670 0x5653e05c8140 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.085 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/1630569331 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05bbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.085 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1630569331 >> 172.21.15.103:0/1630569331 conn(0x5653e05a1070 msgr2=0x5653e05a7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:24.085 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1630569331 shutdown_connections 2024-05-23T00:03:24.086 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.083+0000 7f1a6a84a740 1 -- 172.21.15.103:0/1630569331 wait complete. 2024-05-23T00:03:24.086 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.084+0000 7f1a6a84a740 1 -- start start 2024-05-23T00:03:24.086 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.084+0000 7f1a6a84a740 1 Processor -- start 2024-05-23T00:03:24.087 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05b8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.087 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.087 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a6a84a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05afd40 0x5653e05b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.087 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1a601ab460 con 0x5653e05afd40 2024-05-23T00:03:24.087 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f1a601ac7e0 con 0x5653e05a4d50 2024-05-23T00:03:24.088 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a6a84a740 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a601aba40 con 0x5653e05a5670 2024-05-23T00:03:24.088 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a6a84a740 10 client.0 _ll_register_callbacks cb 0x7f1a580119c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-23T00:03:24.088 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a65a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05afd40 0x5653e05b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.088 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a65a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05afd40 0x5653e05b3e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:44966/0 (socket says 172.21.15.103:44966) 2024-05-23T00:03:24.088 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a65a00640 1 -- 172.21.15.103:0/2433357261 learned_addr learned my addr 172.21.15.103:0/2433357261 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:03:24.088 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a65000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05b8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.088 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.085+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.086+0000 7f1a5fe00640 1 -- 172.21.15.103:0/2433357261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 msgr2=0x5653e05b8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.086+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05b8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.086+0000 7f1a5fe00640 1 -- 172.21.15.103:0/2433357261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05afd40 msgr2=0x5653e05b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-23T00:03:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.086+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05afd40 0x5653e05b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.086+0000 7f1a5fe00640 1 -- 172.21.15.103:0/2433357261 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653e05b4630 con 0x5653e05a5670 2024-05-23T00:03:24.090 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.086+0000 7f1a5fe00640 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05b37f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5653e05adcf0 tx=0x7f1a4c003f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97ce6061fcfc6996 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.086+0000 7f1a46a00640 10 client.0 ms_handle_connect on v2:172.21.15.174:3300/0 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a46a00640 1 -- 172.21.15.103:0/2433357261 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4c027070 con 0x5653e05a5670 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a6a84a740 1 -- 172.21.15.103:0/2433357261 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a4c015040 con 0x5653e05a5670 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a46a00640 1 -- 172.21.15.103:0/2433357261 <== mon.1 v2:172.21.15.174:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9524+0+0 (secure 0 0 0) 0x7f1a4c01a880 con 0x5653e05a5670 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a46a00640 1 -- 172.21.15.103:0/2433357261 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f1a4c01aaa0 con 0x5653e05a5670 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a6a84a740 10 client.26152 fetch_fsmap learned FSMap version 32 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a6a84a740 1 -- 172.21.15.103:0/2433357261 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5653e0626ac0 con 0x5653e05a5670 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a46a00640 1 -- 172.21.15.103:0/2433357261 <== mon.1 v2:172.21.15.174:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a4c015040 con 0x5653e05a5670 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a46a00640 1 -- 172.21.15.103:0/2433357261 <== mon.1 v2:172.21.15.174:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f1a4c022070 con 0x5653e05a5670 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a6a84a740 10 client.26152 fetch_fsmap finished waiting for FSMap version 32 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a6a84a740 10 client.26152 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.087+0000 7f1a6a84a740 20 client.26152 populate_metadata read hostname 'smithi103' 2024-05-23T00:03:24.091 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.088+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5653e0626f30 0x5653e0629320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:03:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.088+0000 7f1a6a84a740 4 client.26152 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-23T00:03:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.088+0000 7f1a6a84a740 1 -- 172.21.15.103:0/2433357261 --> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5653e0516a10 con 0x5653e0626f30 2024-05-23T00:03:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.088+0000 7f1a65000640 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5653e0626f30 0x5653e0629320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:03:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.088+0000 7f1a65000640 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5653e0626f30 0x5653e0629320 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12055072edb46e61 server_cookie=31885b62982e8814 in_seq=0 out_seq=0 2024-05-23T00:03:24.092 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.088+0000 7f1a46a00640 10 client.26152 ms_handle_connect on v2:172.21.15.105:6838/160741513 2024-05-23T00:03:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.151+0000 7f1a46a00640 1 -- 172.21.15.103:0/2433357261 <== mds.0 v2:172.21.15.105:6838/160741513 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5653e0516a10 con 0x5653e0626f30 2024-05-23T00:03:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.151+0000 7f1a46a00640 10 client.26152 handle_command_reply: tid=0 2024-05-23T00:03:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.151+0000 7f1a6a84a740 1 client.26152 shutdown 2024-05-23T00:03:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.151+0000 7f1a6a84a740 1 -- 172.21.15.103:0/2433357261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 msgr2=0x5653e05b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.151+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05b37f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x5653e05adcf0 tx=0x7f1a4c003f40 comp rx=0 tx=0).stop 2024-05-23T00:03:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 1 -- 172.21.15.103:0/2433357261 shutdown_connections 2024-05-23T00:03:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.105:6838/160741513,v1:172.21.15.105:6839/160741513] conn(0x5653e0626f30 0x5653e0629320 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.154 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653e05afd40 0x5653e05b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x5653e05a5670 0x5653e05b37f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 1 --2- 172.21.15.103:0/2433357261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x5653e05a4d50 0x5653e05b8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 1 -- 172.21.15.103:0/2433357261 >> 172.21.15.103:0/2433357261 conn(0x5653e05a1070 msgr2=0x5653e059f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a46a00640 0 client.26152 ms_handle_reset on v2:172.21.15.105:6838/160741513 2024-05-23T00:03:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 1 -- 172.21.15.103:0/2433357261 shutdown_connections 2024-05-23T00:03:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 1 -- 172.21.15.103:0/2433357261 wait complete. 2024-05-23T00:03:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.152+0000 7f1a6a84a740 20 client.26152 trim_cache size 0 max 16384 2024-05-23T00:03:24.155 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:{ 2024-05-23T00:03:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "message": "", 2024-05-23T00:03:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout: "return_code": 0 2024-05-23T00:03:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stdout:} 2024-05-23T00:03:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 -- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a480a2e20 msgr2=0x7f1a480a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a480a2e20 0x7f1a480a52e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f1a6006ad70 tx=0x7f1a58009600 comp rx=0 tx=0).stop 2024-05-23T00:03:24.156 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 -- 172.21.15.103:0/3817035032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 msgr2=0x7f1a6019e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 0x7f1a6019e720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1a5402a940 tx=0x7f1a54003b90 comp rx=0 tx=0).stop 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 -- 172.21.15.103:0/3817035032 shutdown_connections 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f1a480a2e20 0x7f1a480a52e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1a60112700 0x7f1a601a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1a600766c0 0x7f1a6019ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 --2- 172.21.15.103:0/3817035032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a60075da0 0x7f1a6019e720 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 -- 172.21.15.103:0/3817035032 >> 172.21.15.103:0/3817035032 conn(0x7f1a600ff570 msgr2=0x7f1a60113d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 -- 172.21.15.103:0/3817035032 shutdown_connections 2024-05-23T00:03:24.157 INFO:teuthology.task.background_exec.ubuntu@smithi103.front.sepia.ceph.com.smithi103.stderr:2024-05-23T00:03:24.154+0000 7f1a67a00640 1 -- 172.21.15.103:0/3817035032 wait complete. 2024-05-23T00:03:24.247 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c: In function 'attr_list_path': 2024-05-23T00:03:24.247 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-05-23T00:03:24.247 INFO:tasks.workunit.client.0.smithi103.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-05-23T00:03:24.247 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~ 2024-05-23T00:03:24.247 INFO:tasks.workunit.client.0.smithi103.stderr:In file included from ../src/global.h:25, 2024-05-23T00:03:24.247 INFO:tasks.workunit.client.0.smithi103.stderr: from fsstress.c:12: 2024-05-23T00:03:24.247 INFO:tasks.workunit.client.0.smithi103.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-05-23T00:03:24.247 INFO:tasks.workunit.client.0.smithi103.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~ 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c: In function 'attr_remove_path': 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~ 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr:In file included from ../src/global.h:25, 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr: from fsstress.c:12: 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~ 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c: In function 'attr_set_path': 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-05-23T00:03:24.248 INFO:tasks.workunit.client.0.smithi103.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-05-23T00:03:24.249 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~ 2024-05-23T00:03:24.249 INFO:tasks.workunit.client.0.smithi103.stderr:In file included from ../src/global.h:25, 2024-05-23T00:03:24.249 INFO:tasks.workunit.client.0.smithi103.stderr: from fsstress.c:12: 2024-05-23T00:03:24.249 INFO:tasks.workunit.client.0.smithi103.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-05-23T00:03:24.249 INFO:tasks.workunit.client.0.smithi103.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-05-23T00:03:24.249 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~ 2024-05-23T00:03:24.259 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c: In function 'allocsp_f': 2024-05-23T00:03:24.259 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-05-23T00:03:24.259 INFO:tasks.workunit.client.0.smithi103.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-05-23T00:03:24.259 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~~~~~~~ 2024-05-23T00:03:24.259 INFO:tasks.workunit.client.0.smithi103.stderr: | XFS_IOC_RESVSP64 2024-05-23T00:03:24.259 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-05-23T00:03:24.278 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c: In function 'freesp_f': 2024-05-23T00:03:24.279 INFO:tasks.workunit.client.0.smithi103.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-05-23T00:03:24.279 INFO:tasks.workunit.client.0.smithi103.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-05-23T00:03:24.279 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~~~~~~ 2024-05-23T00:03:24.279 INFO:tasks.workunit.client.0.smithi103.stderr: | XFS_IOC_RESVSP64 2024-05-23T00:03:24.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:23 smithi105 ceph-mon[36426]: pgmap v90: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 379 KiB/s wr, 230 op/s 2024-05-23T00:03:24.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:23 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:24.322 INFO:tasks.workunit.client.0.smithi103.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-05-23T00:03:24.322 INFO:tasks.workunit.client.0.smithi103.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-05-23T00:03:24.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:23 smithi103 ceph-mon[31370]: pgmap v90: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 379 KiB/s wr, 230 op/s 2024-05-23T00:03:24.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:23 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:24.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:23 smithi174 ceph-mon[36179]: pgmap v90: 129 pgs: 129 active+clean; 77 MiB data, 540 MiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 379 KiB/s wr, 230 op/s 2024-05-23T00:03:24.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:23 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:24.715 INFO:tasks.workunit.client.0.smithi103.stderr:doio.c: In function 'parse_cmdline': 2024-05-23T00:03:24.715 INFO:tasks.workunit.client.0.smithi103.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-05-23T00:03:24.715 INFO:tasks.workunit.client.0.smithi103.stderr: 3113 | fprintf(stderr, 2024-05-23T00:03:24.715 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~~~~~ 2024-05-23T00:03:24.715 INFO:tasks.workunit.client.0.smithi103.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-05-23T00:03:24.715 INFO:tasks.workunit.client.0.smithi103.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-23T00:03:24.715 INFO:tasks.workunit.client.0.smithi103.stderr: 3115 | Prog, TagName, tok); 2024-05-23T00:03:24.715 INFO:tasks.workunit.client.0.smithi103.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-05-23T00:03:24.750 INFO:tasks.workunit.client.0.smithi103.stderr:fsx.c: In function 'main': 2024-05-23T00:03:24.751 INFO:tasks.workunit.client.0.smithi103.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-05-23T00:03:24.751 INFO:tasks.workunit.client.0.smithi103.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-05-23T00:03:24.751 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-23T00:03:24.751 INFO:tasks.workunit.client.0.smithi103.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-05-23T00:03:24.751 INFO:tasks.workunit.client.0.smithi103.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-05-23T00:03:24.751 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-23T00:03:24.884 INFO:tasks.workunit.client.0.smithi103.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-05-23T00:03:24.884 INFO:tasks.workunit.client.0.smithi103.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-05-23T00:03:26.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:25 smithi105 ceph-mon[36426]: pgmap v91: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 504 KiB/s wr, 246 op/s 2024-05-23T00:03:26.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:25 smithi103 ceph-mon[31370]: pgmap v91: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 504 KiB/s wr, 246 op/s 2024-05-23T00:03:26.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:25 smithi174 ceph-mon[36179]: pgmap v91: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 504 KiB/s wr, 246 op/s 2024-05-23T00:03:26.956 INFO:tasks.workunit.client.0.smithi103.stdout: [DEP] 2024-05-23T00:03:26.959 INFO:tasks.workunit.client.0.smithi103.stdout: [DEP] 2024-05-23T00:03:26.960 INFO:tasks.workunit.client.0.smithi103.stdout: [DEP] 2024-05-23T00:03:27.499 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] dirstress 2024-05-23T00:03:27.505 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] fill 2024-05-23T00:03:27.506 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] fill2 2024-05-23T00:03:27.508 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] getpagesize 2024-05-23T00:03:27.998 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] lstat64 2024-05-23T00:03:27.998 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] permname 2024-05-23T00:03:27.998 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] nametest 2024-05-23T00:03:27.999 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] holes 2024-05-23T00:03:28.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:27 smithi105 ceph-mon[36426]: pgmap v92: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 130 KiB/s wr, 20 op/s 2024-05-23T00:03:28.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:27 smithi103 ceph-mon[31370]: pgmap v92: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 130 KiB/s wr, 20 op/s 2024-05-23T00:03:28.343 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] randholes 2024-05-23T00:03:28.354 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] runas 2024-05-23T00:03:28.360 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] truncfile 2024-05-23T00:03:28.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:27 smithi174 ceph-mon[36179]: pgmap v92: 129 pgs: 129 active+clean; 78 MiB data, 541 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 130 KiB/s wr, 20 op/s 2024-05-23T00:03:28.420 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] usemem 2024-05-23T00:03:28.693 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] mmapcat 2024-05-23T00:03:28.696 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] append_reader 2024-05-23T00:03:28.709 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] append_writer 2024-05-23T00:03:28.845 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] dirperf 2024-05-23T00:03:29.006 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] metaperf 2024-05-23T00:03:29.011 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] devzero 2024-05-23T00:03:29.026 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] feature 2024-05-23T00:03:29.385 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] alloc 2024-05-23T00:03:29.415 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] fault 2024-05-23T00:03:29.474 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] fstest 2024-05-23T00:03:29.645 INFO:tasks.workunit.client.0.smithi103.stdout: [CC] t_access_root 2024-05-23T00:03:29.734 INFO:tasks.workunit.client.0.smithi103.stderr:alloc.c: In function 'main': 2024-05-23T00:03:29.734 INFO:tasks.workunit.client.0.smithi103.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-05-23T00:03:29.734 INFO:tasks.workunit.client.0.smithi103.stderr: 154 | bozo! 2024-05-23T00:03:29.734 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~ 2024-05-23T00:03:29.734 INFO:tasks.workunit.client.0.smithi103.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-05-23T00:03:29.734 INFO:tasks.workunit.client.0.smithi103.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-05-23T00:03:29.735 INFO:tasks.workunit.client.0.smithi103.stderr: 154 | bozo! 2024-05-23T00:03:29.735 INFO:tasks.workunit.client.0.smithi103.stderr: | ^ 2024-05-23T00:03:29.735 INFO:tasks.workunit.client.0.smithi103.stderr: | ; 2024-05-23T00:03:29.735 INFO:tasks.workunit.client.0.smithi103.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-05-23T00:03:29.735 INFO:tasks.workunit.client.0.smithi103.stderr: 184 | rflag = 1; 2024-05-23T00:03:29.735 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~ 2024-05-23T00:03:29.735 INFO:tasks.workunit.client.0.smithi103.stderr: | nflag 2024-05-23T00:03:29.735 INFO:tasks.workunit.client.0.smithi103.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-05-23T00:03:29.736 INFO:tasks.workunit.client.0.smithi103.stderr: 320 | bozo! 2024-05-23T00:03:29.736 INFO:tasks.workunit.client.0.smithi103.stderr: | ^ 2024-05-23T00:03:29.736 INFO:tasks.workunit.client.0.smithi103.stderr: | ; 2024-05-23T00:03:29.736 INFO:tasks.workunit.client.0.smithi103.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-05-23T00:03:29.736 INFO:tasks.workunit.client.0.smithi103.stderr: 128 | struct flock64 f; 2024-05-23T00:03:29.736 INFO:tasks.workunit.client.0.smithi103.stderr: | ^ 2024-05-23T00:03:29.741 INFO:tasks.workunit.client.0.smithi103.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-05-23T00:03:29.741 INFO:tasks.workunit.client.0.smithi103.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr:fstest.c: In function 'run_child': 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~ 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~ 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-23T00:03:29.827 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~ 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~ 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-05-23T00:03:29.828 INFO:tasks.workunit.client.0.smithi103.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-05-23T00:03:29.942 INFO:tasks.workunit.client.0.smithi103.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-05-23T00:03:29.942 INFO:tasks.workunit.client.0.smithi103.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-05-23T00:03:29.942 INFO:tasks.workunit.client.0.smithi103.stderr:make: *** [Makefile:51: default] Error 2 2024-05-23T00:03:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-05-23T00:03:29.944 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-05-23T00:03:29.945 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-23T00:03:30.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:29 smithi105 ceph-mon[36426]: pgmap v93: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 152 KiB/s wr, 41 op/s 2024-05-23T00:03:30.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:29 smithi103 ceph-mon[31370]: pgmap v93: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 152 KiB/s wr, 41 op/s 2024-05-23T00:03:30.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:29 smithi174 ceph-mon[36179]: pgmap v93: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 152 KiB/s wr, 41 op/s 2024-05-23T00:03:31.060 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi103 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=935217f1cb4dc722d5303f5ad364c883e9133376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-05-23T00:03:31.292 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=944ffd0891f944b483c44adb67388137 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi103 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=935217f1cb4dc722d5303f5ad364c883e9133376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-05-23T00:03:31.295 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-23T00:03:31.305 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:03:31.331 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-05-23T00:03:31.346 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-05-23T00:03:31.346 INFO:teuthology.task.exec:Executing custom commands... 2024-05-23T00:03:31.346 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:03:31.347 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-05-23T00:03:31.471 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-05-23T00:03:32.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:31 smithi105 ceph-mon[36426]: pgmap v94: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 149 KiB/s wr, 47 op/s 2024-05-23T00:03:32.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:31 smithi103 ceph-mon[31370]: pgmap v94: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 149 KiB/s wr, 47 op/s 2024-05-23T00:03:32.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:31 smithi174 ceph-mon[36179]: pgmap v94: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 149 KiB/s wr, 47 op/s 2024-05-23T00:03:34.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:33 smithi105 ceph-mon[36426]: pgmap v95: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 149 KiB/s wr, 47 op/s 2024-05-23T00:03:34.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:33 smithi103 ceph-mon[31370]: pgmap v95: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 149 KiB/s wr, 47 op/s 2024-05-23T00:03:34.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:33 smithi174 ceph-mon[36179]: pgmap v95: 129 pgs: 129 active+clean; 79 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 149 KiB/s wr, 47 op/s 2024-05-23T00:03:36.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:35 smithi105 ceph-mon[36426]: pgmap v96: 129 pgs: 129 active+clean; 81 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 440 KiB/s wr, 89 op/s 2024-05-23T00:03:36.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:35 smithi103 ceph-mon[31370]: pgmap v96: 129 pgs: 129 active+clean; 81 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 440 KiB/s wr, 89 op/s 2024-05-23T00:03:36.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:35 smithi174 ceph-mon[36179]: pgmap v96: 129 pgs: 129 active+clean; 81 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 440 KiB/s wr, 89 op/s 2024-05-23T00:03:38.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:37 smithi105 ceph-mon[36426]: pgmap v97: 129 pgs: 129 active+clean; 81 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 315 KiB/s wr, 73 op/s 2024-05-23T00:03:38.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:37 smithi103 ceph-mon[31370]: pgmap v97: 129 pgs: 129 active+clean; 81 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 315 KiB/s wr, 73 op/s 2024-05-23T00:03:38.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:37 smithi174 ceph-mon[36179]: pgmap v97: 129 pgs: 129 active+clean; 81 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 315 KiB/s wr, 73 op/s 2024-05-23T00:03:39.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:38 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:39.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:38 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:39.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:38 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:40.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:39 smithi105 ceph-mon[36426]: pgmap v98: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 401 KiB/s wr, 81 op/s 2024-05-23T00:03:40.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:39 smithi103 ceph-mon[31370]: pgmap v98: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 401 KiB/s wr, 81 op/s 2024-05-23T00:03:40.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:39 smithi174 ceph-mon[36179]: pgmap v98: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 401 KiB/s wr, 81 op/s 2024-05-23T00:03:42.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:41 smithi105 ceph-mon[36426]: pgmap v99: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 379 KiB/s wr, 61 op/s 2024-05-23T00:03:42.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:41 smithi103 ceph-mon[31370]: pgmap v99: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 379 KiB/s wr, 61 op/s 2024-05-23T00:03:42.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:41 smithi174 ceph-mon[36179]: pgmap v99: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 379 KiB/s wr, 61 op/s 2024-05-23T00:03:43.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:42 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:43.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:42 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:43.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:42 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:03:44.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:43 smithi105 ceph-mon[36426]: pgmap v100: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 377 KiB/s wr, 51 op/s 2024-05-23T00:03:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:43 smithi103 ceph-mon[31370]: pgmap v100: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 377 KiB/s wr, 51 op/s 2024-05-23T00:03:44.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:43 smithi174 ceph-mon[36179]: pgmap v100: 129 pgs: 129 active+clean; 82 MiB data, 550 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 377 KiB/s wr, 51 op/s 2024-05-23T00:03:46.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:46 smithi105 ceph-mon[36426]: pgmap v101: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 510 KiB/s wr, 63 op/s 2024-05-23T00:03:46.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:46 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:03:46.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:46 smithi103 ceph-mon[31370]: pgmap v101: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 510 KiB/s wr, 63 op/s 2024-05-23T00:03:46.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:46 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:03:46.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:46 smithi174 ceph-mon[36179]: pgmap v101: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 510 KiB/s wr, 63 op/s 2024-05-23T00:03:46.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:46 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:03:47.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:47 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:03:47.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:47 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:03:47.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:47 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:03:47.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:47 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:03:47.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:47 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:03:47.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:47 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:03:47.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:47 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:03:47.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:47 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:03:47.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:47 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:03:48.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:48 smithi105 ceph-mon[36426]: pgmap v102: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 219 KiB/s wr, 21 op/s 2024-05-23T00:03:48.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:48 smithi103 ceph-mon[31370]: pgmap v102: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 219 KiB/s wr, 21 op/s 2024-05-23T00:03:48.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:48 smithi174 ceph-mon[36179]: pgmap v102: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 219 KiB/s wr, 21 op/s 2024-05-23T00:03:50.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:50 smithi105 ceph-mon[36426]: pgmap v103: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 221 KiB/s wr, 24 op/s 2024-05-23T00:03:50.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:50 smithi103 ceph-mon[31370]: pgmap v103: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 221 KiB/s wr, 24 op/s 2024-05-23T00:03:50.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:50 smithi174 ceph-mon[36179]: pgmap v103: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 221 KiB/s wr, 24 op/s 2024-05-23T00:03:52.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:52 smithi105 ceph-mon[36426]: pgmap v104: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 135 KiB/s wr, 16 op/s 2024-05-23T00:03:52.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:52 smithi103 ceph-mon[31370]: pgmap v104: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 135 KiB/s wr, 16 op/s 2024-05-23T00:03:52.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:52 smithi174 ceph-mon[36179]: pgmap v104: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 135 KiB/s wr, 16 op/s 2024-05-23T00:03:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:54 smithi105 ceph-mon[36426]: pgmap v105: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 135 KiB/s wr, 15 op/s 2024-05-23T00:03:54.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:54 smithi103 ceph-mon[31370]: pgmap v105: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 135 KiB/s wr, 15 op/s 2024-05-23T00:03:54.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:54 smithi174 ceph-mon[36179]: pgmap v105: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 135 KiB/s wr, 15 op/s 2024-05-23T00:03:56.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:56 smithi105 ceph-mon[36426]: pgmap v106: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 136 KiB/s wr, 18 op/s 2024-05-23T00:03:56.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:56 smithi103 ceph-mon[31370]: pgmap v106: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 136 KiB/s wr, 18 op/s 2024-05-23T00:03:56.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:56 smithi174 ceph-mon[36179]: pgmap v106: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 136 KiB/s wr, 18 op/s 2024-05-23T00:03:58.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:03:58 smithi105 ceph-mon[36426]: pgmap v107: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-05-23T00:03:58.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:03:58 smithi103 ceph-mon[31370]: pgmap v107: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-05-23T00:03:58.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:03:58 smithi174 ceph-mon[36179]: pgmap v107: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2024-05-23T00:04:00.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:00 smithi105 ceph-mon[36426]: pgmap v108: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-05-23T00:04:00.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:00 smithi103 ceph-mon[31370]: pgmap v108: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-05-23T00:04:00.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:00 smithi174 ceph-mon[36179]: pgmap v108: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.7 KiB/s wr, 8 op/s 2024-05-23T00:04:02.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:02 smithi105 ceph-mon[36426]: pgmap v109: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-05-23T00:04:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:02 smithi103 ceph-mon[31370]: pgmap v109: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-05-23T00:04:02.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:02 smithi174 ceph-mon[36179]: pgmap v109: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-05-23T00:04:04.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:04 smithi103 ceph-mon[31370]: pgmap v110: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-05-23T00:04:04.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:04 smithi174 ceph-mon[36179]: pgmap v110: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-05-23T00:04:04.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:04 smithi105 ceph-mon[36426]: pgmap v110: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-05-23T00:04:06.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:06 smithi103 ceph-mon[31370]: pgmap v111: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-05-23T00:04:06.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:06 smithi174 ceph-mon[36179]: pgmap v111: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-05-23T00:04:06.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:06 smithi105 ceph-mon[36426]: pgmap v111: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-05-23T00:04:08.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:08 smithi103 ceph-mon[31370]: pgmap v112: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-05-23T00:04:08.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:08 smithi174 ceph-mon[36179]: pgmap v112: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-05-23T00:04:08.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:08 smithi105 ceph-mon[36426]: pgmap v112: 129 pgs: 129 active+clean; 83 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-05-23T00:04:10.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:10 smithi174 ceph-mon[36179]: pgmap v113: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-05-23T00:04:10.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:10 smithi105 ceph-mon[36426]: pgmap v113: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-05-23T00:04:10.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:10 smithi103 ceph-mon[31370]: pgmap v113: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 27 KiB/s wr, 23 op/s 2024-05-23T00:04:12.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:12 smithi174 ceph-mon[36179]: pgmap v114: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:04:12.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:12 smithi105 ceph-mon[36426]: pgmap v114: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:04:12.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:12 smithi103 ceph-mon[31370]: pgmap v114: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:04:14.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:14 smithi174 ceph-mon[36179]: pgmap v115: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-05-23T00:04:14.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:14 smithi105 ceph-mon[36426]: pgmap v115: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-05-23T00:04:14.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:14 smithi103 ceph-mon[31370]: pgmap v115: 129 pgs: 129 active+clean; 84 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-05-23T00:04:16.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:16 smithi174 ceph-mon[36179]: pgmap v116: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-05-23T00:04:16.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:16 smithi105 ceph-mon[36426]: pgmap v116: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-05-23T00:04:16.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:16 smithi103 ceph-mon[31370]: pgmap v116: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 32 KiB/s wr, 24 op/s 2024-05-23T00:04:18.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:18 smithi174 ceph-mon[36179]: pgmap v117: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-23T00:04:18.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:18 smithi105 ceph-mon[36426]: pgmap v117: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-23T00:04:18.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:18 smithi103 ceph-mon[31370]: pgmap v117: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2024-05-23T00:04:20.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:20 smithi174 ceph-mon[36179]: pgmap v118: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-05-23T00:04:20.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:20 smithi105 ceph-mon[36426]: pgmap v118: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-05-23T00:04:20.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:20 smithi103 ceph-mon[31370]: pgmap v118: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 9 op/s 2024-05-23T00:04:22.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:22 smithi174 ceph-mon[36179]: pgmap v119: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-05-23T00:04:22.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:22 smithi105 ceph-mon[36426]: pgmap v119: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-05-23T00:04:22.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:22 smithi103 ceph-mon[31370]: pgmap v119: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-05-23T00:04:24.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:24 smithi105 ceph-mon[36426]: pgmap v120: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-05-23T00:04:24.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:24 smithi103 ceph-mon[31370]: pgmap v120: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-05-23T00:04:24.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:24 smithi174 ceph-mon[36179]: pgmap v120: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 6.7 KiB/s wr, 6 op/s 2024-05-23T00:04:26.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:26 smithi105 ceph-mon[36426]: pgmap v121: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-05-23T00:04:26.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:26 smithi103 ceph-mon[31370]: pgmap v121: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-05-23T00:04:26.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:26 smithi174 ceph-mon[36179]: pgmap v121: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-05-23T00:04:28.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:28 smithi105 ceph-mon[36426]: pgmap v122: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-23T00:04:28.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:28 smithi103 ceph-mon[31370]: pgmap v122: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-23T00:04:28.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:28 smithi174 ceph-mon[36179]: pgmap v122: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-23T00:04:30.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:30 smithi105 ceph-mon[36426]: pgmap v123: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:04:30.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:30 smithi103 ceph-mon[31370]: pgmap v123: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:04:30.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:30 smithi174 ceph-mon[36179]: pgmap v123: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:04:32.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:32 smithi105 ceph-mon[36426]: pgmap v124: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:04:32.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:32 smithi103 ceph-mon[31370]: pgmap v124: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:04:32.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:32 smithi174 ceph-mon[36179]: pgmap v124: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:04:34.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:34 smithi105 ceph-mon[36426]: pgmap v125: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:04:34.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:34 smithi103 ceph-mon[31370]: pgmap v125: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:04:34.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:34 smithi174 ceph-mon[36179]: pgmap v125: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2024-05-23T00:04:36.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:36 smithi105 ceph-mon[36426]: pgmap v126: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 9 op/s 2024-05-23T00:04:36.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:36 smithi103 ceph-mon[31370]: pgmap v126: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 9 op/s 2024-05-23T00:04:36.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:36 smithi174 ceph-mon[36179]: pgmap v126: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.9 KiB/s wr, 9 op/s 2024-05-23T00:04:37.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:37 smithi103 ceph-mon[31370]: pgmap v127: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-05-23T00:04:37.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:37 smithi174 ceph-mon[36179]: pgmap v127: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-05-23T00:04:38.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:37 smithi105 ceph-mon[36426]: pgmap v127: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-05-23T00:04:40.117 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:39 smithi174 ceph-mon[36179]: pgmap v128: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.1 KiB/s wr, 9 op/s 2024-05-23T00:04:40.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:39 smithi105 ceph-mon[36426]: pgmap v128: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.1 KiB/s wr, 9 op/s 2024-05-23T00:04:40.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:39 smithi103 ceph-mon[31370]: pgmap v128: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 6.1 KiB/s wr, 9 op/s 2024-05-23T00:04:42.116 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:41 smithi174 ceph-mon[36179]: pgmap v129: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-05-23T00:04:42.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:41 smithi105 ceph-mon[36426]: pgmap v129: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-05-23T00:04:42.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:41 smithi103 ceph-mon[31370]: pgmap v129: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2024-05-23T00:04:44.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:43 smithi105 ceph-mon[36426]: pgmap v130: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.6 KiB/s wr, 6 op/s 2024-05-23T00:04:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:43 smithi103 ceph-mon[31370]: pgmap v130: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.6 KiB/s wr, 6 op/s 2024-05-23T00:04:44.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:43 smithi174 ceph-mon[36179]: pgmap v130: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.6 KiB/s wr, 6 op/s 2024-05-23T00:04:46.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:45 smithi105 ceph-mon[36426]: pgmap v131: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-05-23T00:04:46.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:45 smithi103 ceph-mon[31370]: pgmap v131: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-05-23T00:04:46.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:45 smithi174 ceph-mon[36179]: pgmap v131: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 9 op/s 2024-05-23T00:04:48.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:47 smithi105 ceph-mon[36426]: pgmap v132: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-05-23T00:04:48.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:47 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:04:48.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:47 smithi103 ceph-mon[31370]: pgmap v132: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-05-23T00:04:48.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:47 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:04:48.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:47 smithi174 ceph-mon[36179]: pgmap v132: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-05-23T00:04:48.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:47 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:04:49.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:48 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:04:49.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:48 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:04:49.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:48 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:04:49.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:48 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:04:49.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:48 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:04:49.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:48 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:04:49.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:48 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:04:49.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:48 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:04:49.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:48 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:04:50.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:50 smithi105 ceph-mon[36426]: pgmap v133: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-05-23T00:04:50.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:50 smithi103 ceph-mon[31370]: pgmap v133: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-05-23T00:04:50.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:50 smithi174 ceph-mon[36179]: pgmap v133: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2024-05-23T00:04:52.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:52 smithi105 ceph-mon[36426]: pgmap v134: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-05-23T00:04:52.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:52 smithi103 ceph-mon[31370]: pgmap v134: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-05-23T00:04:52.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:52 smithi174 ceph-mon[36179]: pgmap v134: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-05-23T00:04:54.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:54 smithi105 ceph-mon[36426]: pgmap v135: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 5 op/s 2024-05-23T00:04:54.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:54 smithi103 ceph-mon[31370]: pgmap v135: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 5 op/s 2024-05-23T00:04:54.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:54 smithi174 ceph-mon[36179]: pgmap v135: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.8 KiB/s wr, 5 op/s 2024-05-23T00:04:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:56 smithi105 ceph-mon[36426]: pgmap v136: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 8 op/s 2024-05-23T00:04:56.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:56 smithi103 ceph-mon[31370]: pgmap v136: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 8 op/s 2024-05-23T00:04:56.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:56 smithi174 ceph-mon[36179]: pgmap v136: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 8 op/s 2024-05-23T00:04:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:04:58 smithi105 ceph-mon[36426]: pgmap v137: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2024-05-23T00:04:58.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:04:58 smithi103 ceph-mon[31370]: pgmap v137: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2024-05-23T00:04:58.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:04:58 smithi174 ceph-mon[36179]: pgmap v137: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2024-05-23T00:05:00.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:00 smithi105 ceph-mon[36426]: pgmap v138: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-23T00:05:00.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:00 smithi103 ceph-mon[31370]: pgmap v138: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-23T00:05:00.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:00 smithi174 ceph-mon[36179]: pgmap v138: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2024-05-23T00:05:02.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:02 smithi105 ceph-mon[36426]: pgmap v139: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.9 KiB/s wr, 6 op/s 2024-05-23T00:05:02.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:02 smithi103 ceph-mon[31370]: pgmap v139: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.9 KiB/s wr, 6 op/s 2024-05-23T00:05:02.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:02 smithi174 ceph-mon[36179]: pgmap v139: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.9 KiB/s wr, 6 op/s 2024-05-23T00:05:04.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:04 smithi105 ceph-mon[36426]: pgmap v140: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 KiB/s wr, 6 op/s 2024-05-23T00:05:04.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:04 smithi103 ceph-mon[31370]: pgmap v140: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 KiB/s wr, 6 op/s 2024-05-23T00:05:04.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:04 smithi174 ceph-mon[36179]: pgmap v140: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.9 KiB/s wr, 6 op/s 2024-05-23T00:05:06.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:06 smithi105 ceph-mon[36426]: pgmap v141: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-05-23T00:05:06.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:06 smithi103 ceph-mon[31370]: pgmap v141: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-05-23T00:05:06.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:06 smithi174 ceph-mon[36179]: pgmap v141: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-05-23T00:05:08.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:08 smithi105 ceph-mon[36426]: pgmap v142: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:05:08.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:08 smithi103 ceph-mon[31370]: pgmap v142: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:05:08.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:08 smithi174 ceph-mon[36179]: pgmap v142: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:05:10.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:10 smithi105 ceph-mon[36426]: pgmap v143: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-05-23T00:05:10.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:10 smithi103 ceph-mon[31370]: pgmap v143: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-05-23T00:05:10.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:10 smithi174 ceph-mon[36179]: pgmap v143: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 29 KiB/s wr, 23 op/s 2024-05-23T00:05:12.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:12 smithi105 ceph-mon[36426]: pgmap v144: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:05:12.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:12 smithi103 ceph-mon[31370]: pgmap v144: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:05:12.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:12 smithi174 ceph-mon[36179]: pgmap v144: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-05-23T00:05:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:14 smithi105 ceph-mon[36426]: pgmap v145: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-05-23T00:05:14.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:14 smithi103 ceph-mon[31370]: pgmap v145: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-05-23T00:05:14.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:14 smithi174 ceph-mon[36179]: pgmap v145: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 26 KiB/s wr, 20 op/s 2024-05-23T00:05:16.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:16 smithi105 ceph-mon[36426]: pgmap v146: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-05-23T00:05:16.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:16 smithi103 ceph-mon[31370]: pgmap v146: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-05-23T00:05:16.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:16 smithi174 ceph-mon[36179]: pgmap v146: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-05-23T00:05:18.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:18 smithi105 ceph-mon[36426]: pgmap v147: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.3 KiB/s wr, 5 op/s 2024-05-23T00:05:18.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:18 smithi103 ceph-mon[31370]: pgmap v147: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.3 KiB/s wr, 5 op/s 2024-05-23T00:05:18.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:18 smithi174 ceph-mon[36179]: pgmap v147: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.3 KiB/s wr, 5 op/s 2024-05-23T00:05:20.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:20 smithi105 ceph-mon[36426]: pgmap v148: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-05-23T00:05:20.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:20 smithi103 ceph-mon[31370]: pgmap v148: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-05-23T00:05:20.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:20 smithi174 ceph-mon[36179]: pgmap v148: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.5 KiB/s wr, 7 op/s 2024-05-23T00:05:22.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:22 smithi105 ceph-mon[36426]: pgmap v149: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-05-23T00:05:22.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:22 smithi103 ceph-mon[31370]: pgmap v149: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-05-23T00:05:22.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:22 smithi174 ceph-mon[36179]: pgmap v149: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-05-23T00:05:24.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:24 smithi105 ceph-mon[36426]: pgmap v150: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-05-23T00:05:24.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:24 smithi103 ceph-mon[31370]: pgmap v150: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-05-23T00:05:24.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:24 smithi174 ceph-mon[36179]: pgmap v150: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-05-23T00:05:26.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:26 smithi105 ceph-mon[36426]: pgmap v151: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:05:26.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:26 smithi103 ceph-mon[31370]: pgmap v151: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:05:26.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:26 smithi174 ceph-mon[36179]: pgmap v151: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-23T00:05:28.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:28 smithi105 ceph-mon[36426]: pgmap v152: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-05-23T00:05:28.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:28 smithi103 ceph-mon[31370]: pgmap v152: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-05-23T00:05:28.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:28 smithi174 ceph-mon[36179]: pgmap v152: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-05-23T00:05:30.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:30 smithi105 ceph-mon[36426]: pgmap v153: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-05-23T00:05:30.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:30 smithi103 ceph-mon[31370]: pgmap v153: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-05-23T00:05:30.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:30 smithi174 ceph-mon[36179]: pgmap v153: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2024-05-23T00:05:32.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:32 smithi105 ceph-mon[36426]: pgmap v154: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-23T00:05:32.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:32 smithi103 ceph-mon[31370]: pgmap v154: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-23T00:05:32.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:32 smithi174 ceph-mon[36179]: pgmap v154: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-23T00:05:33.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:33 smithi105 ceph-mon[36426]: pgmap v155: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-23T00:05:33.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:33 smithi103 ceph-mon[31370]: pgmap v155: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-23T00:05:33.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:33 smithi174 ceph-mon[36179]: pgmap v155: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-23T00:05:36.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:35 smithi105 ceph-mon[36426]: pgmap v156: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-05-23T00:05:36.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:35 smithi103 ceph-mon[31370]: pgmap v156: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-05-23T00:05:36.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:35 smithi174 ceph-mon[36179]: pgmap v156: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2024-05-23T00:05:38.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:37 smithi105 ceph-mon[36426]: pgmap v157: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:05:38.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:37 smithi103 ceph-mon[31370]: pgmap v157: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:05:38.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:37 smithi174 ceph-mon[36179]: pgmap v157: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:05:40.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:39 smithi105 ceph-mon[36426]: pgmap v158: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-23T00:05:40.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:39 smithi103 ceph-mon[31370]: pgmap v158: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-23T00:05:40.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:39 smithi174 ceph-mon[36179]: pgmap v158: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-23T00:05:42.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:41 smithi105 ceph-mon[36426]: pgmap v159: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:05:42.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:41 smithi103 ceph-mon[31370]: pgmap v159: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:05:42.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:41 smithi174 ceph-mon[36179]: pgmap v159: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:05:44.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:43 smithi105 ceph-mon[36426]: pgmap v160: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:43 smithi103 ceph-mon[31370]: pgmap v160: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:44.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:43 smithi174 ceph-mon[36179]: pgmap v160: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:46.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:45 smithi105 ceph-mon[36426]: pgmap v161: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:05:46.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:45 smithi103 ceph-mon[31370]: pgmap v161: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:05:46.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:45 smithi174 ceph-mon[36179]: pgmap v161: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:05:48.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:47 smithi105 ceph-mon[36426]: pgmap v162: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:48.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:47 smithi103 ceph-mon[31370]: pgmap v162: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:48.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:47 smithi174 ceph-mon[36179]: pgmap v162: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:49.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:48 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:05:49.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:48 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:05:49.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:48 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:05:50.268 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:49 smithi174 ceph-mon[36179]: pgmap v163: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:05:50.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:49 smithi105 ceph-mon[36426]: pgmap v163: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:05:50.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:49 smithi103 ceph-mon[31370]: pgmap v163: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:05:51.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:50 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:05:51.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:50 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:05:51.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:50 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:05:51.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:50 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:05:51.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:50 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:05:51.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:50 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:05:51.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:50 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:05:51.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:50 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:05:51.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:50 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:05:52.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:51 smithi105 ceph-mon[36426]: pgmap v164: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:05:52.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:51 smithi103 ceph-mon[31370]: pgmap v164: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:05:52.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:51 smithi174 ceph-mon[36179]: pgmap v164: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:05:54.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:53 smithi105 ceph-mon[36426]: pgmap v165: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:54.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:53 smithi103 ceph-mon[31370]: pgmap v165: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:54.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:53 smithi174 ceph-mon[36179]: pgmap v165: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:56.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:55 smithi105 ceph-mon[36426]: pgmap v166: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:05:56.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:56 smithi103 ceph-mon[31370]: pgmap v166: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:05:56.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:56 smithi174 ceph-mon[36179]: pgmap v166: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:05:58.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:05:58 smithi105 ceph-mon[36426]: pgmap v167: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:58.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:05:58 smithi103 ceph-mon[31370]: pgmap v167: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:05:58.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:05:58 smithi174 ceph-mon[36179]: pgmap v167: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:00.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:00 smithi105 ceph-mon[36426]: pgmap v168: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:00.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:00 smithi103 ceph-mon[31370]: pgmap v168: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:00.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:00 smithi174 ceph-mon[36179]: pgmap v168: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:02.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:02 smithi105 ceph-mon[36426]: pgmap v169: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:02 smithi103 ceph-mon[31370]: pgmap v169: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:02.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:02 smithi174 ceph-mon[36179]: pgmap v169: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:04.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:04 smithi105 ceph-mon[36426]: pgmap v170: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:04.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:04 smithi103 ceph-mon[31370]: pgmap v170: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:04.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:04 smithi174 ceph-mon[36179]: pgmap v170: 129 pgs: 129 active+clean; 84 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:06.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:06 smithi105 ceph-mon[36426]: pgmap v171: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:06.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:06 smithi103 ceph-mon[31370]: pgmap v171: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:06.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:06 smithi174 ceph-mon[36179]: pgmap v171: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:08 smithi105 ceph-mon[36426]: pgmap v172: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-23T00:06:08.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:08 smithi103 ceph-mon[31370]: pgmap v172: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-23T00:06:08.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:08 smithi174 ceph-mon[36179]: pgmap v172: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-05-23T00:06:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:10 smithi105 ceph-mon[36426]: pgmap v173: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:10.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:10 smithi103 ceph-mon[31370]: pgmap v173: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:10.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:10 smithi174 ceph-mon[36179]: pgmap v173: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:12.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:12 smithi105 ceph-mon[36426]: pgmap v174: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-23T00:06:12.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:12 smithi103 ceph-mon[31370]: pgmap v174: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-23T00:06:12.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:12 smithi174 ceph-mon[36179]: pgmap v174: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-23T00:06:14.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:14 smithi105 ceph-mon[36426]: pgmap v175: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-23T00:06:14.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:14 smithi103 ceph-mon[31370]: pgmap v175: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-23T00:06:14.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:14 smithi174 ceph-mon[36179]: pgmap v175: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-05-23T00:06:16.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:16 smithi105 ceph-mon[36426]: pgmap v176: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:16.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:16 smithi103 ceph-mon[31370]: pgmap v176: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:16.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:16 smithi174 ceph-mon[36179]: pgmap v176: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-05-23T00:06:17.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:17 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:17.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:17 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:17.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:17 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:18.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:18 smithi103 ceph-mon[31370]: pgmap v177: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-23T00:06:18.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:18 smithi174 ceph-mon[36179]: pgmap v177: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-23T00:06:18.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:18 smithi105 ceph-mon[36426]: pgmap v177: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-23T00:06:20.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:20 smithi174 ceph-mon[36179]: pgmap v178: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-23T00:06:20.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:20 smithi105 ceph-mon[36426]: pgmap v178: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-23T00:06:20.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:20 smithi103 ceph-mon[31370]: pgmap v178: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-23T00:06:22.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:22 smithi174 ceph-mon[36179]: pgmap v179: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:22.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:22 smithi105 ceph-mon[36426]: pgmap v179: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:22.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:22 smithi103 ceph-mon[31370]: pgmap v179: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:24.360 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:24 smithi105 ceph-mon[36426]: pgmap v180: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:24.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:24 smithi174 ceph-mon[36179]: pgmap v180: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:24.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:24 smithi103 ceph-mon[31370]: pgmap v180: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:26.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:26 smithi174 ceph-mon[36179]: pgmap v181: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:26.381 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:26 smithi103 ceph-mon[31370]: pgmap v181: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:26 smithi105 ceph-mon[36426]: pgmap v181: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:27.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:27 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:27.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:27 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:27.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:27 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:28.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:28 smithi105 ceph-mon[36426]: pgmap v182: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:28.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:28 smithi103 ceph-mon[31370]: pgmap v182: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:28.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:28 smithi174 ceph-mon[36179]: pgmap v182: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:30.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:30 smithi105 ceph-mon[36426]: pgmap v183: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:30.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:30 smithi103 ceph-mon[31370]: pgmap v183: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:30.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:30 smithi174 ceph-mon[36179]: pgmap v183: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:32.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:32 smithi105 ceph-mon[36426]: pgmap v184: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:32.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:32 smithi103 ceph-mon[31370]: pgmap v184: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:32.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:32 smithi174 ceph-mon[36179]: pgmap v184: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:33.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:33 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:33.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:33 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:33.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:33 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:34.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:34 smithi105 ceph-mon[36426]: pgmap v185: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:34.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:34 smithi103 ceph-mon[31370]: pgmap v185: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:34.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:34 smithi174 ceph-mon[36179]: pgmap v185: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:36.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:36 smithi105 ceph-mon[36426]: pgmap v186: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:36.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:36 smithi103 ceph-mon[31370]: pgmap v186: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:36.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:36 smithi174 ceph-mon[36179]: pgmap v186: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:38.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:38 smithi105 ceph-mon[36426]: pgmap v187: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:38.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:38 smithi103 ceph-mon[31370]: pgmap v187: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:38.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:38 smithi174 ceph-mon[36179]: pgmap v187: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:40.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:40 smithi105 ceph-mon[36426]: pgmap v188: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:40.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:40 smithi103 ceph-mon[31370]: pgmap v188: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:40.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:40 smithi174 ceph-mon[36179]: pgmap v188: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:42.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:42 smithi105 ceph-mon[36426]: pgmap v189: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:42.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:42 smithi103 ceph-mon[31370]: pgmap v189: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:42.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:42 smithi174 ceph-mon[36179]: pgmap v189: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:44.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:44 smithi105 ceph-mon[36426]: pgmap v190: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:44.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:44 smithi103 ceph-mon[31370]: pgmap v190: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:44.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:44 smithi174 ceph-mon[36179]: pgmap v190: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:46.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:46 smithi105 ceph-mon[36426]: pgmap v191: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:46.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:46 smithi103 ceph-mon[31370]: pgmap v191: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:46.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:46 smithi174 ceph-mon[36179]: pgmap v191: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:48.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:48 smithi105 ceph-mon[36426]: pgmap v192: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:48.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:48 smithi103 ceph-mon[31370]: pgmap v192: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:48.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:48 smithi174 ceph-mon[36179]: pgmap v192: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:50.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:50 smithi105 ceph-mon[36426]: pgmap v193: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:50.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:50 smithi103 ceph-mon[31370]: pgmap v193: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:50.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:50 smithi174 ceph-mon[36179]: pgmap v193: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:06:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:51 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:06:51.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:51 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:06:51.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:51 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:06:52.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:52 smithi105 ceph-mon[36426]: pgmap v194: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:52.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:52 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:06:52.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:52 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:06:52.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:52 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:06:52.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:52 smithi103 ceph-mon[31370]: pgmap v194: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:52.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:52 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:06:52.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:52 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:06:52.580 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:52 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:06:52.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:52 smithi174 ceph-mon[36179]: pgmap v194: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:06:52.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:52 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:06:52.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:52 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:06:52.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:52 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:06:53.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:53 smithi174 ceph-mon[36179]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:53.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:53 smithi174 ceph-mon[36179]: pgmap v195: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:54.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:53 smithi105 ceph-mon[36426]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:54.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:53 smithi105 ceph-mon[36426]: pgmap v195: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:54.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:53 smithi103 ceph-mon[31370]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-23T00:06:54.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:53 smithi103 ceph-mon[31370]: pgmap v195: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:56.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:55 smithi105 ceph-mon[36426]: pgmap v196: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:56.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:55 smithi103 ceph-mon[31370]: pgmap v196: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:56.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:55 smithi174 ceph-mon[36179]: pgmap v196: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:06:58.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:57 smithi105 ceph-mon[36426]: pgmap v197: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:58.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:57 smithi103 ceph-mon[31370]: pgmap v197: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:06:58.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:57 smithi174 ceph-mon[36179]: pgmap v197: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:00.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:06:59 smithi105 ceph-mon[36426]: pgmap v198: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:00.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:06:59 smithi103 ceph-mon[31370]: pgmap v198: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:00.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:06:59 smithi174 ceph-mon[36179]: pgmap v198: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:02.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:01 smithi105 ceph-mon[36426]: pgmap v199: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:02.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:01 smithi103 ceph-mon[31370]: pgmap v199: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:02.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:01 smithi174 ceph-mon[36179]: pgmap v199: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:04.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:03 smithi105 ceph-mon[36426]: pgmap v200: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:04.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:03 smithi103 ceph-mon[31370]: pgmap v200: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:04.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:03 smithi174 ceph-mon[36179]: pgmap v200: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:06.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:05 smithi105 ceph-mon[36426]: pgmap v201: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-23T00:07:06.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:05 smithi103 ceph-mon[31370]: pgmap v201: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-23T00:07:06.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:05 smithi174 ceph-mon[36179]: pgmap v201: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-23T00:07:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:07 smithi105 ceph-mon[36426]: pgmap v202: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-23T00:07:08.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:07 smithi103 ceph-mon[31370]: pgmap v202: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-23T00:07:08.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:07 smithi174 ceph-mon[36179]: pgmap v202: 129 pgs: 129 active+clean; 84 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-23T00:07:09.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:08 smithi105 ceph-mon[36426]: osdmap e97: 12 total, 12 up, 12 in 2024-05-23T00:07:09.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:08 smithi103 ceph-mon[31370]: osdmap e97: 12 total, 12 up, 12 in 2024-05-23T00:07:09.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:08 smithi174 ceph-mon[36179]: osdmap e97: 12 total, 12 up, 12 in 2024-05-23T00:07:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:09 smithi105 ceph-mon[36426]: pgmap v204: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:10.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:09 smithi103 ceph-mon[31370]: pgmap v204: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:10.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:09 smithi174 ceph-mon[36179]: pgmap v204: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:12.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:11 smithi105 ceph-mon[36426]: pgmap v205: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:12.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:11 smithi103 ceph-mon[31370]: pgmap v205: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:12.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:11 smithi174 ceph-mon[36179]: pgmap v205: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:14.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:14 smithi105 ceph-mon[36426]: pgmap v206: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:14.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:14 smithi105 ceph-mon[36426]: osdmap e98: 12 total, 12 up, 12 in 2024-05-23T00:07:14.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:14 smithi103 ceph-mon[31370]: pgmap v206: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:14.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:14 smithi103 ceph-mon[31370]: osdmap e98: 12 total, 12 up, 12 in 2024-05-23T00:07:14.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:14 smithi174 ceph-mon[36179]: pgmap v206: 129 pgs: 129 active+clean; 84 MiB data, 566 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:07:14.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:14 smithi174 ceph-mon[36179]: osdmap e98: 12 total, 12 up, 12 in 2024-05-23T00:07:16.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:16 smithi105 ceph-mon[36426]: pgmap v208: 129 pgs: 129 active+clean; 84 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:07:16.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:16 smithi103 ceph-mon[31370]: pgmap v208: 129 pgs: 129 active+clean; 84 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:07:16.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:16 smithi174 ceph-mon[36179]: pgmap v208: 129 pgs: 129 active+clean; 84 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:07:18.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:18 smithi105 ceph-mon[36426]: pgmap v209: 129 pgs: 129 active+clean; 84 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2024-05-23T00:07:18.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:18 smithi103 ceph-mon[31370]: pgmap v209: 129 pgs: 129 active+clean; 84 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2024-05-23T00:07:18.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:18 smithi174 ceph-mon[36179]: pgmap v209: 129 pgs: 129 active+clean; 84 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 229 B/s wr, 7 op/s 2024-05-23T00:07:20.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:20 smithi105 ceph-mon[36426]: pgmap v210: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:20.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:20 smithi103 ceph-mon[31370]: pgmap v210: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:20.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:20 smithi174 ceph-mon[36179]: pgmap v210: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:22.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:22 smithi105 ceph-mon[36426]: pgmap v211: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:22.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:22 smithi103 ceph-mon[31370]: pgmap v211: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:22.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:22 smithi174 ceph-mon[36179]: pgmap v211: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:24.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:24 smithi105 ceph-mon[36426]: pgmap v212: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:24.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:24 smithi103 ceph-mon[31370]: pgmap v212: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:24.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:24 smithi174 ceph-mon[36179]: pgmap v212: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:07:26.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:26 smithi105 ceph-mon[36426]: pgmap v213: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:07:26.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:26 smithi103 ceph-mon[31370]: pgmap v213: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:07:26.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:26 smithi174 ceph-mon[36179]: pgmap v213: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:07:28.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:28 smithi105 ceph-mon[36426]: pgmap v214: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:28.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:28 smithi103 ceph-mon[31370]: pgmap v214: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:28.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:28 smithi174 ceph-mon[36179]: pgmap v214: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:30.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:30 smithi105 ceph-mon[36426]: pgmap v215: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:30.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:30 smithi103 ceph-mon[31370]: pgmap v215: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:30.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:30 smithi174 ceph-mon[36179]: pgmap v215: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:32.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:32 smithi105 ceph-mon[36426]: pgmap v216: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:32.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:32 smithi103 ceph-mon[31370]: pgmap v216: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:32.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:32 smithi174 ceph-mon[36179]: pgmap v216: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:34.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:34 smithi105 ceph-mon[36426]: pgmap v217: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:34.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:34 smithi103 ceph-mon[31370]: pgmap v217: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:34.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:34 smithi174 ceph-mon[36179]: pgmap v217: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:36.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:36 smithi103 ceph-mon[31370]: pgmap v218: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:36.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:36 smithi174 ceph-mon[36179]: pgmap v218: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:36.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:36 smithi105 ceph-mon[36426]: pgmap v218: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:38.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:38 smithi103 ceph-mon[31370]: pgmap v219: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:38.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:38 smithi174 ceph-mon[36179]: pgmap v219: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:38.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:38 smithi105 ceph-mon[36426]: pgmap v219: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:39.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:39 smithi103 ceph-mon[31370]: pgmap v220: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:39.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:39 smithi174 ceph-mon[36179]: pgmap v220: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:40.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:39 smithi105 ceph-mon[36426]: pgmap v220: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:42.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:41 smithi105 ceph-mon[36426]: pgmap v221: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:42.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:41 smithi103 ceph-mon[31370]: pgmap v221: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:42.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:41 smithi174 ceph-mon[36179]: pgmap v221: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:44.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:43 smithi105 ceph-mon[36426]: pgmap v222: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:44.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:43 smithi103 ceph-mon[31370]: pgmap v222: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:44.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:43 smithi174 ceph-mon[36179]: pgmap v222: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:46.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:45 smithi105 ceph-mon[36426]: pgmap v223: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:46.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:45 smithi103 ceph-mon[31370]: pgmap v223: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:46.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:45 smithi174 ceph-mon[36179]: pgmap v223: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:48.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:47 smithi105 ceph-mon[36426]: pgmap v224: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:48.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:47 smithi103 ceph-mon[31370]: pgmap v224: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:48.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:47 smithi174 ceph-mon[36179]: pgmap v224: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:50.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:49 smithi105 ceph-mon[36426]: pgmap v225: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:50.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:49 smithi103 ceph-mon[31370]: pgmap v225: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:50.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:49 smithi174 ceph-mon[36179]: pgmap v225: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:07:52.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:51 smithi105 ceph-mon[36426]: pgmap v226: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:52.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:51 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:07:52.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:51 smithi103 ceph-mon[31370]: pgmap v226: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:52.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:51 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:07:52.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:51 smithi174 ceph-mon[36179]: pgmap v226: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:07:52.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:51 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:07:54.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:53 smithi105 ceph-mon[36426]: pgmap v227: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:54.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:53 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:07:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:53 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:07:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:53 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:07:54.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:53 smithi103 ceph-mon[31370]: pgmap v227: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:54.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:53 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:07:54.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:53 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:07:54.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:53 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:07:54.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:53 smithi174 ceph-mon[36179]: pgmap v227: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:54.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:53 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:07:54.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:53 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:07:54.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:53 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:07:56.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:55 smithi105 ceph-mon[36426]: pgmap v228: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:56.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:55 smithi103 ceph-mon[31370]: pgmap v228: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:56.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:55 smithi174 ceph-mon[36179]: pgmap v228: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:07:58.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:57 smithi105 ceph-mon[36426]: pgmap v229: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:58.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:57 smithi103 ceph-mon[31370]: pgmap v229: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:07:58.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:57 smithi174 ceph-mon[36179]: pgmap v229: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:00.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:07:59 smithi105 ceph-mon[36426]: pgmap v230: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:00.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:07:59 smithi103 ceph-mon[31370]: pgmap v230: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:00.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:07:59 smithi174 ceph-mon[36179]: pgmap v230: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:02.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:01 smithi105 ceph-mon[36426]: pgmap v231: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:02.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:01 smithi103 ceph-mon[31370]: pgmap v231: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:02.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:01 smithi174 ceph-mon[36179]: pgmap v231: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:04.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:04 smithi105 ceph-mon[36426]: pgmap v232: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:04.330 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:04 smithi103 ceph-mon[31370]: pgmap v232: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:04.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:04 smithi174 ceph-mon[36179]: pgmap v232: 129 pgs: 129 active+clean; 84 MiB data, 575 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:06.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:06 smithi105 ceph-mon[36426]: pgmap v233: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-23T00:08:06.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:06 smithi103 ceph-mon[31370]: pgmap v233: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-23T00:08:06.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:06 smithi174 ceph-mon[36179]: pgmap v233: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-05-23T00:08:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:08 smithi105 ceph-mon[36426]: pgmap v234: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-23T00:08:08.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:08 smithi105 ceph-mon[36426]: osdmap e99: 12 total, 12 up, 12 in 2024-05-23T00:08:08.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:08 smithi103 ceph-mon[31370]: pgmap v234: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-23T00:08:08.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:08 smithi103 ceph-mon[31370]: osdmap e99: 12 total, 12 up, 12 in 2024-05-23T00:08:08.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:08 smithi174 ceph-mon[36179]: pgmap v234: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-05-23T00:08:08.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:08 smithi174 ceph-mon[36179]: osdmap e99: 12 total, 12 up, 12 in 2024-05-23T00:08:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:10 smithi105 ceph-mon[36426]: pgmap v236: 129 pgs: 129 active+clean; 84 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:10.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:10 smithi103 ceph-mon[31370]: pgmap v236: 129 pgs: 129 active+clean; 84 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:10.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:10 smithi174 ceph-mon[36179]: pgmap v236: 129 pgs: 129 active+clean; 84 MiB data, 586 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:12.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:12 smithi105 ceph-mon[36426]: pgmap v237: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:12.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:12 smithi103 ceph-mon[31370]: pgmap v237: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:12.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:12 smithi174 ceph-mon[36179]: pgmap v237: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:14.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:14 smithi105 ceph-mon[36426]: pgmap v238: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:14.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:14 smithi105 ceph-mon[36426]: osdmap e100: 12 total, 12 up, 12 in 2024-05-23T00:08:14.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:14 smithi103 ceph-mon[31370]: pgmap v238: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:14.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:14 smithi103 ceph-mon[31370]: osdmap e100: 12 total, 12 up, 12 in 2024-05-23T00:08:14.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:14 smithi174 ceph-mon[36179]: pgmap v238: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-05-23T00:08:14.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:14 smithi174 ceph-mon[36179]: osdmap e100: 12 total, 12 up, 12 in 2024-05-23T00:08:16.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:16 smithi105 ceph-mon[36426]: pgmap v240: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:08:16.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:16 smithi103 ceph-mon[31370]: pgmap v240: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:08:16.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:16 smithi174 ceph-mon[36179]: pgmap v240: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:08:18.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:18 smithi105 ceph-mon[36426]: pgmap v241: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2024-05-23T00:08:18.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:18 smithi103 ceph-mon[31370]: pgmap v241: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2024-05-23T00:08:18.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:18 smithi174 ceph-mon[36179]: pgmap v241: 129 pgs: 129 active+clean; 84 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 207 B/s wr, 6 op/s 2024-05-23T00:08:20.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:20 smithi105 ceph-mon[36426]: pgmap v242: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:20.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:20 smithi103 ceph-mon[31370]: pgmap v242: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:20.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:20 smithi174 ceph-mon[36179]: pgmap v242: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:22.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:22 smithi103 ceph-mon[31370]: pgmap v243: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:22.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:22 smithi174 ceph-mon[36179]: pgmap v243: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:22.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:22 smithi105 ceph-mon[36426]: pgmap v243: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:24.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:24 smithi103 ceph-mon[31370]: pgmap v244: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:24.359 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:24 smithi105 ceph-mon[36426]: pgmap v244: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:24.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:24 smithi174 ceph-mon[36179]: pgmap v244: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:08:26.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:26 smithi103 ceph-mon[31370]: pgmap v245: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:08:26.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:26 smithi174 ceph-mon[36179]: pgmap v245: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:08:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:26 smithi105 ceph-mon[36426]: pgmap v245: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:08:28.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:28 smithi103 ceph-mon[31370]: pgmap v246: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:28.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:28 smithi174 ceph-mon[36179]: pgmap v246: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:28.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:28 smithi105 ceph-mon[36426]: pgmap v246: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:30.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:30 smithi103 ceph-mon[31370]: pgmap v247: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:30.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:30 smithi174 ceph-mon[36179]: pgmap v247: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:30.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:30 smithi105 ceph-mon[36426]: pgmap v247: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:32.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:32 smithi174 ceph-mon[36179]: pgmap v248: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:32.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:32 smithi105 ceph-mon[36426]: pgmap v248: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:32.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:32 smithi103 ceph-mon[31370]: pgmap v248: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:34.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:34 smithi174 ceph-mon[36179]: pgmap v249: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:34.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:34 smithi105 ceph-mon[36426]: pgmap v249: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:34.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:34 smithi103 ceph-mon[31370]: pgmap v249: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:36.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:36 smithi174 ceph-mon[36179]: pgmap v250: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:36.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:36 smithi105 ceph-mon[36426]: pgmap v250: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:36.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:36 smithi103 ceph-mon[31370]: pgmap v250: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:38.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:38 smithi174 ceph-mon[36179]: pgmap v251: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:38.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:38 smithi105 ceph-mon[36426]: pgmap v251: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:38.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:38 smithi103 ceph-mon[31370]: pgmap v251: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:40.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:40 smithi174 ceph-mon[36179]: pgmap v252: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:08:40.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:40 smithi105 ceph-mon[36426]: pgmap v252: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:08:40.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:40 smithi103 ceph-mon[31370]: pgmap v252: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-23T00:08:42.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:42 smithi174 ceph-mon[36179]: pgmap v253: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:42.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:42 smithi105 ceph-mon[36426]: pgmap v253: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:42.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:42 smithi103 ceph-mon[31370]: pgmap v253: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:44.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:44 smithi174 ceph-mon[36179]: pgmap v254: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:44.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:44 smithi105 ceph-mon[36426]: pgmap v254: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:44.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:44 smithi103 ceph-mon[31370]: pgmap v254: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:46.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:46 smithi105 ceph-mon[36426]: pgmap v255: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:46.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:46 smithi103 ceph-mon[31370]: pgmap v255: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:46.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:46 smithi174 ceph-mon[36179]: pgmap v255: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:48.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:48 smithi105 ceph-mon[36426]: pgmap v256: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:48.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:48 smithi103 ceph-mon[31370]: pgmap v256: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:48.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:48 smithi174 ceph-mon[36179]: pgmap v256: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:50.419 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:50 smithi174 ceph-mon[36179]: pgmap v257: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:50.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:50 smithi105 ceph-mon[36426]: pgmap v257: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:50.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:50 smithi103 ceph-mon[31370]: pgmap v257: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:08:52.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:52 smithi105 ceph-mon[36426]: pgmap v258: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:52.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:52 smithi103 ceph-mon[31370]: pgmap v258: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:52.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:52 smithi174 ceph-mon[36179]: pgmap v258: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:08:54.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:54 smithi105 ceph-mon[36426]: pgmap v259: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:54.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:54 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:08:54.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:54 smithi103 ceph-mon[31370]: pgmap v259: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:54.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:54 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:08:54.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:54 smithi174 ceph-mon[36179]: pgmap v259: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:54.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:54 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:08:55.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:55 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:08:55.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:55 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:08:55.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:55 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:08:55.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:55 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:08:55.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:55 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:08:55.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:55 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:08:55.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:55 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:08:55.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:55 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:08:55.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:55 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:08:56.480 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:56 smithi103 ceph-mon[31370]: pgmap v260: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:56.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:56 smithi105 ceph-mon[36426]: pgmap v260: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:56.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:56 smithi174 ceph-mon[36179]: pgmap v260: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:08:58.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:08:58 smithi105 ceph-mon[36426]: pgmap v261: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:58.584 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:08:58 smithi103 ceph-mon[31370]: pgmap v261: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:08:58.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:08:58 smithi174 ceph-mon[36179]: pgmap v261: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:00.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:00 smithi105 ceph-mon[36426]: pgmap v262: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:00.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:00 smithi103 ceph-mon[31370]: pgmap v262: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:00.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:00 smithi174 ceph-mon[36179]: pgmap v262: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:02.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:02 smithi105 ceph-mon[36426]: pgmap v263: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:02.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:02 smithi103 ceph-mon[31370]: pgmap v263: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:02.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:02 smithi174 ceph-mon[36179]: pgmap v263: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:04.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:04 smithi105 ceph-mon[36426]: pgmap v264: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:04.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:04 smithi103 ceph-mon[31370]: pgmap v264: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:04.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:04 smithi174 ceph-mon[36179]: pgmap v264: 129 pgs: 129 active+clean; 84 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:06.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:06 smithi105 ceph-mon[36426]: pgmap v265: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:09:06.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:06 smithi103 ceph-mon[31370]: pgmap v265: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:09:06.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:06 smithi174 ceph-mon[36179]: pgmap v265: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:09:08.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:08 smithi105 ceph-mon[36426]: pgmap v266: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:09:08.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:08 smithi105 ceph-mon[36426]: osdmap e101: 12 total, 12 up, 12 in 2024-05-23T00:09:08.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:08 smithi103 ceph-mon[31370]: pgmap v266: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:09:08.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:08 smithi103 ceph-mon[31370]: osdmap e101: 12 total, 12 up, 12 in 2024-05-23T00:09:08.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:08 smithi174 ceph-mon[36179]: pgmap v266: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:09:08.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:08 smithi174 ceph-mon[36179]: osdmap e101: 12 total, 12 up, 12 in 2024-05-23T00:09:10.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:10 smithi105 ceph-mon[36426]: pgmap v268: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:10.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:10 smithi103 ceph-mon[31370]: pgmap v268: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:10.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:10 smithi174 ceph-mon[36179]: pgmap v268: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:12.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:12 smithi105 ceph-mon[36426]: pgmap v269: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:12.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:12 smithi103 ceph-mon[31370]: pgmap v269: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:12.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:12 smithi174 ceph-mon[36179]: pgmap v269: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:14.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:14 smithi105 ceph-mon[36426]: pgmap v270: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:14.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:14 smithi105 ceph-mon[36426]: osdmap e102: 12 total, 12 up, 12 in 2024-05-23T00:09:14.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:14 smithi103 ceph-mon[31370]: pgmap v270: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:14.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:14 smithi103 ceph-mon[31370]: osdmap e102: 12 total, 12 up, 12 in 2024-05-23T00:09:14.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:14 smithi174 ceph-mon[36179]: pgmap v270: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:09:14.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:14 smithi174 ceph-mon[36179]: osdmap e102: 12 total, 12 up, 12 in 2024-05-23T00:09:16.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:16 smithi105 ceph-mon[36426]: pgmap v272: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:09:16.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:16 smithi103 ceph-mon[31370]: pgmap v272: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:09:16.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:16 smithi174 ceph-mon[36179]: pgmap v272: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:09:18.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:18 smithi105 ceph-mon[36426]: pgmap v273: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2024-05-23T00:09:18.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:18 smithi103 ceph-mon[31370]: pgmap v273: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2024-05-23T00:09:18.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:18 smithi174 ceph-mon[36179]: pgmap v273: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 209 B/s wr, 6 op/s 2024-05-23T00:09:20.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:20 smithi105 ceph-mon[36426]: pgmap v274: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:20.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:20 smithi103 ceph-mon[31370]: pgmap v274: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:20.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:20 smithi174 ceph-mon[36179]: pgmap v274: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:22.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:22 smithi105 ceph-mon[36426]: pgmap v275: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:22.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:22 smithi103 ceph-mon[31370]: pgmap v275: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:22.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:22 smithi174 ceph-mon[36179]: pgmap v275: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:24.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:24 smithi105 ceph-mon[36426]: pgmap v276: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:24.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:24 smithi103 ceph-mon[31370]: pgmap v276: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:24.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:24 smithi174 ceph-mon[36179]: pgmap v276: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:09:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:26 smithi105 ceph-mon[36426]: pgmap v277: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:09:26.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:26 smithi103 ceph-mon[31370]: pgmap v277: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:09:26.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:26 smithi174 ceph-mon[36179]: pgmap v277: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:09:28.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:28 smithi105 ceph-mon[36426]: pgmap v278: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:28.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:28 smithi103 ceph-mon[31370]: pgmap v278: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:28.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:28 smithi174 ceph-mon[36179]: pgmap v278: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:30.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:30 smithi105 ceph-mon[36426]: pgmap v279: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:30.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:30 smithi103 ceph-mon[31370]: pgmap v279: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:30.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:30 smithi174 ceph-mon[36179]: pgmap v279: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:32.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:32 smithi105 ceph-mon[36426]: pgmap v280: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:32.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:32 smithi103 ceph-mon[31370]: pgmap v280: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:32.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:32 smithi174 ceph-mon[36179]: pgmap v280: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:34.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:34 smithi105 ceph-mon[36426]: pgmap v281: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:34.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:34 smithi103 ceph-mon[31370]: pgmap v281: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:34.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:34 smithi174 ceph-mon[36179]: pgmap v281: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:36.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:36 smithi105 ceph-mon[36426]: pgmap v282: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:36.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:36 smithi103 ceph-mon[31370]: pgmap v282: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:36.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:36 smithi174 ceph-mon[36179]: pgmap v282: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:38.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:38 smithi105 ceph-mon[36426]: pgmap v283: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:09:38.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:38 smithi103 ceph-mon[31370]: pgmap v283: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:09:38.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:38 smithi174 ceph-mon[36179]: pgmap v283: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:09:40.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:40 smithi105 ceph-mon[36426]: pgmap v284: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-23T00:09:40.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:40 smithi103 ceph-mon[31370]: pgmap v284: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-23T00:09:40.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:40 smithi174 ceph-mon[36179]: pgmap v284: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-23T00:09:42.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:42 smithi105 ceph-mon[36426]: pgmap v285: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-23T00:09:42.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:42 smithi103 ceph-mon[31370]: pgmap v285: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-23T00:09:42.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:42 smithi174 ceph-mon[36179]: pgmap v285: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-23T00:09:44.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:44 smithi105 ceph-mon[36426]: pgmap v286: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:09:44.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:44 smithi103 ceph-mon[31370]: pgmap v286: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:09:44.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:44 smithi174 ceph-mon[36179]: pgmap v286: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-23T00:09:46.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:46 smithi105 ceph-mon[36426]: pgmap v287: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:46.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:46 smithi103 ceph-mon[31370]: pgmap v287: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:46.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:46 smithi174 ceph-mon[36179]: pgmap v287: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:48.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:48 smithi105 ceph-mon[36426]: pgmap v288: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:48.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:48 smithi103 ceph-mon[31370]: pgmap v288: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:48.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:48 smithi174 ceph-mon[36179]: pgmap v288: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:50.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:50 smithi105 ceph-mon[36426]: pgmap v289: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:50.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:50 smithi103 ceph-mon[31370]: pgmap v289: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:50.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:50 smithi174 ceph-mon[36179]: pgmap v289: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:09:52.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:52 smithi105 ceph-mon[36426]: pgmap v290: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:52.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:52 smithi103 ceph-mon[31370]: pgmap v290: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:52.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:52 smithi174 ceph-mon[36179]: pgmap v290: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:09:54.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:54 smithi105 ceph-mon[36426]: pgmap v291: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:54.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:54 smithi103 ceph-mon[31370]: pgmap v291: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:54.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:54 smithi174 ceph-mon[36179]: pgmap v291: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:55.578 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:55 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:09:55.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:55 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:09:55.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:55 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:09:56.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:56 smithi103 ceph-mon[31370]: pgmap v292: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:56.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:56 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:09:56.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:56 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:09:56.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:56 smithi174 ceph-mon[36179]: pgmap v292: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:56.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:56 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:09:56.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:56 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:09:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:56 smithi105 ceph-mon[36426]: pgmap v292: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:09:56.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:56 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:09:56.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:56 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:09:57.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:57 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:09:57.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:57 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:09:57.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:57 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:09:58.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:09:58 smithi174 ceph-mon[36179]: pgmap v293: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:09:58 smithi105 ceph-mon[36426]: pgmap v293: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:09:58.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:09:58 smithi103 ceph-mon[31370]: pgmap v293: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:00.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[31366]: 2024-05-23T00:09:59.999+0000 7fe445c00700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-05-23T00:10:00.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[31366]: 2024-05-23T00:09:59.999+0000 7fe445c00700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-05-23T00:10:00.329 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[31366]: 2024-05-23T00:09:59.999+0000 7fe445c00700 -1 log_channel(cluster) log [ERR] : mds.i(mds.2): Metadata damage detected 2024-05-23T00:10:00.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:00 smithi174 ceph-mon[36179]: pgmap v294: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:00.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:00 smithi174 ceph-mon[36179]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-05-23T00:10:00.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:00 smithi174 ceph-mon[36179]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-05-23T00:10:00.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:00 smithi174 ceph-mon[36179]: mds.i(mds.2): Metadata damage detected 2024-05-23T00:10:00.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:00 smithi105 ceph-mon[36426]: pgmap v294: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:00.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:00 smithi105 ceph-mon[36426]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-05-23T00:10:00.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:00 smithi105 ceph-mon[36426]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-05-23T00:10:00.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:00 smithi105 ceph-mon[36426]: mds.i(mds.2): Metadata damage detected 2024-05-23T00:10:00.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:00 smithi103 ceph-mon[31370]: pgmap v294: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:00.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:00 smithi103 ceph-mon[31370]: Health detail: HEALTH_ERR 1 MDSs report damaged metadata 2024-05-23T00:10:00.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:00 smithi103 ceph-mon[31370]: [ERR] MDS_DAMAGE: 1 MDSs report damaged metadata 2024-05-23T00:10:00.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:00 smithi103 ceph-mon[31370]: mds.i(mds.2): Metadata damage detected 2024-05-23T00:10:02.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:02 smithi174 ceph-mon[36179]: pgmap v295: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:02.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:02 smithi105 ceph-mon[36426]: pgmap v295: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:02.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:02 smithi103 ceph-mon[31370]: pgmap v295: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:04.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:04 smithi174 ceph-mon[36179]: pgmap v296: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:04.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:04 smithi105 ceph-mon[36426]: pgmap v296: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:04.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:04 smithi103 ceph-mon[31370]: pgmap v296: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:06.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:06 smithi174 ceph-mon[36179]: pgmap v297: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:10:06.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:06 smithi105 ceph-mon[36426]: pgmap v297: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:10:06.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:06 smithi103 ceph-mon[31370]: pgmap v297: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:10:08.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:08 smithi105 ceph-mon[36426]: pgmap v298: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:10:08.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:08 smithi105 ceph-mon[36426]: osdmap e103: 12 total, 12 up, 12 in 2024-05-23T00:10:08.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:08 smithi103 ceph-mon[31370]: pgmap v298: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:10:08.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:08 smithi103 ceph-mon[31370]: osdmap e103: 12 total, 12 up, 12 in 2024-05-23T00:10:08.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:08 smithi174 ceph-mon[36179]: pgmap v298: 129 pgs: 129 active+clean; 84 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:10:08.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:08 smithi174 ceph-mon[36179]: osdmap e103: 12 total, 12 up, 12 in 2024-05-23T00:10:10.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:10 smithi105 ceph-mon[36426]: pgmap v300: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:10.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:10 smithi103 ceph-mon[31370]: pgmap v300: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:10.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:10 smithi174 ceph-mon[36179]: pgmap v300: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:12.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:12 smithi105 ceph-mon[36426]: pgmap v301: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:12.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:12 smithi103 ceph-mon[31370]: pgmap v301: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:12.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:12 smithi174 ceph-mon[36179]: pgmap v301: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:14 smithi105 ceph-mon[36426]: pgmap v302: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:14 smithi105 ceph-mon[36426]: osdmap e104: 12 total, 12 up, 12 in 2024-05-23T00:10:14.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:14 smithi103 ceph-mon[31370]: pgmap v302: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:14.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:14 smithi103 ceph-mon[31370]: osdmap e104: 12 total, 12 up, 12 in 2024-05-23T00:10:14.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:14 smithi174 ceph-mon[36179]: pgmap v302: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:10:14.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:14 smithi174 ceph-mon[36179]: osdmap e104: 12 total, 12 up, 12 in 2024-05-23T00:10:16.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:16 smithi105 ceph-mon[36426]: pgmap v304: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:10:16.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:16 smithi103 ceph-mon[31370]: pgmap v304: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:10:16.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:16 smithi174 ceph-mon[36179]: pgmap v304: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:10:18.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:18 smithi105 ceph-mon[36426]: pgmap v305: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2024-05-23T00:10:18.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:18 smithi103 ceph-mon[31370]: pgmap v305: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2024-05-23T00:10:18.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:18 smithi174 ceph-mon[36179]: pgmap v305: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2024-05-23T00:10:20.669 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:20 smithi174 ceph-mon[36179]: pgmap v306: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:20.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:20 smithi105 ceph-mon[36426]: pgmap v306: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:20.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:20 smithi103 ceph-mon[31370]: pgmap v306: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:22.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:22 smithi105 ceph-mon[36426]: pgmap v307: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:22.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:22 smithi103 ceph-mon[31370]: pgmap v307: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:22.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:22 smithi174 ceph-mon[36179]: pgmap v307: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:24.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:24 smithi105 ceph-mon[36426]: pgmap v308: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:24.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:24 smithi103 ceph-mon[31370]: pgmap v308: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:24.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:24 smithi174 ceph-mon[36179]: pgmap v308: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:10:26.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:26 smithi105 ceph-mon[36426]: pgmap v309: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:10:26.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:26 smithi103 ceph-mon[31370]: pgmap v309: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:10:26.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:26 smithi174 ceph-mon[36179]: pgmap v309: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:10:28.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:28 smithi105 ceph-mon[36426]: pgmap v310: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:28.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:28 smithi103 ceph-mon[31370]: pgmap v310: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:28.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:28 smithi174 ceph-mon[36179]: pgmap v310: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:30.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:30 smithi105 ceph-mon[36426]: pgmap v311: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:30.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:30 smithi103 ceph-mon[31370]: pgmap v311: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:30.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:30 smithi174 ceph-mon[36179]: pgmap v311: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:32.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:32 smithi174 ceph-mon[36179]: pgmap v312: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:32.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:32 smithi105 ceph-mon[36426]: pgmap v312: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:32.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:32 smithi103 ceph-mon[31370]: pgmap v312: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:34.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:34 smithi105 ceph-mon[36426]: pgmap v313: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:34.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:34 smithi103 ceph-mon[31370]: pgmap v313: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:34.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:34 smithi174 ceph-mon[36179]: pgmap v313: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:36.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:36 smithi105 ceph-mon[36426]: pgmap v314: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:10:36.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:36 smithi103 ceph-mon[31370]: pgmap v314: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:10:36.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:36 smithi174 ceph-mon[36179]: pgmap v314: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:10:38.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:38 smithi105 ceph-mon[36426]: pgmap v315: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:38.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:38 smithi103 ceph-mon[31370]: pgmap v315: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:38.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:38 smithi174 ceph-mon[36179]: pgmap v315: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:40.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:40 smithi105 ceph-mon[36426]: pgmap v316: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:40.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:40 smithi103 ceph-mon[31370]: pgmap v316: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:40.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:40 smithi174 ceph-mon[36179]: pgmap v316: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-23T00:10:42.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:42 smithi105 ceph-mon[36426]: pgmap v317: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:42.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:42 smithi103 ceph-mon[31370]: pgmap v317: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:42.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:42 smithi174 ceph-mon[36179]: pgmap v317: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:10:44.567 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-05-23T00:10:44.767 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.764+0000 7f886b200640 1 -- 172.21.15.103:0/3855571733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88641080d0 msgr2=0x7f88641084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:44.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.764+0000 7f886b200640 1 --2- 172.21.15.103:0/3855571733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88641080d0 0x7f88641084b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8854009900 tx=0x7f885402a410 comp rx=0 tx=0).stop 2024-05-23T00:10:44.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.764+0000 7f886b200640 1 -- 172.21.15.103:0/3855571733 shutdown_connections 2024-05-23T00:10:44.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.764+0000 7f886b200640 1 --2- 172.21.15.103:0/3855571733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f886410d1a0 0x7f8864114d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:44.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.765+0000 7f886b200640 1 --2- 172.21.15.103:0/3855571733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8864108a80 0x7f886410cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:44.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.765+0000 7f886b200640 1 --2- 172.21.15.103:0/3855571733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f88641080d0 0x7f88641084b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:44.768 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.765+0000 7f886b200640 1 -- 172.21.15.103:0/3855571733 >> 172.21.15.103:0/3855571733 conn(0x7f8864075e00 msgr2=0x7f8864078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:44.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.765+0000 7f886b200640 1 -- 172.21.15.103:0/3855571733 shutdown_connections 2024-05-23T00:10:44.769 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.765+0000 7f886b200640 1 -- 172.21.15.103:0/3855571733 wait complete. 2024-05-23T00:10:44.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.766+0000 7f886b200640 1 Processor -- start 2024-05-23T00:10:44.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.766+0000 7f886b200640 1 -- start start 2024-05-23T00:10:44.771 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f886b200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88641080d0 0x7f886419e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:44.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f886b200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f886410d1a0 0x7f886419ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:44.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f886b200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88641a2fc0 0x7f88641a3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:44.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f886b200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8864118250 con 0x7f88641a2fc0 2024-05-23T00:10:44.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f886b200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f88641180d0 con 0x7f886410d1a0 2024-05-23T00:10:44.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f886b200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88641183d0 con 0x7f88641080d0 2024-05-23T00:10:44.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f8869200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88641a2fc0 0x7f88641a3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:44.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f885be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f886410d1a0 0x7f886419ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:44.772 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f8863e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88641080d0 0x7f886419e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f885be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f886410d1a0 0x7f886419ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:37466/0 (socket says 172.21.15.103:37466) 2024-05-23T00:10:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f885be00640 1 -- 172.21.15.103:0/580444147 learned_addr learned my addr 172.21.15.103:0/580444147 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:10:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f885be00640 1 -- 172.21.15.103:0/580444147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88641080d0 msgr2=0x7f886419e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f885be00640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88641080d0 0x7f886419e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f885be00640 1 -- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88641a2fc0 msgr2=0x7f88641a3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f885be00640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88641a2fc0 0x7f88641a3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f885be00640 1 -- 172.21.15.103:0/580444147 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88641a3ab0 con 0x7f886410d1a0 2024-05-23T00:10:44.773 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.767+0000 7f8869200640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88641a2fc0 0x7f88641a3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:10:44.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.768+0000 7f885be00640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f886410d1a0 0x7f886419ec90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f884c00e9f0 tx=0x7f884c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:44.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.768+0000 7f8861600640 1 -- 172.21.15.103:0/580444147 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c00ccc0 con 0x7f886410d1a0 2024-05-23T00:10:44.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.768+0000 7f8861600640 1 -- 172.21.15.103:0/580444147 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f884c005500 con 0x7f886410d1a0 2024-05-23T00:10:44.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.768+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88641a3da0 con 0x7f886410d1a0 2024-05-23T00:10:44.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.768+0000 7f8861600640 1 -- 172.21.15.103:0/580444147 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c010690 con 0x7f886410d1a0 2024-05-23T00:10:44.774 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.768+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88641ab540 con 0x7f886410d1a0 2024-05-23T00:10:44.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.769+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886404f630 con 0x7f886410d1a0 2024-05-23T00:10:44.775 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.771+0000 7f8861600640 1 -- 172.21.15.103:0/580444147 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f884c0046c0 con 0x7f886410d1a0 2024-05-23T00:10:44.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.773+0000 7f8861600640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f88480a2da0 0x7f88480a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:44.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.773+0000 7f8861600640 1 -- 172.21.15.103:0/580444147 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9612+0+0 (secure 0 0 0) 0x7f884c0cfc50 con 0x7f886410d1a0 2024-05-23T00:10:44.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.773+0000 7f8863e00640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f88480a2da0 0x7f88480a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:44.776 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.774+0000 7f8863e00640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f88480a2da0 0x7f88480a5260 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8854009230 tx=0x7f88540058d0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:44.778 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.775+0000 7f8861600640 1 -- 172.21.15.103:0/580444147 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f884c09c3d0 con 0x7f886410d1a0 2024-05-23T00:10:44.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:44 smithi105 ceph-mon[36426]: pgmap v318: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:44.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:44 smithi103 ceph-mon[31370]: pgmap v318: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:44.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:44 smithi174 ceph-mon[36179]: pgmap v318: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:45.000 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:44.997+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f8864108a80 con 0x7f88480a2da0 2024-05-23T00:10:45.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.012+0000 7f8861600640 1 -- 172.21.15.103:0/580444147 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f8864108a80 con 0x7f88480a2da0 2024-05-23T00:10:45.015 INFO:teuthology.orchestra.run.smithi103.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-05-23T00:00:00", "created": "2024-05-23T00:00:42", "first": "2024-05-23T00:01:00", "last": "2024-05-23T00:10:00", "last_pruned": "2024-05-23T00:10:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-05-23T00:10:45.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.014+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f88480a2da0 msgr2=0x7f88480a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.014+0000 7f886b200640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f88480a2da0 0x7f88480a5260 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8854009230 tx=0x7f88540058d0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f886410d1a0 msgr2=0x7f886419ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.018 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f886410d1a0 0x7f886419ec90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f884c00e9f0 tx=0x7f884c00edf0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 shutdown_connections 2024-05-23T00:10:45.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f88480a2da0 0x7f88480a5260 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f88641a2fc0 0x7f88641a3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f886410d1a0 0x7f886419ec90 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 --2- 172.21.15.103:0/580444147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88641080d0 0x7f886419e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 >> 172.21.15.103:0/580444147 conn(0x7f8864075e00 msgr2=0x7f8864078180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:45.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 shutdown_connections 2024-05-23T00:10:45.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.015+0000 7f886b200640 1 -- 172.21.15.103:0/580444147 wait complete. 2024-05-23T00:10:45.037 DEBUG:teuthology.orchestra.run.smithi103:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-05-23T00:10:45.234 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.231+0000 7fc573e00640 1 -- 172.21.15.103:0/1810953319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574109890 msgr2=0x7fc574109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.231+0000 7fc573e00640 1 --2- 172.21.15.103:0/1810953319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574109890 0x7fc574109c70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fc564009900 tx=0x7fc56402a3f0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.231+0000 7fc573e00640 1 -- 172.21.15.103:0/1810953319 shutdown_connections 2024-05-23T00:10:45.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.231+0000 7fc573e00640 1 --2- 172.21.15.103:0/1810953319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc574104240 0x7fc574110750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.231+0000 7fc573e00640 1 --2- 172.21.15.103:0/1810953319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc574103880 0x7fc574103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.231+0000 7fc573e00640 1 --2- 172.21.15.103:0/1810953319 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574109890 0x7fc574109c70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.231+0000 7fc573e00640 1 -- 172.21.15.103:0/1810953319 >> 172.21.15.103:0/1810953319 conn(0x7fc5740ff390 msgr2=0x7fc5741017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:45.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.232+0000 7fc573e00640 1 -- 172.21.15.103:0/1810953319 shutdown_connections 2024-05-23T00:10:45.235 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.232+0000 7fc573e00640 1 -- 172.21.15.103:0/1810953319 wait complete. 2024-05-23T00:10:45.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.232+0000 7fc573e00640 1 Processor -- start 2024-05-23T00:10:45.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc573e00640 1 -- start start 2024-05-23T00:10:45.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc573e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc574103880 0x7fc57419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:45.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc573e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574104240 0x7fc57419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:45.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc573e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc574109890 0x7fc57419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:45.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc573e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc574113cb0 con 0x7fc574103880 2024-05-23T00:10:45.238 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc573e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc574113b30 con 0x7fc574104240 2024-05-23T00:10:45.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc573e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc574113e30 con 0x7fc574109890 2024-05-23T00:10:45.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc572a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc574103880 0x7fc57419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:45.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc572000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574104240 0x7fc57419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:45.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc572000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574104240 0x7fc57419a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:37488/0 (socket says 172.21.15.103:37488) 2024-05-23T00:10:45.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc572000640 1 -- 172.21.15.103:0/1753085010 learned_addr learned my addr 172.21.15.103:0/1753085010 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:10:45.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc572a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc574103880 0x7fc57419a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33892/0 (socket says 172.21.15.103:33892) 2024-05-23T00:10:45.239 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.233+0000 7fc573400640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc574109890 0x7fc57419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc573400640 1 -- 172.21.15.103:0/1753085010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574104240 msgr2=0x7fc57419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc573400640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574104240 0x7fc57419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc573400640 1 -- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc574103880 msgr2=0x7fc57419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc573400640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc574103880 0x7fc57419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc573400640 1 -- 172.21.15.103:0/1753085010 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc57419f1d0 con 0x7fc574109890 2024-05-23T00:10:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc572a00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc574103880 0x7fc57419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:10:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc572000640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574104240 0x7fc57419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:10:45.240 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc573400640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc574109890 0x7fc57419eaa0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009fd0 tx=0x7fc56c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.234+0000 7fc56b400640 1 -- 172.21.15.103:0/1753085010 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c00ce20 con 0x7fc574109890 2024-05-23T00:10:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.235+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc57419f4c0 con 0x7fc574109890 2024-05-23T00:10:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.235+0000 7fc56b400640 1 -- 172.21.15.103:0/1753085010 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc56c010430 con 0x7fc574109890 2024-05-23T00:10:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.235+0000 7fc56b400640 1 -- 172.21.15.103:0/1753085010 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56c00f620 con 0x7fc574109890 2024-05-23T00:10:45.241 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.235+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5741a6d50 con 0x7fc574109890 2024-05-23T00:10:45.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.236+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc57404fe80 con 0x7fc574109890 2024-05-23T00:10:45.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.238+0000 7fc56b400640 1 -- 172.21.15.103:0/1753085010 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fc56c010950 con 0x7fc574109890 2024-05-23T00:10:45.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.239+0000 7fc56b400640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc5580a2cf0 0x7fc5580a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:45.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.239+0000 7fc56b400640 1 -- 172.21.15.103:0/1753085010 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9612+0+0 (secure 0 0 0) 0x7fc56c0cfcf0 con 0x7fc574109890 2024-05-23T00:10:45.242 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.240+0000 7fc572a00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc5580a2cf0 0x7fc5580a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:45.243 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.240+0000 7fc572a00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc5580a2cf0 0x7fc5580a51b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc564000c00 tx=0x7fc564005b50 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:45.244 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.242+0000 7fc56b400640 1 -- 172.21.15.103:0/1753085010 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc56c09c470 con 0x7fc574109890 2024-05-23T00:10:45.473 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.470+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 --> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7fc57410be50 con 0x7fc5580a2cf0 2024-05-23T00:10:45.487 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.484+0000 7fc56b400640 1 -- 172.21.15.103:0/1753085010 <== mgr.25390 v2:172.21.15.103:6800/1519813025 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7fc57410be50 con 0x7fc5580a2cf0 2024-05-23T00:10:45.488 INFO:teuthology.orchestra.run.smithi103.stdout:/ 1m 6m3h 2024-05-23T00:10:45.490 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.487+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc5580a2cf0 msgr2=0x7fc5580a51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.487+0000 7fc573e00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc5580a2cf0 0x7fc5580a51b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc564000c00 tx=0x7fc564005b50 comp rx=0 tx=0).stop 2024-05-23T00:10:45.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.487+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc574109890 msgr2=0x7fc57419eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.487+0000 7fc573e00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc574109890 0x7fc57419eaa0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009fd0 tx=0x7fc56c00ef10 comp rx=0 tx=0).stop 2024-05-23T00:10:45.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.488+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 shutdown_connections 2024-05-23T00:10:45.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.488+0000 7fc573e00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc5580a2cf0 0x7fc5580a51b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.488+0000 7fc573e00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc574109890 0x7fc57419eaa0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.491 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.488+0000 7fc573e00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc574104240 0x7fc57419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.488+0000 7fc573e00640 1 --2- 172.21.15.103:0/1753085010 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc574103880 0x7fc57419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.488+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 >> 172.21.15.103:0/1753085010 conn(0x7fc5740ff390 msgr2=0x7fc5740ffa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:45.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.488+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 shutdown_connections 2024-05-23T00:10:45.492 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.488+0000 7fc573e00640 1 -- 172.21.15.103:0/1753085010 wait complete. 2024-05-23T00:10:45.510 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-23T00:10:45.544 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-23T00:10:45.544 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b9e3da87b1ec7920937cb4902b7e3fd9eda86b4a/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_935217f1cb4dc722d5303f5ad364c883e9133376/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-05-23T00:10:45.555 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-23T00:10:45.583 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:10:45.791 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 -- 172.21.15.103:0/1099049364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608104220 msgr2=0x7fa608110730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 --2- 172.21.15.103:0/1099049364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608104220 0x7fa608110730 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa600009cd0 tx=0x7fa60002a8f0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 -- 172.21.15.103:0/1099049364 shutdown_connections 2024-05-23T00:10:45.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 --2- 172.21.15.103:0/1099049364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608104220 0x7fa608110730 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 --2- 172.21.15.103:0/1099049364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa608103860 0x7fa608103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 --2- 172.21.15.103:0/1099049364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa608109870 0x7fa608109c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.792 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 -- 172.21.15.103:0/1099049364 >> 172.21.15.103:0/1099049364 conn(0x7fa6080ff310 msgr2=0x7fa608101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:45.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 -- 172.21.15.103:0/1099049364 shutdown_connections 2024-05-23T00:10:45.793 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.789+0000 7fa607200640 1 -- 172.21.15.103:0/1099049364 wait complete. 2024-05-23T00:10:45.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.790+0000 7fa607200640 1 Processor -- start 2024-05-23T00:10:45.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.790+0000 7fa607200640 1 -- start start 2024-05-23T00:10:45.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.790+0000 7fa607200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa608103860 0x7fa60819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:45.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.790+0000 7fa607200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608104220 0x7fa60819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:45.795 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.790+0000 7fa607200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa608109870 0x7fa60819e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.790+0000 7fa607200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa608113c90 con 0x7fa608103860 2024-05-23T00:10:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.790+0000 7fa607200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa608113b10 con 0x7fa608109870 2024-05-23T00:10:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.790+0000 7fa607200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa608113e10 con 0x7fa608104220 2024-05-23T00:10:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa608103860 0x7fa60819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa608103860 0x7fa60819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33920/0 (socket says 172.21.15.103:33920) 2024-05-23T00:10:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 -- 172.21.15.103:0/1034674395 learned_addr learned my addr 172.21.15.103:0/1034674395 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:10:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa606800640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa608109870 0x7fa60819e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:45.796 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605400640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608104220 0x7fa60819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:45.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 -- 172.21.15.103:0/1034674395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608104220 msgr2=0x7fa60819a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608104220 0x7fa60819a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 -- 172.21.15.103:0/1034674395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa608109870 msgr2=0x7fa60819e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:45.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa608109870 0x7fa60819e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:45.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 -- 172.21.15.103:0/1034674395 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60819f100 con 0x7fa608103860 2024-05-23T00:10:45.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa605e00640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa608103860 0x7fa60819a110 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800d950 tx=0x7fa5f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:45.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.791+0000 7fa5fea00640 1 -- 172.21.15.103:0/1034674395 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8017070 con 0x7fa608103860 2024-05-23T00:10:45.797 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.792+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa60819f3f0 con 0x7fa608103860 2024-05-23T00:10:45.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.792+0000 7fa5fea00640 1 -- 172.21.15.103:0/1034674395 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5f8009c30 con 0x7fa608103860 2024-05-23T00:10:45.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.792+0000 7fa5fea00640 1 -- 172.21.15.103:0/1034674395 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8020060 con 0x7fa608103860 2024-05-23T00:10:45.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.792+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6081a6c80 con 0x7fa608103860 2024-05-23T00:10:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.793+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa608068a00 con 0x7fa608103860 2024-05-23T00:10:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.795+0000 7fa5fea00640 1 -- 172.21.15.103:0/1034674395 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fa5f80109d0 con 0x7fa608103860 2024-05-23T00:10:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.796+0000 7fa5fea00640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa5ec0a2d00 0x7fa5ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.796+0000 7fa5fea00640 1 -- 172.21.15.103:0/1034674395 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9612+0+0 (secure 0 0 0) 0x7fa5f80cf430 con 0x7fa608103860 2024-05-23T00:10:45.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.796+0000 7fa605400640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa5ec0a2d00 0x7fa5ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:45.800 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.797+0000 7fa605400640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa5ec0a2d00 0x7fa5ec0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa6080693e0 tx=0x7fa5f0005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:45.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:45.801+0000 7fa5fea00640 1 -- 172.21.15.103:0/1034674395 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa5f8009de0 con 0x7fa608103860 2024-05-23T00:10:46.006 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.004+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6081a6f80 con 0x7fa608103860 2024-05-23T00:10:46.010 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.008+0000 7fa5fea00640 1 -- 172.21.15.103:0/1034674395 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) ==== 74+0+25507 (secure 0 0 0) 0x7fa5f809b560 con 0x7fa608103860 2024-05-23T00:10:46.011 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:10:46.011 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":104,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-23T00:10:13.613282+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/284924627":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3929139004":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3827978298":"2024-05-24T00:00:32.661117+0000","172.21.15.103:6801/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/2531882208":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:10:46.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa5ec0a2d00 msgr2=0x7fa5ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa5ec0a2d00 0x7fa5ec0a51c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa6080693e0 tx=0x7fa5f0005f90 comp rx=0 tx=0).stop 2024-05-23T00:10:46.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa608103860 msgr2=0x7fa60819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa608103860 0x7fa60819a110 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa5f800d950 tx=0x7fa5f800de10 comp rx=0 tx=0).stop 2024-05-23T00:10:46.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 shutdown_connections 2024-05-23T00:10:46.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa5ec0a2d00 0x7fa5ec0a51c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa608109870 0x7fa60819e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608104220 0x7fa60819a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 --2- 172.21.15.103:0/1034674395 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa608103860 0x7fa60819a110 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 >> 172.21.15.103:0/1034674395 conn(0x7fa6080ff310 msgr2=0x7fa6080ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:46.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 shutdown_connections 2024-05-23T00:10:46.015 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.011+0000 7fa607200640 1 -- 172.21.15.103:0/1034674395 wait complete. 2024-05-23T00:10:46.034 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-23T00:10:46.035 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-23T00:10:46.247 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.244+0000 7fe385200640 1 -- 172.21.15.103:0/2389198512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe38010bd00 msgr2=0x7fe38010e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.244+0000 7fe385200640 1 --2- 172.21.15.103:0/2389198512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe38010bd00 0x7fe38010e180 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fe378009ca0 tx=0x7fe37802a950 comp rx=0 tx=0).stop 2024-05-23T00:10:46.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.245+0000 7fe385200640 1 -- 172.21.15.103:0/2389198512 shutdown_connections 2024-05-23T00:10:46.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.245+0000 7fe385200640 1 --2- 172.21.15.103:0/2389198512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe38010bd00 0x7fe38010e180 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.245+0000 7fe385200640 1 --2- 172.21.15.103:0/2389198512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe380103250 0x7fe38010b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.245+0000 7fe385200640 1 --2- 172.21.15.103:0/2389198512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe380102930 0x7fe380102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.248 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.245+0000 7fe385200640 1 -- 172.21.15.103:0/2389198512 >> 172.21.15.103:0/2389198512 conn(0x7fe3800fc0f0 msgr2=0x7fe3800fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:46.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.245+0000 7fe385200640 1 -- 172.21.15.103:0/2389198512 shutdown_connections 2024-05-23T00:10:46.249 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.245+0000 7fe385200640 1 -- 172.21.15.103:0/2389198512 wait complete. 2024-05-23T00:10:46.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.246+0000 7fe385200640 1 Processor -- start 2024-05-23T00:10:46.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.246+0000 7fe385200640 1 -- start start 2024-05-23T00:10:46.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.246+0000 7fe385200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe380102930 0x7fe38019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:46.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.246+0000 7fe385200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe380103250 0x7fe38019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:46.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.246+0000 7fe385200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe38010bd00 0x7fe38019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:46.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.246+0000 7fe385200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe380111b30 con 0x7fe38010bd00 2024-05-23T00:10:46.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.246+0000 7fe385200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fe3801119b0 con 0x7fe380103250 2024-05-23T00:10:46.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.246+0000 7fe385200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe380111cb0 con 0x7fe380102930 2024-05-23T00:10:46.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37f400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe380102930 0x7fe38019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:46.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe38010bd00 0x7fe38019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:46.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37f400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe380102930 0x7fe38019a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:40790/0 (socket says 172.21.15.103:40790) 2024-05-23T00:10:46.252 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37ea00640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe380103250 0x7fe38019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:46.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37f400640 1 -- 172.21.15.103:0/1581485318 learned_addr learned my addr 172.21.15.103:0/1581485318 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:10:46.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37fe00640 1 -- 172.21.15.103:0/1581485318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe380102930 msgr2=0x7fe38019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37fe00640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe380102930 0x7fe38019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37fe00640 1 -- 172.21.15.103:0/1581485318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe380103250 msgr2=0x7fe38019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37fe00640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe380103250 0x7fe38019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.247+0000 7fe37fe00640 1 -- 172.21.15.103:0/1581485318 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38019f220 con 0x7fe38010bd00 2024-05-23T00:10:46.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.248+0000 7fe37fe00640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe38010bd00 0x7fe38019eaf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe378005e30 tx=0x7fe378003b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:46.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.248+0000 7fe377e00640 1 -- 172.21.15.103:0/1581485318 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe378034040 con 0x7fe38010bd00 2024-05-23T00:10:46.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.248+0000 7fe377e00640 1 -- 172.21.15.103:0/1581485318 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe37802f940 con 0x7fe38010bd00 2024-05-23T00:10:46.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.248+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe38019f4b0 con 0x7fe38010bd00 2024-05-23T00:10:46.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.248+0000 7fe377e00640 1 -- 172.21.15.103:0/1581485318 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3780045e0 con 0x7fe38010bd00 2024-05-23T00:10:46.260 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.248+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3801a6da0 con 0x7fe38010bd00 2024-05-23T00:10:46.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.249+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3800685b0 con 0x7fe38010bd00 2024-05-23T00:10:46.262 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.251+0000 7fe377e00640 1 -- 172.21.15.103:0/1581485318 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fe378037020 con 0x7fe38010bd00 2024-05-23T00:10:46.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.253+0000 7fe377e00640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe3640a2d80 0x7fe3640a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:46.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.254+0000 7fe377e00640 1 -- 172.21.15.103:0/1581485318 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9612+0+0 (secure 0 0 0) 0x7fe3780f03c0 con 0x7fe38010bd00 2024-05-23T00:10:46.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.254+0000 7fe37f400640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe3640a2d80 0x7fe3640a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:46.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.254+0000 7fe37f400640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe3640a2d80 0x7fe3640a5240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe370009c90 tx=0x7fe370005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:46.263 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.258+0000 7fe377e00640 1 -- 172.21.15.103:0/1581485318 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fe3780bcc50 con 0x7fe38010bd00 2024-05-23T00:10:46.465 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.463+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe3801a6f70 con 0x7fe38010bd00 2024-05-23T00:10:46.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.464+0000 7fe377e00640 1 -- 172.21.15.103:0/1581485318 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) ==== 93+0+10 (secure 0 0 0) 0x7fe3780bc550 con 0x7fe38010bd00 2024-05-23T00:10:46.467 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-23T00:10:46.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe3640a2d80 msgr2=0x7fe3640a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe3640a2d80 0x7fe3640a5240 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe370009c90 tx=0x7fe370005f90 comp rx=0 tx=0).stop 2024-05-23T00:10:46.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe38010bd00 msgr2=0x7fe38019eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe38010bd00 0x7fe38019eaf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fe378005e30 tx=0x7fe378003b80 comp rx=0 tx=0).stop 2024-05-23T00:10:46.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 shutdown_connections 2024-05-23T00:10:46.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fe3640a2d80 0x7fe3640a5240 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe38010bd00 0x7fe38019eaf0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe380103250 0x7fe38019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 --2- 172.21.15.103:0/1581485318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe380102930 0x7fe38019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 >> 172.21.15.103:0/1581485318 conn(0x7fe3800fc0f0 msgr2=0x7fe3801004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:46.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 shutdown_connections 2024-05-23T00:10:46.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.467+0000 7fe385200640 1 -- 172.21.15.103:0/1581485318 wait complete. 2024-05-23T00:10:46.489 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-23T00:10:46.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3728781525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ff0109850 msgr2=0x7f0ff0109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.697 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3728781525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ff0109850 0x7f0ff0109c30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009900 tx=0x7f0fe002a3f0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3728781525 shutdown_connections 2024-05-23T00:10:46.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3728781525 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ff0104200 0x7f0ff0110710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3728781525 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff0103840 0x7f0ff0103cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3728781525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ff0109850 0x7f0ff0109c30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3728781525 >> 172.21.15.103:0/3728781525 conn(0x7f0ff00ff310 msgr2=0x7f0ff0101730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:46.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3728781525 shutdown_connections 2024-05-23T00:10:46.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.694+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3728781525 wait complete. 2024-05-23T00:10:46.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.695+0000 7f0ff4a00640 1 Processor -- start 2024-05-23T00:10:46.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.695+0000 7f0ff4a00640 1 -- start start 2024-05-23T00:10:46.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0ff4a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ff0103840 0x7f0ff019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:46.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0ff4a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff0104200 0x7f0ff019a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:46.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0ff4a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ff0109850 0x7f0ff019ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:46.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0ff4a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff0113cb0 con 0x7f0ff0104200 2024-05-23T00:10:46.701 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0ff4a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff0113b30 con 0x7f0ff0103840 2024-05-23T00:10:46.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0ff4a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0ff0113e30 con 0x7f0ff0109850 2024-05-23T00:10:46.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0feea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff0104200 0x7f0ff019a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:46.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0feea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff0104200 0x7f0ff019a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:33960/0 (socket says 172.21.15.103:33960) 2024-05-23T00:10:46.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0feea00640 1 -- 172.21.15.103:0/3371744870 learned_addr learned my addr 172.21.15.103:0/3371744870 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:10:46.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0fef400640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ff0103840 0x7f0ff019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:46.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0fefe00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ff0109850 0x7f0ff019ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:46.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0feea00640 1 -- 172.21.15.103:0/3371744870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ff0109850 msgr2=0x7f0ff019ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0feea00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ff0109850 0x7f0ff019ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0feea00640 1 -- 172.21.15.103:0/3371744870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ff0103840 msgr2=0x7f0ff019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0feea00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ff0103840 0x7f0ff019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.696+0000 7f0feea00640 1 -- 172.21.15.103:0/3371744870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ff019f160 con 0x7f0ff0104200 2024-05-23T00:10:46.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.697+0000 7f0feea00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff0104200 0x7f0ff019a6b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800e9f0 tx=0x7f0fd800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:46.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.697+0000 7f0fe7e00640 1 -- 172.21.15.103:0/3371744870 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd800cd80 con 0x7f0ff0104200 2024-05-23T00:10:46.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.697+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ff019f450 con 0x7f0ff0104200 2024-05-23T00:10:46.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.697+0000 7f0fe7e00640 1 -- 172.21.15.103:0/3371744870 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0fd8010910 con 0x7f0ff0104200 2024-05-23T00:10:46.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.697+0000 7f0fe7e00640 1 -- 172.21.15.103:0/3371744870 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd80188f0 con 0x7f0ff0104200 2024-05-23T00:10:46.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.697+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ff01a6ce0 con 0x7f0ff0104200 2024-05-23T00:10:46.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.699+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ff0068a00 con 0x7f0ff0104200 2024-05-23T00:10:46.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.701+0000 7f0fe7e00640 1 -- 172.21.15.103:0/3371744870 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f0fd8010ac0 con 0x7f0ff0104200 2024-05-23T00:10:46.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.702+0000 7f0fe7e00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f0fd40a2d00 0x7f0fd40a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:46.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.702+0000 7f0fe7e00640 1 -- 172.21.15.103:0/3371744870 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9612+0+0 (secure 0 0 0) 0x7f0fd80cfca0 con 0x7f0ff0104200 2024-05-23T00:10:46.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.702+0000 7f0fef400640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f0fd40a2d00 0x7f0fd40a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:46.706 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.703+0000 7f0fef400640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f0fd40a2d00 0x7f0fd40a51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0000c00 tx=0x7f0fe0005b50 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:46.709 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.706+0000 7f0fe7e00640 1 -- 172.21.15.103:0/3371744870 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f0fd809bdd0 con 0x7f0ff0104200 2024-05-23T00:10:46.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:46 smithi105 ceph-mon[36426]: pgmap v319: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:10:46.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:46 smithi105 ceph-mon[36426]: from='client.26156 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:10:46.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:46 smithi105 ceph-mon[36426]: from='client.26164 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:10:46.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:46 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1034674395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:10:46.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:46 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1581485318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:10:46.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:46 smithi103 ceph-mon[31370]: pgmap v319: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:10:46.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:46 smithi103 ceph-mon[31370]: from='client.26156 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:10:46.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:46 smithi103 ceph-mon[31370]: from='client.26164 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:10:46.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:46 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1034674395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:10:46.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:46 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1581485318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:10:46.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:46 smithi174 ceph-mon[36179]: pgmap v319: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:10:46.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:46 smithi174 ceph-mon[36179]: from='client.26156 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:10:46.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:46 smithi174 ceph-mon[36179]: from='client.26164 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-23T00:10:46.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:46 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1034674395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:10:46.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:46 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1581485318' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:10:46.907 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.905+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0ff010b820 con 0x7f0ff0104200 2024-05-23T00:10:46.908 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.906+0000 7f0fe7e00640 1 -- 172.21.15.103:0/3371744870 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) ==== 104+0+11 (secure 0 0 0) 0x7f0fd80a00d0 con 0x7f0ff0104200 2024-05-23T00:10:46.908 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:10:46.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f0fd40a2d00 msgr2=0x7f0fd40a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f0fd40a2d00 0x7f0fd40a51c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0000c00 tx=0x7f0fe0005b50 comp rx=0 tx=0).stop 2024-05-23T00:10:46.911 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff0104200 msgr2=0x7f0ff019a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:46.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff0104200 0x7f0ff019a6b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800e9f0 tx=0x7f0fd800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 shutdown_connections 2024-05-23T00:10:46.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f0fd40a2d00 0x7f0fd40a51c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ff0109850 0x7f0ff019ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ff0104200 0x7f0ff019a6b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 --2- 172.21.15.103:0/3371744870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ff0103840 0x7f0ff019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:46.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.908+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 >> 172.21.15.103:0/3371744870 conn(0x7f0ff00ff310 msgr2=0x7f0ff00ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:46.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.909+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 shutdown_connections 2024-05-23T00:10:46.913 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:46.909+0000 7f0ff4a00640 1 -- 172.21.15.103:0/3371744870 wait complete. 2024-05-23T00:10:46.931 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-23T00:10:47.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 -- 172.21.15.103:0/1073782431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898073c10 msgr2=0x7f8898073ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 --2- 172.21.15.103:0/1073782431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898073c10 0x7f8898073ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8888009ca0 tx=0x7f888802a950 comp rx=0 tx=0).stop 2024-05-23T00:10:47.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 -- 172.21.15.103:0/1073782431 shutdown_connections 2024-05-23T00:10:47.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 --2- 172.21.15.103:0/1073782431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8898112750 0x7f8898114b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 --2- 172.21.15.103:0/1073782431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8898074530 0x7f8898112210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 --2- 172.21.15.103:0/1073782431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898073c10 0x7f8898073ff0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 -- 172.21.15.103:0/1073782431 >> 172.21.15.103:0/1073782431 conn(0x7f88980fd230 msgr2=0x7f88980ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:47.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 -- 172.21.15.103:0/1073782431 shutdown_connections 2024-05-23T00:10:47.139 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.135+0000 7f889ca00640 1 -- 172.21.15.103:0/1073782431 wait complete. 2024-05-23T00:10:47.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.136+0000 7f889ca00640 1 Processor -- start 2024-05-23T00:10:47.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.136+0000 7f889ca00640 1 -- start start 2024-05-23T00:10:47.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f889ca00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8898073c10 0x7f889819e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:47.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f889ca00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898074530 0x7f889819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:47.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f889ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8898112750 0x7f88981a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:47.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f889ca00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8898118120 con 0x7f8898112750 2024-05-23T00:10:47.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f889ca00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8898117fa0 con 0x7f8898074530 2024-05-23T00:10:47.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f889ca00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88981182a0 con 0x7f8898073c10 2024-05-23T00:10:47.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f8897400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8898073c10 0x7f889819e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:47.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f8897e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8898112750 0x7f88981a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:47.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f8897400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8898073c10 0x7f889819e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:40812/0 (socket says 172.21.15.103:40812) 2024-05-23T00:10:47.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f8897400640 1 -- 172.21.15.103:0/878409301 learned_addr learned my addr 172.21.15.103:0/878409301 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:10:47.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.137+0000 7f8896a00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898074530 0x7f889819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:47.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f8897e00640 1 -- 172.21.15.103:0/878409301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8898073c10 msgr2=0x7f889819e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f8897e00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8898073c10 0x7f889819e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f8897e00640 1 -- 172.21.15.103:0/878409301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898074530 msgr2=0x7f889819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f8897e00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898074530 0x7f889819eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f8897e00640 1 -- 172.21.15.103:0/878409301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88981a3650 con 0x7f8898112750 2024-05-23T00:10:47.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f8897e00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8898112750 0x7f88981a2f20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f889000d650 tx=0x7f889000db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:47.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f888fe00640 1 -- 172.21.15.103:0/878409301 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890017070 con 0x7f8898112750 2024-05-23T00:10:47.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f888fe00640 1 -- 172.21.15.103:0/878409301 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8890009820 con 0x7f8898112750 2024-05-23T00:10:47.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88981a3940 con 0x7f8898112750 2024-05-23T00:10:47.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f888fe00640 1 -- 172.21.15.103:0/878409301 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8890004d00 con 0x7f8898112750 2024-05-23T00:10:47.144 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.138+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88981ab1d0 con 0x7f8898112750 2024-05-23T00:10:47.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.140+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8898068a00 con 0x7f8898112750 2024-05-23T00:10:47.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.142+0000 7f888fe00640 1 -- 172.21.15.103:0/878409301 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f88900099d0 con 0x7f8898112750 2024-05-23T00:10:47.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.143+0000 7f888fe00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f887c0a2d00 0x7f887c0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:47.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.143+0000 7f888fe00640 1 -- 172.21.15.103:0/878409301 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9612+0+0 (secure 0 0 0) 0x7f88900cf150 con 0x7f8898112750 2024-05-23T00:10:47.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.143+0000 7f8897400640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f887c0a2d00 0x7f887c0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:47.147 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.144+0000 7f8897400640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f887c0a2d00 0x7f887c0a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8888009a60 tx=0x7f8888005c50 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:47.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.148+0000 7f888fe00640 1 -- 172.21.15.103:0/878409301 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f889009b280 con 0x7f8898112750 2024-05-23T00:10:47.356 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.354+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8898104ba0 con 0x7f8898112750 2024-05-23T00:10:47.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.355+0000 7f888fe00640 1 -- 172.21.15.103:0/878409301 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) ==== 100+0+11 (secure 0 0 0) 0x7f889009f580 con 0x7f8898112750 2024-05-23T00:10:47.357 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-23T00:10:47.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f887c0a2d00 msgr2=0x7f887c0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f887c0a2d00 0x7f887c0a51c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8888009a60 tx=0x7f8888005c50 comp rx=0 tx=0).stop 2024-05-23T00:10:47.360 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8898112750 msgr2=0x7f88981a2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8898112750 0x7f88981a2f20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f889000d650 tx=0x7f889000db10 comp rx=0 tx=0).stop 2024-05-23T00:10:47.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 shutdown_connections 2024-05-23T00:10:47.361 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f887c0a2d00 0x7f887c0a51c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8898112750 0x7f88981a2f20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8898074530 0x7f889819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 --2- 172.21.15.103:0/878409301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8898073c10 0x7f889819e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 >> 172.21.15.103:0/878409301 conn(0x7f88980fd230 msgr2=0x7f88980fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:47.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 shutdown_connections 2024-05-23T00:10:47.362 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.358+0000 7f889ca00640 1 -- 172.21.15.103:0/878409301 wait complete. 2024-05-23T00:10:47.381 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-23T00:10:47.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 -- 172.21.15.103:0/3806359452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 msgr2=0x7f58dc114ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 --2- 172.21.15.103:0/3806359452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 0x7f58dc114ae0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009ca0 tx=0x7f58d402a8c0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 -- 172.21.15.103:0/3806359452 shutdown_connections 2024-05-23T00:10:47.589 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 --2- 172.21.15.103:0/3806359452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 0x7f58dc114ae0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 --2- 172.21.15.103:0/3806359452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58dc0766c0 0x7f58dc1121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 --2- 172.21.15.103:0/3806359452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58dc075da0 0x7f58dc076180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 -- 172.21.15.103:0/3806359452 >> 172.21.15.103:0/3806359452 conn(0x7f58dc0ff520 msgr2=0x7f58dc101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:47.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 -- 172.21.15.103:0/3806359452 shutdown_connections 2024-05-23T00:10:47.590 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.586+0000 7f58e3800640 1 -- 172.21.15.103:0/3806359452 wait complete. 2024-05-23T00:10:47.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.587+0000 7f58e3800640 1 Processor -- start 2024-05-23T00:10:47.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.587+0000 7f58e3800640 1 -- start start 2024-05-23T00:10:47.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e3800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58dc075da0 0x7f58dc19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:47.592 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e3800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58dc0766c0 0x7f58dc19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:47.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e3800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 0x7f58dc1a2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:47.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e3800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc118280 con 0x7f58dc1126f0 2024-05-23T00:10:47.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e3800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc118100 con 0x7f58dc0766c0 2024-05-23T00:10:47.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e3800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58dc118400 con 0x7f58dc075da0 2024-05-23T00:10:47.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e0e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58dc075da0 0x7f58dc19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:47.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e1800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 0x7f58dc1a2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:47.593 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58dbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58dc0766c0 0x7f58dc19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:47.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e0e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58dc075da0 0x7f58dc19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:40834/0 (socket says 172.21.15.103:40834) 2024-05-23T00:10:47.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58dbe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58dc0766c0 0x7f58dc19ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:37568/0 (socket says 172.21.15.103:37568) 2024-05-23T00:10:47.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.588+0000 7f58e0e00640 1 -- 172.21.15.103:0/1829128100 learned_addr learned my addr 172.21.15.103:0/1829128100 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:10:47.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e0e00640 1 -- 172.21.15.103:0/1829128100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58dc0766c0 msgr2=0x7f58dc19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e0e00640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58dc0766c0 0x7f58dc19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.594 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e0e00640 1 -- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 msgr2=0x7f58dc1a2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e0e00640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 0x7f58dc1a2fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e0e00640 1 -- 172.21.15.103:0/1829128100 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58dc1a36f0 con 0x7f58dc075da0 2024-05-23T00:10:47.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58dbe00640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58dc0766c0 0x7f58dc19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:10:47.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e1800640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 0x7f58dc1a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:10:47.595 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e0e00640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58dc075da0 0x7f58dc19e700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f58d000e9f0 tx=0x7f58d000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:47.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58d9600640 1 -- 172.21.15.103:0/1829128100 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d000cd80 con 0x7f58dc075da0 2024-05-23T00:10:47.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58d9600640 1 -- 172.21.15.103:0/1829128100 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58d0010910 con 0x7f58dc075da0 2024-05-23T00:10:47.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58d9600640 1 -- 172.21.15.103:0/1829128100 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d0018990 con 0x7f58dc075da0 2024-05-23T00:10:47.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58dc1a39e0 con 0x7f58dc075da0 2024-05-23T00:10:47.596 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.589+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58dc1ab270 con 0x7f58dc075da0 2024-05-23T00:10:47.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.590+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58dc068690 con 0x7f58dc075da0 2024-05-23T00:10:47.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.592+0000 7f58d9600640 1 -- 172.21.15.103:0/1829128100 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f58d0010ac0 con 0x7f58dc075da0 2024-05-23T00:10:47.597 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.594+0000 7f58d9600640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f58c40a2d80 0x7f58c40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:47.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.594+0000 7f58d9600640 1 -- 172.21.15.103:0/1829128100 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9612+0+0 (secure 0 0 0) 0x7f58d00d0080 con 0x7f58dc075da0 2024-05-23T00:10:47.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.594+0000 7f58dbe00640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f58c40a2d80 0x7f58c40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:47.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.595+0000 7f58dbe00640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f58c40a2d80 0x7f58c40a5240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f58dc1019f0 tx=0x7f58cc005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:47.598 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.596+0000 7f58d9600640 1 -- 172.21.15.103:0/1829128100 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f58d009c1b0 con 0x7f58dc075da0 2024-05-23T00:10:47.798 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.795+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f58dc02d200 con 0x7f58dc075da0 2024-05-23T00:10:47.799 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.797+0000 7f58d9600640 1 -- 172.21.15.103:0/1829128100 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v104) ==== 103+0+11 (secure 0 0 0) 0x7f58d00a04b0 con 0x7f58dc075da0 2024-05-23T00:10:47.799 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:10:47.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f58c40a2d80 msgr2=0x7f58c40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f58c40a2d80 0x7f58c40a5240 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f58dc1019f0 tx=0x7f58cc005f90 comp rx=0 tx=0).stop 2024-05-23T00:10:47.802 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58dc075da0 msgr2=0x7f58dc19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:47.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58dc075da0 0x7f58dc19e700 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f58d000e9f0 tx=0x7f58d000eeb0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 shutdown_connections 2024-05-23T00:10:47.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f58c40a2d80 0x7f58c40a5240 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58dc1126f0 0x7f58dc1a2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58dc0766c0 0x7f58dc19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 --2- 172.21.15.103:0/1829128100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58dc075da0 0x7f58dc19e700 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:47.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.799+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 >> 172.21.15.103:0/1829128100 conn(0x7f58dc0ff520 msgr2=0x7f58dc114180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:47.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.800+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 shutdown_connections 2024-05-23T00:10:47.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:47.800+0000 7f58e3800640 1 -- 172.21.15.103:0/1829128100 wait complete. 2024-05-23T00:10:47.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:47 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3371744870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:10:47.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:47 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/878409301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:10:47.818 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:47 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3371744870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:10:47.819 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:47 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/878409301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:10:47.822 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-23T00:10:47.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:47 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3371744870' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:10:47.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:47 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/878409301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:10:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 -- 172.21.15.103:0/1059362546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 msgr2=0x7fc4641082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 --2- 172.21.15.103:0/1059362546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 0x7fc4641082c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc454009900 tx=0x7fc45402a410 comp rx=0 tx=0).stop 2024-05-23T00:10:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 -- 172.21.15.103:0/1059362546 shutdown_connections 2024-05-23T00:10:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 --2- 172.21.15.103:0/1059362546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc46410cfa0 0x7fc464114b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:48.026 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 --2- 172.21.15.103:0/1059362546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc464108890 0x7fc46410c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:48.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 --2- 172.21.15.103:0/1059362546 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 0x7fc4641082c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:48.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 -- 172.21.15.103:0/1059362546 >> 172.21.15.103:0/1059362546 conn(0x7fc4640773f0 msgr2=0x7fc464079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:48.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 -- 172.21.15.103:0/1059362546 shutdown_connections 2024-05-23T00:10:48.027 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.023+0000 7fc46a600640 1 -- 172.21.15.103:0/1059362546 wait complete. 2024-05-23T00:10:48.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.024+0000 7fc46a600640 1 Processor -- start 2024-05-23T00:10:48.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.024+0000 7fc46a600640 1 -- start start 2024-05-23T00:10:48.029 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc46a600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 0x7fc46419e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc46a600640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc46410cfa0 0x7fc46419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc46a600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4641a2df0 0x7fc4641a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc46a600640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc464118050 con 0x7fc4641a2df0 2024-05-23T00:10:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc46a600640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fc464117ed0 con 0x7fc46410cfa0 2024-05-23T00:10:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc46a600640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4641181d0 con 0x7fc464107ee0 2024-05-23T00:10:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc469200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 0x7fc46419e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:48.030 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc469200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 0x7fc46419e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:40854/0 (socket says 172.21.15.103:40854) 2024-05-23T00:10:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc469200640 1 -- 172.21.15.103:0/2391553977 learned_addr learned my addr 172.21.15.103:0/2391553977 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:10:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc469c00640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4641a2df0 0x7fc4641a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.025+0000 7fc45be00640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc46410cfa0 0x7fc46419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc469200640 1 -- 172.21.15.103:0/2391553977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc46410cfa0 msgr2=0x7fc46419ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc469200640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc46410cfa0 0x7fc46419ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc469200640 1 -- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4641a2df0 msgr2=0x7fc4641a32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:48.031 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc469200640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4641a2df0 0x7fc4641a32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc469200640 1 -- 172.21.15.103:0/2391553977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4641a38e0 con 0x7fc464107ee0 2024-05-23T00:10:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc469c00640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4641a2df0 0x7fc4641a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-23T00:10:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc469200640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 0x7fc46419e530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc454006010 tx=0x7fc454004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc462000640 1 -- 172.21.15.103:0/2391553977 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45403f030 con 0x7fc464107ee0 2024-05-23T00:10:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc462000640 1 -- 172.21.15.103:0/2391553977 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc454004440 con 0x7fc464107ee0 2024-05-23T00:10:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4641a3b70 con 0x7fc464107ee0 2024-05-23T00:10:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.026+0000 7fc462000640 1 -- 172.21.15.103:0/2391553977 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc454035070 con 0x7fc464107ee0 2024-05-23T00:10:48.032 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.027+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4641ab330 con 0x7fc464107ee0 2024-05-23T00:10:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.028+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc464068a00 con 0x7fc464107ee0 2024-05-23T00:10:48.033 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.030+0000 7fc462000640 1 -- 172.21.15.103:0/2391553977 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fc454042030 con 0x7fc464107ee0 2024-05-23T00:10:48.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.031+0000 7fc462000640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4480a2d00 0x7fc4480a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:10:48.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.031+0000 7fc462000640 1 -- 172.21.15.103:0/2391553977 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 9612+0+0 (secure 0 0 0) 0x7fc4540eff20 con 0x7fc464107ee0 2024-05-23T00:10:48.034 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.031+0000 7fc45be00640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4480a2d00 0x7fc4480a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:10:48.035 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.032+0000 7fc45be00640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4480a2d00 0x7fc4480a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc4641071c0 tx=0x7fc44c005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:10:48.036 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.033+0000 7fc462000640 1 -- 172.21.15.103:0/2391553977 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fc4540bc6a0 con 0x7fc464107ee0 2024-05-23T00:10:48.290 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.288+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fc464108890 con 0x7fc464107ee0 2024-05-23T00:10:48.303 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.300+0000 7fc462000640 1 -- 172.21.15.103:0/2391553977 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+740913 (secure 0 0 0) 0x7fc4540bbfa0 con 0x7fc464107ee0 2024-05-23T00:10:48.305 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:10:48.306 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:10:48.306 INFO:teuthology.orchestra.run.smithi103.stdout: "epoch": 27, 2024-05-23T00:10:48.306 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.306 INFO:teuthology.orchestra.run.smithi103.stdout: "active_gid": 25390, 2024-05-23T00:10:48.306 INFO:teuthology.orchestra.run.smithi103.stdout: "active_name": "x", 2024-05-23T00:10:48.306 INFO:teuthology.orchestra.run.smithi103.stdout: "active_addrs": { 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v2", 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6800", 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1519813025 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v1", 2024-05-23T00:10:48.307 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:6801", 2024-05-23T00:10:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1519813025 2024-05-23T00:10:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "active_addr": "172.21.15.103:6801/1519813025", 2024-05-23T00:10:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "active_change": "2024-05-23T00:00:32.661421+0000", 2024-05-23T00:10:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "active_mgr_features": 4540701547738038271, 2024-05-23T00:10:48.308 INFO:teuthology.orchestra.run.smithi103.stdout: "available": true, 2024-05-23T00:10:48.309 INFO:teuthology.orchestra.run.smithi103.stdout: "standbys": [ 2024-05-23T00:10:48.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.309 INFO:teuthology.orchestra.run.smithi103.stdout: "gid": 25384, 2024-05-23T00:10:48.309 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "z", 2024-05-23T00:10:48.309 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_features": 4540701547738038271, 2024-05-23T00:10:48.309 INFO:teuthology.orchestra.run.smithi103.stdout: "available_modules": [ 2024-05-23T00:10:48.309 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.309 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "alerts", 2024-05-23T00:10:48.310 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.310 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.310 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.310 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:48.310 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:48.310 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.310 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.310 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.311 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:48.311 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.311 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.311 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.311 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How frequently to reexamine health status", 2024-05-23T00:10:48.311 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.311 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.311 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.312 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.312 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.312 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.312 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.312 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.312 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.312 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.313 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.313 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.313 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.313 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.314 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.314 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.314 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.314 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.314 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.314 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.314 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.314 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.315 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.315 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.315 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.315 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.315 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.315 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.315 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.316 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.317 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.317 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.317 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.317 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.317 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.317 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.317 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.317 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.318 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.318 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.318 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.318 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.318 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.318 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.318 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.318 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.319 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.319 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.319 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.319 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.319 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.319 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.319 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.320 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.320 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.320 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.320 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.320 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.320 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.320 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.320 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_destination": { 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_destination", 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.321 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.322 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.322 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.322 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Email address to send alerts to", 2024-05-23T00:10:48.322 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.322 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.322 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.322 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_from_name": { 2024-05-23T00:10:48.323 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_from_name", 2024-05-23T00:10:48.323 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.323 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.323 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.323 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "Ceph", 2024-05-23T00:10:48.323 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.323 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.323 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.324 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Email From: name", 2024-05-23T00:10:48.324 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.324 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.324 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.324 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_host": { 2024-05-23T00:10:48.324 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_host", 2024-05-23T00:10:48.324 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.325 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.325 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.325 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.325 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.325 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.325 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.325 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP server", 2024-05-23T00:10:48.325 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.326 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.326 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.326 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_password": { 2024-05-23T00:10:48.326 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_password", 2024-05-23T00:10:48.326 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.326 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.326 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.327 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.327 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.327 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.327 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.327 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Password to authenticate with", 2024-05-23T00:10:48.327 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.327 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.327 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_port": { 2024-05-23T00:10:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_port", 2024-05-23T00:10:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "465", 2024-05-23T00:10:48.328 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP port", 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_sender": { 2024-05-23T00:10:48.329 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_sender", 2024-05-23T00:10:48.330 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.330 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.330 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.330 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.330 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.330 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.330 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.330 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP envelope sender", 2024-05-23T00:10:48.331 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.331 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.331 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.331 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.331 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_ssl": { 2024-05-23T00:10:48.331 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_ssl", 2024-05-23T00:10:48.331 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.331 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.332 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.332 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.332 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.332 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.332 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.332 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-23T00:10:48.332 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.333 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.333 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.333 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.333 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_user": { 2024-05-23T00:10:48.333 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_user", 2024-05-23T00:10:48.333 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.333 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.333 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.334 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.334 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.334 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.334 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.334 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "User to authenticate as", 2024-05-23T00:10:48.334 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.334 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.334 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.335 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.335 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.335 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.335 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "balancer", 2024-05-23T00:10:48.335 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.335 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.335 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.336 INFO:teuthology.orchestra.run.smithi103.stdout: "active": { 2024-05-23T00:10:48.336 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "active", 2024-05-23T00:10:48.336 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.336 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.336 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.336 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.336 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.336 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.337 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.337 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "automatically balance PGs across cluster", 2024-05-23T00:10:48.337 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.337 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.337 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.337 INFO:teuthology.orchestra.run.smithi103.stdout: "begin_time": { 2024-05-23T00:10:48.338 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "begin_time", 2024-05-23T00:10:48.338 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.338 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.338 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.338 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0000", 2024-05-23T00:10:48.338 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.338 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.338 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.339 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "beginning time of day to automatically balance", 2024-05-23T00:10:48.339 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-23T00:10:48.339 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.339 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.339 INFO:teuthology.orchestra.run.smithi103.stdout: "begin_weekday": { 2024-05-23T00:10:48.339 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "begin_weekday", 2024-05-23T00:10:48.339 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:48.340 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.340 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.340 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:48.340 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0", 2024-05-23T00:10:48.340 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "6", 2024-05-23T00:10:48.340 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.340 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-23T00:10:48.340 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_max_iterations": { 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_max_iterations", 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.341 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "25", 2024-05-23T00:10:48.342 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:48.342 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "250", 2024-05-23T00:10:48.342 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.342 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-23T00:10:48.342 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.342 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.342 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.343 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_metrics": { 2024-05-23T00:10:48.343 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_metrics", 2024-05-23T00:10:48.343 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.343 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.343 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.343 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "pgs,objects,bytes", 2024-05-23T00:10:48.343 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_step": { 2024-05-23T00:10:48.344 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_step", 2024-05-23T00:10:48.345 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:48.345 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.345 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.345 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0.5", 2024-05-23T00:10:48.345 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0.001", 2024-05-23T00:10:48.345 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "0.999", 2024-05-23T00:10:48.345 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.345 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "aggressiveness of optimization", 2024-05-23T00:10:48.346 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-23T00:10:48.346 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.346 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.346 INFO:teuthology.orchestra.run.smithi103.stdout: "end_time": { 2024-05-23T00:10:48.346 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "end_time", 2024-05-23T00:10:48.346 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.346 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.347 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.347 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "2359", 2024-05-23T00:10:48.347 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.347 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.347 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.347 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "ending time of day to automatically balance", 2024-05-23T00:10:48.347 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-23T00:10:48.347 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.348 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.348 INFO:teuthology.orchestra.run.smithi103.stdout: "end_weekday": { 2024-05-23T00:10:48.348 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "end_weekday", 2024-05-23T00:10:48.348 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:48.348 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.348 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.348 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:48.349 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0", 2024-05-23T00:10:48.349 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "6", 2024-05-23T00:10:48.349 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.349 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-23T00:10:48.349 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-23T00:10:48.349 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.349 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.350 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.350 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.350 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.350 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.350 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.350 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.350 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.350 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.351 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.351 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.351 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.351 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.351 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.351 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.352 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.352 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.352 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.352 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.352 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.352 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.352 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.353 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.353 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.353 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.353 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.353 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.353 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.353 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.353 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.354 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.354 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.354 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.354 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.354 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.354 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.354 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.355 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.355 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.355 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.355 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.355 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.355 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.355 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.355 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.356 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.356 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.356 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.356 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.356 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.356 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.356 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.356 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.357 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.357 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.357 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.357 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.357 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.357 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.357 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.357 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.359 INFO:teuthology.orchestra.run.smithi103.stdout: "min_score": { 2024-05-23T00:10:48.359 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "min_score", 2024-05-23T00:10:48.359 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:48.359 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.359 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.359 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:48.359 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.359 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.360 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.360 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-23T00:10:48.360 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.360 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.360 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.360 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.360 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": { 2024-05-23T00:10:48.360 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mode", 2024-05-23T00:10:48.361 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.361 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.361 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.361 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "upmap", 2024-05-23T00:10:48.361 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.361 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.361 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.361 INFO:teuthology.orchestra.run.smithi103.stdout: "crush-compat", 2024-05-23T00:10:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: "none", 2024-05-23T00:10:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: "read", 2024-05-23T00:10:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap", 2024-05-23T00:10:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap-read" 2024-05-23T00:10:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Balancer mode", 2024-05-23T00:10:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.362 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.363 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.363 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_ids": { 2024-05-23T00:10:48.363 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pool_ids", 2024-05-23T00:10:48.363 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.363 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.363 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.363 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.364 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:48.365 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:48.365 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.365 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.365 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.365 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:48.365 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.365 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.365 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.366 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-23T00:10:48.366 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.366 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.366 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.366 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap_max_deviation": { 2024-05-23T00:10:48.366 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "upmap_max_deviation", 2024-05-23T00:10:48.367 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.367 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.367 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.367 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:48.367 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:48.367 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.367 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.367 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap_max_optimizations": { 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "upmap_max_optimizations", 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.368 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.369 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:48.369 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.369 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.369 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.369 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-23T00:10:48.369 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.369 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.369 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.370 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.370 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.370 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.370 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cephadm", 2024-05-23T00:10:48.370 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.370 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_down_multiplier": { 2024-05-23T00:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_down_multiplier", 2024-05-23T00:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3.0", 2024-05-23T00:10:48.371 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-23T00:10:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.372 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_refresh_rate": { 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_refresh_rate", 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "20", 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.373 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-23T00:10:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_starting_port": { 2024-05-23T00:10:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_starting_port", 2024-05-23T00:10:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.374 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.375 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.375 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "4721", 2024-05-23T00:10:48.375 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.375 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.375 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.375 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-23T00:10:48.375 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.375 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.376 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.376 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_ptrace": { 2024-05-23T00:10:48.376 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_ptrace", 2024-05-23T00:10:48.376 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.376 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.376 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.376 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.377 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.377 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.377 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.377 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-23T00:10:48.377 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-23T00:10:48.377 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.377 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.377 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.378 INFO:teuthology.orchestra.run.smithi103.stdout: "autotune_interval": { 2024-05-23T00:10:48.378 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "autotune_interval", 2024-05-23T00:10:48.378 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.378 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.378 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.378 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:48.378 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.378 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.379 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.379 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-23T00:10:48.379 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.379 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.379 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.379 INFO:teuthology.orchestra.run.smithi103.stdout: "autotune_memory_target_ratio": { 2024-05-23T00:10:48.379 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "autotune_memory_target_ratio", 2024-05-23T00:10:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0.7", 2024-05-23T00:10:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.380 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-23T00:10:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm_log_destination": { 2024-05-23T00:10:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cephadm_log_destination", 2024-05-23T00:10:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.381 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.382 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.382 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.382 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.382 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.382 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.382 INFO:teuthology.orchestra.run.smithi103.stdout: "file", 2024-05-23T00:10:48.382 INFO:teuthology.orchestra.run.smithi103.stdout: "file,syslog", 2024-05-23T00:10:48.382 INFO:teuthology.orchestra.run.smithi103.stdout: "syslog" 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: "cgroups_split": { 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cgroups_split", 2024-05-23T00:10:48.383 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.384 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.384 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.384 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.384 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.384 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.384 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.384 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-23T00:10:48.384 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.385 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.385 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.385 INFO:teuthology.orchestra.run.smithi103.stdout: "config_checks_enabled": { 2024-05-23T00:10:48.385 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_checks_enabled", 2024-05-23T00:10:48.385 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.385 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.385 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.386 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.386 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.386 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.386 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.386 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-23T00:10:48.386 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.386 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.386 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.387 INFO:teuthology.orchestra.run.smithi103.stdout: "config_dashboard": { 2024-05-23T00:10:48.387 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_dashboard", 2024-05-23T00:10:48.387 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.387 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.387 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.387 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.387 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.388 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.388 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.388 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-23T00:10:48.388 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.388 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.388 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.388 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_alertmanager": { 2024-05-23T00:10:48.389 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_alertmanager", 2024-05-23T00:10:48.389 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.389 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.389 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.389 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-23T00:10:48.389 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.389 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.389 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.390 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:48.390 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.390 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.390 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.390 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_base": { 2024-05-23T00:10:48.390 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_base", 2024-05-23T00:10:48.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.391 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.391 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-23T00:10:48.391 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.391 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.391 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.391 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Container image name, without the tag", 2024-05-23T00:10:48.391 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_elasticsearch": { 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_elasticsearch", 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.392 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-23T00:10:48.393 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.393 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.393 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.393 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "elasticsearch container image", 2024-05-23T00:10:48.393 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.393 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.393 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.394 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_grafana": { 2024-05-23T00:10:48.394 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_grafana", 2024-05-23T00:10:48.394 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.394 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.394 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.394 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-23T00:10:48.394 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.394 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_haproxy": { 2024-05-23T00:10:48.395 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_haproxy", 2024-05-23T00:10:48.396 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.396 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.396 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.396 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-23T00:10:48.401 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.401 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.401 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.402 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "HAproxy container image", 2024-05-23T00:10:48.402 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.402 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.402 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.402 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_agent": { 2024-05-23T00:10:48.402 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_agent", 2024-05-23T00:10:48.402 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.403 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.403 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.403 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-23T00:10:48.403 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.403 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.403 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.403 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger agent container image", 2024-05-23T00:10:48.403 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.404 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.404 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.404 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_collector": { 2024-05-23T00:10:48.404 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_collector", 2024-05-23T00:10:48.404 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.404 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.404 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.405 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-23T00:10:48.405 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.405 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.405 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.405 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger collector container image", 2024-05-23T00:10:48.406 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.406 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.406 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.406 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_query": { 2024-05-23T00:10:48.406 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_query", 2024-05-23T00:10:48.406 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.406 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.407 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.407 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-23T00:10:48.407 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.407 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.407 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.407 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger query container image", 2024-05-23T00:10:48.407 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.407 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.408 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.408 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_keepalived": { 2024-05-23T00:10:48.408 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_keepalived", 2024-05-23T00:10:48.408 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.408 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.408 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.408 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-23T00:10:48.409 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.409 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.409 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.409 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Keepalived container image", 2024-05-23T00:10:48.409 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.409 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.409 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_loki": { 2024-05-23T00:10:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_loki", 2024-05-23T00:10:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-23T00:10:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.410 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.411 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.411 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Loki container image", 2024-05-23T00:10:48.411 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.411 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.411 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.411 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.411 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_node_exporter": { 2024-05-23T00:10:48.411 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_node_exporter", 2024-05-23T00:10:48.412 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.412 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.412 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.412 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-23T00:10:48.412 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.412 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.412 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.412 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_nvmeof": { 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_nvmeof", 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.413 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.414 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-23T00:10:48.414 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.414 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.414 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.414 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Nvme-of container image", 2024-05-23T00:10:48.414 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.414 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_prometheus": { 2024-05-23T00:10:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_prometheus", 2024-05-23T00:10:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.415 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-23T00:10:48.416 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.416 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.416 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.416 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:48.416 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.416 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.416 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.417 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_promtail": { 2024-05-23T00:10:48.417 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_promtail", 2024-05-23T00:10:48.417 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.417 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.417 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.417 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-23T00:10:48.417 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.417 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.418 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.418 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Promtail container image", 2024-05-23T00:10:48.418 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.418 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.418 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.418 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_samba": { 2024-05-23T00:10:48.418 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_samba", 2024-05-23T00:10:48.419 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.419 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.419 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.419 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-23T00:10:48.419 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.419 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.419 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.419 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Samba/SMB container image", 2024-05-23T00:10:48.420 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.420 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.420 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.420 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_snmp_gateway": { 2024-05-23T00:10:48.420 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_snmp_gateway", 2024-05-23T00:10:48.420 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.420 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.421 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.421 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-23T00:10:48.421 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.421 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.421 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.421 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SNMP Gateway container image", 2024-05-23T00:10:48.421 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.421 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.422 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.422 INFO:teuthology.orchestra.run.smithi103.stdout: "container_init": { 2024-05-23T00:10:48.422 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_init", 2024-05-23T00:10:48.422 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.422 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.422 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.423 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.423 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.423 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.423 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.423 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Run podman/docker with `--init`", 2024-05-23T00:10:48.423 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.423 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.423 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.424 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.424 INFO:teuthology.orchestra.run.smithi103.stdout: "daemon_cache_timeout": { 2024-05-23T00:10:48.424 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "daemon_cache_timeout", 2024-05-23T00:10:48.424 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.424 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.424 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.424 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:48.424 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.425 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.425 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.425 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-23T00:10:48.425 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.425 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.425 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.425 INFO:teuthology.orchestra.run.smithi103.stdout: "default_cephadm_command_timeout": { 2024-05-23T00:10:48.426 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "default_cephadm_command_timeout", 2024-05-23T00:10:48.426 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.426 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.426 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "900", 2024-05-23T00:10:48.426 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.426 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.426 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-23T00:10:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: "default_registry": { 2024-05-23T00:10:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "default_registry", 2024-05-23T00:10:48.427 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.428 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.428 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.428 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io", 2024-05-23T00:10:48.428 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.428 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.428 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.428 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-23T00:10:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: "device_cache_timeout": { 2024-05-23T00:10:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_cache_timeout", 2024-05-23T00:10:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.429 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.430 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.430 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1800", 2024-05-23T00:10:48.430 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.430 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.430 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.430 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache device inventory", 2024-05-23T00:10:48.430 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.430 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.431 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.431 INFO:teuthology.orchestra.run.smithi103.stdout: "device_enhanced_scan": { 2024-05-23T00:10:48.431 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_enhanced_scan", 2024-05-23T00:10:48.431 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.431 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.431 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.431 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.432 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.432 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.432 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.432 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-23T00:10:48.432 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.432 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.432 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.432 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.433 INFO:teuthology.orchestra.run.smithi103.stdout: "facts_cache_timeout": { 2024-05-23T00:10:48.433 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "facts_cache_timeout", 2024-05-23T00:10:48.433 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.433 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.433 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.433 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:48.433 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.434 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.434 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.434 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache host facts data", 2024-05-23T00:10:48.434 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.434 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.434 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.434 INFO:teuthology.orchestra.run.smithi103.stdout: "grafana_dashboards_path": { 2024-05-23T00:10:48.435 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "grafana_dashboards_path", 2024-05-23T00:10:48.435 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.435 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.435 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.435 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-23T00:10:48.435 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.435 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.436 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.436 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-23T00:10:48.436 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.436 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.436 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.436 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.436 INFO:teuthology.orchestra.run.smithi103.stdout: "host_check_interval": { 2024-05-23T00:10:48.436 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "host_check_interval", 2024-05-23T00:10:48.437 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.437 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.437 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.437 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:48.437 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.437 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.437 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.437 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to perform a host check", 2024-05-23T00:10:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: "hw_monitoring": { 2024-05-23T00:10:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "hw_monitoring", 2024-05-23T00:10:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.438 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.439 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.439 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.439 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.439 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.439 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.439 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-23T00:10:48.439 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.439 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.440 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.440 INFO:teuthology.orchestra.run.smithi103.stdout: "inventory_list_all": { 2024-05-23T00:10:48.440 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "inventory_list_all", 2024-05-23T00:10:48.440 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.440 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.440 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.440 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.441 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.441 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.441 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.441 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-23T00:10:48.441 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.441 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.441 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.441 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.442 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.443 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.443 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.443 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.443 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.443 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.443 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.443 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.443 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: "log_refresh_metadata": { 2024-05-23T00:10:48.444 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_refresh_metadata", 2024-05-23T00:10:48.445 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.445 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.445 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.445 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.445 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.445 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.445 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.445 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-23T00:10:48.446 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.446 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.446 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.446 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.446 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.446 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.446 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.447 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.447 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.447 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.447 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.447 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.447 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.447 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-23T00:10:48.447 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.448 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.449 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.449 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.449 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.449 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.449 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.449 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.449 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.450 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.450 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.450 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.450 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.450 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.450 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.450 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.451 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.451 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.451 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.451 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.451 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.451 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.451 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.451 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.452 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.452 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.452 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.452 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.452 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.452 INFO:teuthology.orchestra.run.smithi103.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-23T00:10:48.452 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-23T00:10:48.453 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.454 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.454 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.454 INFO:teuthology.orchestra.run.smithi103.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-23T00:10:48.454 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-23T00:10:48.454 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.454 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.455 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.455 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "*", 2024-05-23T00:10:48.455 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.455 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.455 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.455 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-23T00:10:48.455 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.455 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.456 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.456 INFO:teuthology.orchestra.run.smithi103.stdout: "max_count_per_host": { 2024-05-23T00:10:48.456 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_count_per_host", 2024-05-23T00:10:48.456 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.456 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.456 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.456 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:48.457 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.457 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.457 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.457 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "max number of daemons per service per host", 2024-05-23T00:10:48.457 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.457 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.457 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.457 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.458 INFO:teuthology.orchestra.run.smithi103.stdout: "max_osd_draining_count": { 2024-05-23T00:10:48.458 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_osd_draining_count", 2024-05-23T00:10:48.458 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.458 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.458 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.458 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:48.458 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.458 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.459 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.459 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-23T00:10:48.459 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.459 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.459 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.459 INFO:teuthology.orchestra.run.smithi103.stdout: "migration_current": { 2024-05-23T00:10:48.459 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "migration_current", 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "internal - do not modify", 2024-05-23T00:10:48.460 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.461 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.461 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.461 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": { 2024-05-23T00:10:48.461 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mode", 2024-05-23T00:10:48.461 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.461 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.461 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.462 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "root", 2024-05-23T00:10:48.462 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.462 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.462 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.462 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm-package", 2024-05-23T00:10:48.462 INFO:teuthology.orchestra.run.smithi103.stdout: "root" 2024-05-23T00:10:48.462 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.462 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "mode for remote execution of cephadm", 2024-05-23T00:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: "oob_default_addr": { 2024-05-23T00:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "oob_default_addr", 2024-05-23T00:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.463 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.464 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.464 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "169.254.1.1", 2024-05-23T00:10:48.464 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.464 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.464 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.464 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-23T00:10:48.464 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.464 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.465 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.465 INFO:teuthology.orchestra.run.smithi103.stdout: "prometheus_alerts_path": { 2024-05-23T00:10:48.465 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prometheus_alerts_path", 2024-05-23T00:10:48.465 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.465 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.465 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.465 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-23T00:10:48.466 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.466 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.466 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.466 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-23T00:10:48.466 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.466 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.466 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.467 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_insecure": { 2024-05-23T00:10:48.467 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_insecure", 2024-05-23T00:10:48.467 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.467 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.467 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.467 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.467 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.468 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.468 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.468 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-23T00:10:48.468 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.468 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.468 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.468 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_password": { 2024-05-23T00:10:48.469 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_password", 2024-05-23T00:10:48.469 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.469 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.469 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.469 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.469 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.469 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.469 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-23T00:10:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_url": { 2024-05-23T00:10:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_url", 2024-05-23T00:10:48.470 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.471 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.471 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.471 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.471 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.471 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.471 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.471 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-23T00:10:48.471 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.472 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.472 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.472 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_username": { 2024-05-23T00:10:48.472 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_username", 2024-05-23T00:10:48.472 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.472 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.472 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-23T00:10:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.473 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.474 INFO:teuthology.orchestra.run.smithi103.stdout: "secure_monitoring_stack": { 2024-05-23T00:10:48.474 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "secure_monitoring_stack", 2024-05-23T00:10:48.474 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.474 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.474 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.474 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.474 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.475 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.475 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.475 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-23T00:10:48.475 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.475 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.475 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.475 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.475 INFO:teuthology.orchestra.run.smithi103.stdout: "service_discovery_port": { 2024-05-23T00:10:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "service_discovery_port", 2024-05-23T00:10:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8765", 2024-05-23T00:10:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.476 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.477 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "cephadm service discovery port", 2024-05-23T00:10:48.477 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.477 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.477 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.477 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.477 INFO:teuthology.orchestra.run.smithi103.stdout: "ssh_config_file": { 2024-05-23T00:10:48.477 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssh_config_file", 2024-05-23T00:10:48.477 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.478 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.478 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.478 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.478 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.478 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.478 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.478 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-23T00:10:48.479 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.479 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.479 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.479 INFO:teuthology.orchestra.run.smithi103.stdout: "use_agent": { 2024-05-23T00:10:48.479 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "use_agent", 2024-05-23T00:10:48.479 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.479 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.480 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.480 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.480 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.480 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.480 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.480 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-23T00:10:48.480 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.480 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.481 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.481 INFO:teuthology.orchestra.run.smithi103.stdout: "use_repo_digest": { 2024-05-23T00:10:48.481 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "use_repo_digest", 2024-05-23T00:10:48.481 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.481 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.481 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.481 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.482 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.482 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.482 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.482 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-23T00:10:48.482 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.482 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.482 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_failed_host_check": { 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_failed_host_check", 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.483 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.484 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if the host check fails", 2024-05-23T00:10:48.484 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.484 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.484 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.484 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_stray_daemons": { 2024-05-23T00:10:48.484 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_stray_daemons", 2024-05-23T00:10:48.484 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.485 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.485 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.485 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.485 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.485 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.485 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.485 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-23T00:10:48.485 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.486 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.486 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.486 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_stray_hosts": { 2024-05-23T00:10:48.486 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_stray_hosts", 2024-05-23T00:10:48.486 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.486 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.487 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.487 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.487 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.487 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.487 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.487 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-23T00:10:48.487 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.487 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crash", 2024-05-23T00:10:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.488 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.489 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.489 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.489 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.489 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.489 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.489 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.489 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.489 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.490 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.490 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.490 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.490 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.490 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.490 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.490 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.491 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.491 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.491 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.491 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.491 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.491 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.491 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.492 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.492 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.492 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.492 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.492 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.492 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.492 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.492 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.493 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.494 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.495 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.496 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.496 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.496 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.496 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.496 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.496 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.496 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.497 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.497 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.497 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.497 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.497 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.497 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.497 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.498 INFO:teuthology.orchestra.run.smithi103.stdout: "retain_interval": { 2024-05-23T00:10:48.498 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "retain_interval", 2024-05-23T00:10:48.498 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.498 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.498 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.498 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "31536000", 2024-05-23T00:10:48.498 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.501 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.501 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.501 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-23T00:10:48.501 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.501 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.501 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.502 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.502 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_recent_interval": { 2024-05-23T00:10:48.502 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_recent_interval", 2024-05-23T00:10:48.502 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.502 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.502 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.502 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1209600", 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.503 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "dashboard", 2024-05-23T00:10:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-23T00:10:48.504 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-23T00:10:48.505 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.505 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.505 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.505 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:48.505 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.505 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.505 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.506 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.506 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.506 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.506 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.506 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.506 INFO:teuthology.orchestra.run.smithi103.stdout: "ALERTMANAGER_API_HOST": { 2024-05-23T00:10:48.506 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-23T00:10:48.506 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.507 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.507 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.507 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.507 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.507 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.507 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.507 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.507 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.508 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.508 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.508 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.508 INFO:teuthology.orchestra.run.smithi103.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-23T00:10:48.508 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-23T00:10:48.508 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.508 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.508 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.509 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.509 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.509 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.509 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.509 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.509 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.509 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.509 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.510 INFO:teuthology.orchestra.run.smithi103.stdout: "AUDIT_API_ENABLED": { 2024-05-23T00:10:48.510 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "AUDIT_API_ENABLED", 2024-05-23T00:10:48.510 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.510 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.510 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.510 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.510 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.511 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.511 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.511 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.511 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.511 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.511 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.511 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.511 INFO:teuthology.orchestra.run.smithi103.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-23T00:10:48.512 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-23T00:10:48.512 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.512 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.512 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.512 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.512 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.512 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.512 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.513 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.513 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.513 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.513 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.513 INFO:teuthology.orchestra.run.smithi103.stdout: "ENABLE_BROWSABLE_API": { 2024-05-23T00:10:48.513 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-23T00:10:48.513 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.514 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.514 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.514 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.514 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.514 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.514 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.514 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.514 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.515 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.515 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.515 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.515 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-23T00:10:48.515 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-23T00:10:48.515 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.515 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.515 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.516 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.516 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.516 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.516 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.516 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.516 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.516 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.516 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.517 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-23T00:10:48.517 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-23T00:10:48.517 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.517 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.517 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.517 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.517 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.518 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.518 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.518 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.518 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.518 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.518 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.518 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-23T00:10:48.519 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-23T00:10:48.519 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.519 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.519 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.519 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.519 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.519 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-23T00:10:48.520 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.521 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.521 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.521 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.521 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.521 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.521 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.521 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.521 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.522 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.522 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.522 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-23T00:10:48.522 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-23T00:10:48.522 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.522 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.523 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.523 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.523 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.523 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.523 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.523 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.523 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.523 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.524 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.524 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-23T00:10:48.524 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-23T00:10:48.524 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.524 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.524 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.524 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.525 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-23T00:10:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-23T00:10:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.526 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-23T00:10:48.527 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.528 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.528 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.528 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.528 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.528 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.528 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.528 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.528 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.529 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.529 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.529 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_PASSWORD": { 2024-05-23T00:10:48.529 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-23T00:10:48.529 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.529 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.529 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.530 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:48.530 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.530 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.530 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.530 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.530 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.530 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.530 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.531 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-23T00:10:48.531 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-23T00:10:48.531 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.531 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.531 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.531 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.531 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.532 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.532 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.532 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.532 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.532 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.532 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.532 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.532 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_URL": { 2024-05-23T00:10:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_URL", 2024-05-23T00:10:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_USERNAME": { 2024-05-23T00:10:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-23T00:10:48.534 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.535 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-23T00:10:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-23T00:10:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.536 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.537 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.537 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.537 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.537 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.537 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.537 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.537 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.537 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.538 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-23T00:10:48.538 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-23T00:10:48.538 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.538 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.538 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.538 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.538 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.539 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.539 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.539 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.539 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.539 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.539 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.539 INFO:teuthology.orchestra.run.smithi103.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.540 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.541 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.541 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.541 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.541 INFO:teuthology.orchestra.run.smithi103.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-23T00:10:48.541 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-23T00:10:48.541 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.541 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.542 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.542 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.542 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.542 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.542 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.542 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.542 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.542 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.543 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.543 INFO:teuthology.orchestra.run.smithi103.stdout: "PROMETHEUS_API_HOST": { 2024-05-23T00:10:48.543 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-23T00:10:48.543 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.543 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.543 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.543 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.544 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.544 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.544 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.544 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.544 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.544 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.544 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.545 INFO:teuthology.orchestra.run.smithi103.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-23T00:10:48.545 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-23T00:10:48.545 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.545 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.545 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.545 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.545 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.545 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.546 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.546 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.546 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.546 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.546 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.546 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-23T00:10:48.546 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-23T00:10:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.547 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-23T00:10:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-23T00:10:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.548 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.549 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.549 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.549 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.549 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.549 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.549 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.549 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.549 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.550 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.550 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-23T00:10:48.550 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-23T00:10:48.550 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.550 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.550 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.550 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.551 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.551 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.551 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.551 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.551 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.551 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.551 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.552 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-23T00:10:48.552 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-23T00:10:48.552 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.552 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.552 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.552 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.552 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.552 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-23T00:10:48.553 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.554 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.554 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.554 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.554 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.554 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.554 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.554 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.555 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.556 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.556 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.556 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.556 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.556 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.556 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.556 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.556 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.557 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-23T00:10:48.557 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-23T00:10:48.557 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.557 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.557 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.557 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.557 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.558 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.558 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.558 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.558 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.558 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.558 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.558 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.558 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_ENABLED": { 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.559 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-23T00:10:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-23T00:10:48.560 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-23T00:10:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.561 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-23T00:10:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-23T00:10:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.562 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.563 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-23T00:10:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-23T00:10:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8", 2024-05-23T00:10:48.564 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-23T00:10:48.565 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-23T00:10:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "45", 2024-05-23T00:10:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.566 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_ACCESS_KEY": { 2024-05-23T00:10:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-23T00:10:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.568 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-23T00:10:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-23T00:10:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.569 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.570 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_SECRET_KEY": { 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.571 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_SSL_VERIFY": { 2024-05-23T00:10:48.572 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-23T00:10:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.573 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-23T00:10:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-23T00:10:48.574 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.575 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-23T00:10:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-23T00:10:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.576 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.577 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-23T00:10:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-23T00:10:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:48.578 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.579 INFO:teuthology.orchestra.run.smithi103.stdout: "cross_origin_url": { 2024-05-23T00:10:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cross_origin_url", 2024-05-23T00:10:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.580 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "crt_file": { 2024-05-23T00:10:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crt_file", 2024-05-23T00:10:48.581 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.582 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "debug": { 2024-05-23T00:10:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "debug", 2024-05-23T00:10:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.583 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable/disable debug options", 2024-05-23T00:10:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.584 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "jwt_token_ttl": { 2024-05-23T00:10:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "jwt_token_ttl", 2024-05-23T00:10:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.585 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "28800", 2024-05-23T00:10:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.586 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "key_file": { 2024-05-23T00:10:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "key_file", 2024-05-23T00:10:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.587 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.588 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.589 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.590 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.591 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.592 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.593 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.594 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.595 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.596 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "motd": { 2024-05-23T00:10:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "motd", 2024-05-23T00:10:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.597 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "The message of the day", 2024-05-23T00:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.598 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "redirect_resolve_ip_addr": { 2024-05-23T00:10:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "redirect_resolve_ip_addr", 2024-05-23T00:10:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.599 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "::", 2024-05-23T00:10:48.601 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.602 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8080", 2024-05-23T00:10:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.603 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl": { 2024-05-23T00:10:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl", 2024-05-23T00:10:48.604 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.605 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl_server_port": { 2024-05-23T00:10:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl_server_port", 2024-05-23T00:10:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.606 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8443", 2024-05-23T00:10:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.607 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_behaviour": { 2024-05-23T00:10:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_behaviour", 2024-05-23T00:10:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.608 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "redirect", 2024-05-23T00:10:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "redirect" 2024-05-23T00:10:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.609 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_error_status_code": { 2024-05-23T00:10:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_error_status_code", 2024-05-23T00:10:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.610 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "500", 2024-05-23T00:10:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "400", 2024-05-23T00:10:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "599", 2024-05-23T00:10:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.611 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "url_prefix": { 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "url_prefix", 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.612 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.613 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "devicehealth", 2024-05-23T00:10:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "enable_monitoring": { 2024-05-23T00:10:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enable_monitoring", 2024-05-23T00:10:48.614 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "monitor device health metrics", 2024-05-23T00:10:48.615 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.616 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.617 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.619 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.620 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.621 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.622 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.623 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.624 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "mark_out_threshold": { 2024-05-23T00:10:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mark_out_threshold", 2024-05-23T00:10:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "2419200", 2024-05-23T00:10:48.625 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-23T00:10:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.626 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": { 2024-05-23T00:10:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pool_name", 2024-05-23T00:10:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "device_health_metrics", 2024-05-23T00:10:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.627 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-23T00:10:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "retention_period": { 2024-05-23T00:10:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "retention_period", 2024-05-23T00:10:48.628 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15552000", 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long to retain device health metrics", 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.629 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "scrape_frequency": { 2024-05-23T00:10:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "scrape_frequency", 2024-05-23T00:10:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.630 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "86400", 2024-05-23T00:10:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-23T00:10:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.631 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "self_heal": { 2024-05-23T00:10:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "self_heal", 2024-05-23T00:10:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.632 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-23T00:10:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.633 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.634 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to wake up and check device health", 2024-05-23T00:10:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_threshold": { 2024-05-23T00:10:48.635 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_threshold", 2024-05-23T00:10:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "7257600", 2024-05-23T00:10:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.636 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-23T00:10:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.637 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "diskprediction_local", 2024-05-23T00:10:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.638 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.639 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.640 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.641 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.642 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.643 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.644 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.645 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.646 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "predict_interval": { 2024-05-23T00:10:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "predict_interval", 2024-05-23T00:10:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "86400", 2024-05-23T00:10:48.647 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.648 INFO:teuthology.orchestra.run.smithi103.stdout: "predictor_model": { 2024-05-23T00:10:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "predictor_model", 2024-05-23T00:10:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "prophetstor", 2024-05-23T00:10:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.649 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:48.650 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.651 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "influx", 2024-05-23T00:10:48.652 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": false, 2024-05-23T00:10:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "influxdb python module not found", 2024-05-23T00:10:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "batch_size": { 2024-05-23T00:10:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "batch_size", 2024-05-23T00:10:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.653 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5000", 2024-05-23T00:10:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-23T00:10:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "database": { 2024-05-23T00:10:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "database", 2024-05-23T00:10:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "ceph", 2024-05-23T00:10:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.655 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-23T00:10:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.656 INFO:teuthology.orchestra.run.smithi103.stdout: "hostname": { 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "hostname", 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.657 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB server hostname", 2024-05-23T00:10:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:48.658 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "30", 2024-05-23T00:10:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "5", 2024-05-23T00:10:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-23T00:10:48.659 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.660 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.661 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.663 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.664 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.665 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.666 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.667 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "password": { 2024-05-23T00:10:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "password", 2024-05-23T00:10:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.669 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "password of InfluxDB server user", 2024-05-23T00:10:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.670 INFO:teuthology.orchestra.run.smithi103.stdout: "port": { 2024-05-23T00:10:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "port", 2024-05-23T00:10:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8086", 2024-05-23T00:10:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.671 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB server port", 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl": { 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl", 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.672 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "false", 2024-05-23T00:10:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-23T00:10:48.673 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "threads": { 2024-05-23T00:10:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "threads", 2024-05-23T00:10:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.674 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "32", 2024-05-23T00:10:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-23T00:10:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.675 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "username": { 2024-05-23T00:10:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "username", 2024-05-23T00:10:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.676 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "username of InfluxDB server user", 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.677 INFO:teuthology.orchestra.run.smithi103.stdout: "verify_ssl": { 2024-05-23T00:10:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "verify_ssl", 2024-05-23T00:10:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "true", 2024-05-23T00:10:48.678 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-23T00:10:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.679 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "insights", 2024-05-23T00:10:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.680 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.681 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.682 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.683 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.684 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.684 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.684 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.684 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.684 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.684 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.684 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.684 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.685 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.685 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.685 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.685 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.685 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.685 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.685 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.686 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.686 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.686 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.686 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.686 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.686 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.686 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.686 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.687 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.687 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.687 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.687 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.687 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.687 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.687 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.688 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.688 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.688 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.688 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.688 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.688 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.688 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.688 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.689 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.689 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.689 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.689 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.689 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.689 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.689 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "iostat", 2024-05-23T00:10:48.690 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.690 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.690 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.690 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.690 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.690 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.690 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.691 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.691 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.691 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.691 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.691 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.691 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.691 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.691 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.692 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.693 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.694 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.694 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.694 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.694 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.694 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.694 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.694 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.695 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.695 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.695 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.695 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.695 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.695 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.695 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.695 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.696 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.696 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.696 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.696 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.696 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.696 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.696 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.696 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.697 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.698 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.698 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.698 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.698 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.698 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.698 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.698 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.698 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.699 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.699 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.699 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.699 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.699 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.699 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "k8sevents", 2024-05-23T00:10:48.699 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_event_retention_days": { 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ceph_event_retention_days", 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "7", 2024-05-23T00:10:48.700 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.701 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.701 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.701 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-23T00:10:48.701 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.701 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.701 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.701 INFO:teuthology.orchestra.run.smithi103.stdout: "config_check_secs": { 2024-05-23T00:10:48.702 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_check_secs", 2024-05-23T00:10:48.702 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.702 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.702 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.702 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:48.702 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "10", 2024-05-23T00:10:48.702 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.703 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.703 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-23T00:10:48.703 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.703 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.703 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.703 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.703 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.703 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.704 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.705 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.705 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.705 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.705 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.705 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.705 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.705 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.705 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.706 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.706 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.706 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.706 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.706 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.706 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.706 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.707 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.707 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.707 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.707 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.707 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.707 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.707 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.707 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.709 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.710 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.710 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.710 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.710 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.710 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.710 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.710 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.710 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.711 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.711 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.711 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.711 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.711 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.711 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.711 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.711 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.712 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.712 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.712 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.712 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.712 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.712 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.712 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.713 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.713 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.713 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.713 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.713 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.713 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.713 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.713 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.714 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.714 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.714 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.714 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.714 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.714 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "localpool", 2024-05-23T00:10:48.714 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "failure_domain": { 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "failure_domain", 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "host", 2024-05-23T00:10:48.715 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.716 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.716 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.716 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "failure domain for any created local pool", 2024-05-23T00:10:48.716 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-23T00:10:48.716 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.716 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.716 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.716 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.717 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.717 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.717 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.717 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.717 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.717 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.717 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.717 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.718 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.718 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.718 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.718 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.718 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.718 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.718 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.719 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.719 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.719 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.719 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.719 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.719 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.719 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.720 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.721 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.721 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.721 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.721 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.721 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.721 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.721 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.722 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.722 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.722 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.722 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.722 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.722 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.722 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.722 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.723 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.723 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.723 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.723 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.723 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.723 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.723 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.724 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.724 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.724 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.724 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.724 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.724 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.724 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.724 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.725 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.725 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.725 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.725 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.725 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.725 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": { 2024-05-23T00:10:48.725 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "min_size", 2024-05-23T00:10:48.726 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.726 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.726 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.726 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.726 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.726 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.726 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.726 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default min_size for any created local pool", 2024-05-23T00:10:48.727 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-23T00:10:48.727 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.727 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.727 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.727 INFO:teuthology.orchestra.run.smithi103.stdout: "num_rep": { 2024-05-23T00:10:48.727 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "num_rep", 2024-05-23T00:10:48.727 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.727 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.728 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.728 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3", 2024-05-23T00:10:48.728 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.728 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.728 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.728 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default replica count for any created local pool", 2024-05-23T00:10:48.728 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.729 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.729 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.729 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.729 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": { 2024-05-23T00:10:48.729 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pg_num", 2024-05-23T00:10:48.729 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.729 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.729 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.730 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "128", 2024-05-23T00:10:48.730 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.730 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.730 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.730 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default pg_num for any created local pool", 2024-05-23T00:10:48.730 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.730 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.730 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.731 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.731 INFO:teuthology.orchestra.run.smithi103.stdout: "prefix": { 2024-05-23T00:10:48.731 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prefix", 2024-05-23T00:10:48.731 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.731 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.731 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.731 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.731 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.732 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.732 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.732 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "name prefix for any created local pool", 2024-05-23T00:10:48.732 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.732 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.732 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.732 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.732 INFO:teuthology.orchestra.run.smithi103.stdout: "subtree": { 2024-05-23T00:10:48.733 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "subtree", 2024-05-23T00:10:48.733 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.733 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.733 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.733 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "rack", 2024-05-23T00:10:48.733 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.733 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.733 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.734 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-23T00:10:48.734 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-23T00:10:48.734 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.734 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.734 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.734 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.734 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.734 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.735 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mds_autoscaler", 2024-05-23T00:10:48.735 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.735 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.735 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.735 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.735 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.735 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.735 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.736 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.736 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.736 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.736 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.736 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.736 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.736 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.736 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.737 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.737 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.737 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.737 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.737 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.737 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.737 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.737 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.738 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.738 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.738 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.738 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.738 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.738 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.738 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.739 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.739 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.739 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.739 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.739 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.739 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.739 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.739 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.740 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.741 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.741 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.741 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.741 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.741 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.741 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.741 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.741 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.742 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.742 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.742 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.742 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.742 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.742 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.742 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.742 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.743 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.744 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.744 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.744 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.744 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.744 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mirroring", 2024-05-23T00:10:48.744 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.744 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.745 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.745 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.745 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.745 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.745 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.745 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.745 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.745 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.746 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.746 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.746 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.746 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.746 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.746 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.746 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.747 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.748 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.748 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.748 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.748 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.748 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.748 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.748 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.748 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.749 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.749 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.749 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.749 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.749 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.749 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.749 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.749 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.750 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.750 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.750 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.750 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.750 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.750 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.750 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.750 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.751 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.751 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.751 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.751 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.751 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.751 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.751 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.751 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.752 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.752 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.752 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.752 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.752 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.752 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.753 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.753 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.753 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.753 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.753 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.753 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.753 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.753 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.754 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.754 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.754 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.754 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "nfs", 2024-05-23T00:10:48.754 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.754 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.754 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.754 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.755 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.755 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.755 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.755 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.755 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.755 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.755 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.755 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.756 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.756 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.756 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.756 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.756 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.756 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.756 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.756 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.757 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.757 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.757 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.757 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.757 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.757 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.757 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.757 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.758 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.758 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.758 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.758 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.758 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.758 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.758 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.758 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.759 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.759 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.759 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.759 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.759 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.759 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.759 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.760 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.761 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.761 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.761 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.761 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.761 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.761 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.761 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.761 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.762 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.762 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.762 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.762 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.762 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.762 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.762 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.762 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.763 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.763 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.763 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.763 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.763 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.763 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.763 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.763 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.764 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "orchestrator", 2024-05-23T00:10:48.764 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.764 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.764 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.764 INFO:teuthology.orchestra.run.smithi103.stdout: "fail_fs": { 2024-05-23T00:10:48.764 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "fail_fs", 2024-05-23T00:10:48.764 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.764 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-23T00:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.765 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.766 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.767 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.767 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.767 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.767 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.767 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.767 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.767 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.768 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.769 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.769 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.769 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.769 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.769 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.769 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.769 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.770 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.771 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.772 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.773 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.774 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.775 INFO:teuthology.orchestra.run.smithi103.stdout: "orchestrator": { 2024-05-23T00:10:48.775 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "orchestrator", 2024-05-23T00:10:48.775 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.775 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.775 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.775 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.775 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.775 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.776 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.776 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm", 2024-05-23T00:10:48.776 INFO:teuthology.orchestra.run.smithi103.stdout: "rook", 2024-05-23T00:10:48.776 INFO:teuthology.orchestra.run.smithi103.stdout: "test_orchestrator" 2024-05-23T00:10:48.776 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.776 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Orchestrator backend", 2024-05-23T00:10:48.776 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.776 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.777 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.777 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.777 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.777 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.777 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "osd_perf_query", 2024-05-23T00:10:48.777 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.777 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.778 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.778 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.778 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.778 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.778 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.778 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.778 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.779 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.779 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.779 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.779 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.779 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.779 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.779 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.779 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.780 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.780 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.780 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.780 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.780 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.780 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.780 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.780 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.781 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.782 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.782 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.782 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.782 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.782 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.782 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.782 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.782 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.783 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.783 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.783 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.783 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.783 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.783 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.783 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.783 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.784 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.785 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.785 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.785 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.785 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.785 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.785 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.785 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.785 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.786 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.786 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.786 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.786 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.786 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.786 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.786 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.786 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.787 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.787 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.787 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "osd_support", 2024-05-23T00:10:48.787 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.787 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.787 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.787 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.787 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.788 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.788 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.788 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.788 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.788 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.788 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.788 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.788 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.789 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.789 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.789 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.789 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.789 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.789 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.789 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.789 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.790 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.790 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.790 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.790 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.790 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.790 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.790 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.791 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.791 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.791 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.791 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.791 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.791 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.791 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.791 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.792 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.792 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.792 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.792 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.792 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.792 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.792 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.793 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.793 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.793 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.793 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.793 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.793 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.793 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.793 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.794 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.795 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.795 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.795 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.795 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.795 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.795 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.795 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.795 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.796 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.796 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.796 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.796 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.796 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.796 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.796 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pg_autoscaler", 2024-05-23T00:10:48.797 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.797 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.797 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.797 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.797 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.797 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.797 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.797 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.798 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.798 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.798 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.798 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.798 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.798 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.798 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.798 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.799 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.799 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.799 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.799 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.799 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.799 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.799 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.800 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.800 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.800 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.800 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.800 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.800 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.800 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.800 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.801 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.802 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.802 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.802 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.802 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.802 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.803 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.803 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.803 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.803 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.803 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.803 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.804 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.804 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.804 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.804 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.804 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.804 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.804 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.805 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.805 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.805 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.805 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.805 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.805 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.805 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.806 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.806 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.806 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.806 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.806 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.806 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.807 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:48.807 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:48.807 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.807 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.807 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.807 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:48.808 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.808 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.808 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.808 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.808 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.809 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.809 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.809 INFO:teuthology.orchestra.run.smithi103.stdout: "threshold": { 2024-05-23T00:10:48.809 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "threshold", 2024-05-23T00:10:48.809 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:48.810 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.810 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.810 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3.0", 2024-05-23T00:10:48.810 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1.0", 2024-05-23T00:10:48.810 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.811 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.811 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "scaling threshold", 2024-05-23T00:10:48.811 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-23T00:10:48.811 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.811 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.812 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.812 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.812 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.812 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "progress", 2024-05-23T00:10:48.812 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.813 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.813 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.813 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_pg_recovery_event": { 2024-05-23T00:10:48.813 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_pg_recovery_event", 2024-05-23T00:10:48.813 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.813 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.813 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.814 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.814 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.814 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.814 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.814 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-23T00:10:48.814 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.815 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.815 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.815 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": { 2024-05-23T00:10:48.815 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enabled", 2024-05-23T00:10:48.815 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.815 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.816 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.816 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.816 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.816 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.816 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.816 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.816 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.817 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.817 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.817 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.817 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.817 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.817 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.818 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.818 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.818 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.818 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.819 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.819 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.819 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.819 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.819 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.819 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.820 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.820 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.820 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.820 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.820 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.820 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.820 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.821 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.821 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.821 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.821 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.821 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.821 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.821 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.822 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.822 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.822 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.822 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.822 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.823 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.823 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.823 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.823 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.823 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.823 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.823 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.824 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.824 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.824 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.824 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.824 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.824 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.824 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.825 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.825 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.825 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.825 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.825 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.825 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.826 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.826 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.826 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.826 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.826 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.826 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.828 INFO:teuthology.orchestra.run.smithi103.stdout: "max_completed_events": { 2024-05-23T00:10:48.828 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_completed_events", 2024-05-23T00:10:48.828 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.828 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.828 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.828 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "50", 2024-05-23T00:10:48.828 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.829 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.829 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.829 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "number of past completed events to remember", 2024-05-23T00:10:48.829 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.829 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.829 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.829 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.830 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:48.830 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:48.830 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:48.830 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.830 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.830 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:48.831 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.831 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.831 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.831 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long the module is going to sleep", 2024-05-23T00:10:48.831 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.831 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prometheus", 2024-05-23T00:10:48.832 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.833 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.833 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.833 INFO:teuthology.orchestra.run.smithi103.stdout: "cache": { 2024-05-23T00:10:48.833 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cache", 2024-05-23T00:10:48.833 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.833 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.833 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.834 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.834 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.834 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.834 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.834 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.834 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.835 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.835 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.835 INFO:teuthology.orchestra.run.smithi103.stdout: "exclude_perf_counters": { 2024-05-23T00:10:48.835 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "exclude_perf_counters", 2024-05-23T00:10:48.835 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.836 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.836 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.836 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.836 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.836 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.836 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.836 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-23T00:10:48.837 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-23T00:10:48.837 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.837 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.837 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.837 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.837 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.837 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.838 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.839 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.839 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.839 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.839 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.839 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.839 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.839 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.840 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.840 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.840 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.840 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.840 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.840 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.840 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.841 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.841 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.841 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.841 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.841 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.841 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.842 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.842 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.842 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.842 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.842 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.842 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.842 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.843 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.843 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.843 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.843 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.843 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.843 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.843 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.844 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.844 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.844 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.844 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.844 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.844 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.844 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.845 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.845 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.845 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.845 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.845 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.845 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.846 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.846 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.846 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.846 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.846 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.846 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.846 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.847 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.847 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.847 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.847 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_stats_pools": { 2024-05-23T00:10:48.847 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_stats_pools", 2024-05-23T00:10:48.847 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.848 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.849 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.849 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.849 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-23T00:10:48.849 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-23T00:10:48.849 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.849 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.849 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.850 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "300", 2024-05-23T00:10:48.850 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.850 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.850 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.850 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.850 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.851 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.851 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.851 INFO:teuthology.orchestra.run.smithi103.stdout: "scrape_interval": { 2024-05-23T00:10:48.851 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "scrape_interval", 2024-05-23T00:10:48.851 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:48.851 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.852 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.852 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15.0", 2024-05-23T00:10:48.852 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.852 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.852 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.852 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.853 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.853 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.853 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.853 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.853 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:48.853 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:48.853 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.854 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.854 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.854 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "::", 2024-05-23T00:10:48.854 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.854 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.854 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.855 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-23T00:10:48.855 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.855 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.855 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.855 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:48.856 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:48.856 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.856 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.856 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.856 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "9283", 2024-05-23T00:10:48.856 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.857 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.857 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.857 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-23T00:10:48.857 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.857 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.857 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.858 INFO:teuthology.orchestra.run.smithi103.stdout: "stale_cache_strategy": { 2024-05-23T00:10:48.858 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "stale_cache_strategy", 2024-05-23T00:10:48.858 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.858 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.858 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.859 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "log", 2024-05-23T00:10:48.859 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.859 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.859 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.859 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.859 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.859 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.860 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.860 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_behaviour": { 2024-05-23T00:10:48.860 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_behaviour", 2024-05-23T00:10:48.860 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.860 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.860 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.861 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "default", 2024-05-23T00:10:48.861 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.861 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.861 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.861 INFO:teuthology.orchestra.run.smithi103.stdout: "default", 2024-05-23T00:10:48.861 INFO:teuthology.orchestra.run.smithi103.stdout: "error" 2024-05-23T00:10:48.861 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.862 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.862 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.862 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.862 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.862 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_error_status_code": { 2024-05-23T00:10:48.863 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_error_status_code", 2024-05-23T00:10:48.863 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.863 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.863 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.863 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "500", 2024-05-23T00:10:48.863 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "400", 2024-05-23T00:10:48.863 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "599", 2024-05-23T00:10:48.864 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.864 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.864 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.864 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.864 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.864 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.865 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.865 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.865 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_support", 2024-05-23T00:10:48.865 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.865 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.865 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.866 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.866 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.866 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.866 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.866 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.866 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.866 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.867 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.867 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.867 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.867 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.867 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.867 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.867 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.868 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.868 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.868 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.868 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.868 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.868 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.869 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.870 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.871 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.872 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.873 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.874 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.875 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.876 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "max_concurrent_snap_create": { 2024-05-23T00:10:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_concurrent_snap_create", 2024-05-23T00:10:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.877 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.878 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "mirror_snapshot_schedule": { 2024-05-23T00:10:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mirror_snapshot_schedule", 2024-05-23T00:10:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.879 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.880 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.881 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "trash_purge_schedule": { 2024-05-23T00:10:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "trash_purge_schedule", 2024-05-23T00:10:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.882 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.883 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "restful", 2024-05-23T00:10:48.884 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "enable_auth": { 2024-05-23T00:10:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enable_auth", 2024-05-23T00:10:48.885 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.886 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.887 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "key_file": { 2024-05-23T00:10:48.888 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "key_file", 2024-05-23T00:10:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.889 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.890 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.892 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.893 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.894 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.895 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.896 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.897 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.898 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.899 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.900 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.901 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.902 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.904 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.905 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.906 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.907 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.908 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rgw", 2024-05-23T00:10:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.909 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.910 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.911 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.912 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.913 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.914 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.915 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.916 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.917 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.918 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.919 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.920 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.921 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rook", 2024-05-23T00:10:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.922 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.923 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.924 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.925 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.926 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.927 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.928 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.929 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.930 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.931 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.932 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.933 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.934 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.935 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.936 INFO:teuthology.orchestra.run.smithi103.stdout: "storage_class": { 2024-05-23T00:10:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "storage_class", 2024-05-23T00:10:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.937 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "local", 2024-05-23T00:10:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.938 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-23T00:10:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.939 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "selftest", 2024-05-23T00:10:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.940 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.941 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.942 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.943 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.944 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.945 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.946 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.947 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.948 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:48.949 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.950 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.951 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "roption1": { 2024-05-23T00:10:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "roption1", 2024-05-23T00:10:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.952 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.953 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "roption2": { 2024-05-23T00:10:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "roption2", 2024-05-23T00:10:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.954 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "xyz", 2024-05-23T00:10:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.955 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption1": { 2024-05-23T00:10:48.956 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption1", 2024-05-23T00:10:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.957 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.958 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption2": { 2024-05-23T00:10:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption2", 2024-05-23T00:10:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.959 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.961 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption3": { 2024-05-23T00:10:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption3", 2024-05-23T00:10:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:48.962 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.963 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption4": { 2024-05-23T00:10:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption4", 2024-05-23T00:10:48.964 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.965 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.966 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption5": { 2024-05-23T00:10:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption5", 2024-05-23T00:10:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.967 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption6": { 2024-05-23T00:10:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption6", 2024-05-23T00:10:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:48.969 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption7": { 2024-05-23T00:10:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption7", 2024-05-23T00:10:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.971 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "42", 2024-05-23T00:10:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.972 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "testkey": { 2024-05-23T00:10:48.973 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testkey", 2024-05-23T00:10:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.974 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.975 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "testlkey": { 2024-05-23T00:10:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testlkey", 2024-05-23T00:10:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.976 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.977 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.978 INFO:teuthology.orchestra.run.smithi103.stdout: "testnewline": { 2024-05-23T00:10:48.979 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testnewline", 2024-05-23T00:10:48.979 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.979 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.979 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:48.980 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.980 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.980 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.980 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.981 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.981 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.981 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.981 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.982 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:48.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.982 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:48.983 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snap_schedule", 2024-05-23T00:10:48.983 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:48.983 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:48.984 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:48.984 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_m_granularity": { 2024-05-23T00:10:48.984 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_m_granularity", 2024-05-23T00:10:48.984 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.984 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.985 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.985 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.985 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.985 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.986 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.986 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow minute scheduled snapshots", 2024-05-23T00:10:48.986 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.986 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.987 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.987 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.987 INFO:teuthology.orchestra.run.smithi103.stdout: "dump_on_update": { 2024-05-23T00:10:48.987 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "dump_on_update", 2024-05-23T00:10:48.988 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.988 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.988 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.988 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.988 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.989 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.989 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.989 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "dump database to debug log on update", 2024-05-23T00:10:48.990 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.990 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.990 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.991 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:48.991 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:48.991 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:48.992 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.992 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.992 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:48.992 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.993 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.993 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:48.993 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:48.993 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:48.994 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:48.994 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:48.994 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:48.994 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:48.995 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:48.995 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.995 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.995 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.996 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:48.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:48.996 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:48.996 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:48.997 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:48.997 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:48.997 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:48.998 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:48.998 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:48.998 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:48.998 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:48.999 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:48.999 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:48.999 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:48.999 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.000 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.000 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.000 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.000 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.001 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.001 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.001 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.001 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.002 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.002 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.002 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.003 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.003 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.003 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.004 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.004 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.004 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.004 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.005 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.005 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.005 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.006 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.006 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.006 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.007 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.007 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.007 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.007 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.008 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.008 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.008 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.008 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.009 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.009 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.009 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.010 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.010 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.010 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "stats", 2024-05-23T00:10:49.010 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.010 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.010 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.011 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.011 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.011 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.011 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.011 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.011 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.011 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.011 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.012 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.012 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.012 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.012 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.012 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.012 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.012 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.013 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.013 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.013 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.013 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.013 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.013 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.013 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.013 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.014 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.014 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.014 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.014 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.014 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.014 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.014 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.015 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.015 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.015 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.015 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.015 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.015 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.015 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.015 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.016 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.017 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.017 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.017 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.017 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.017 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.017 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.017 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.017 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.018 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.018 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.018 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.018 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.018 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.018 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.018 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.019 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.019 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.019 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.019 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.019 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.019 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.019 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.019 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.020 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.020 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.020 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.020 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "status", 2024-05-23T00:10:49.020 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.020 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.020 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.021 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.021 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.021 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.021 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.021 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.021 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.021 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.021 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.022 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.022 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.022 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.022 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.022 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.023 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.023 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.023 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.023 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.023 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.023 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.023 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.024 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.024 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.024 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.024 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.024 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.024 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.024 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.024 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.025 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.025 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.025 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.025 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.025 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.025 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.026 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.026 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.026 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.026 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.026 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.026 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.026 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.027 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.027 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.027 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.027 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.027 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.027 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.027 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.028 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.028 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.028 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.028 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.028 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.028 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.028 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.029 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.029 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.029 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.029 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.029 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.029 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.029 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.029 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.030 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.030 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.030 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.030 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.030 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.030 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.031 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "telegraf", 2024-05-23T00:10:49.031 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.031 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.031 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.031 INFO:teuthology.orchestra.run.smithi103.stdout: "address": { 2024-05-23T00:10:49.031 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "address", 2024-05-23T00:10:49.032 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.032 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.032 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.032 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-23T00:10:49.032 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.032 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.032 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.033 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.033 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.033 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.033 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.033 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.033 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.033 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.033 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.034 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.034 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.034 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15", 2024-05-23T00:10:49.034 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.034 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.034 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.034 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.035 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.035 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.035 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.035 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.035 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.035 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.035 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.035 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.036 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.036 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.036 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.036 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.036 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.036 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.036 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.036 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.037 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.037 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.037 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.037 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.037 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.037 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.037 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.037 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.038 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.038 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.038 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.038 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.038 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.038 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.038 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.039 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.039 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.039 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.039 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.039 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.039 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.039 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.039 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.040 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.041 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.042 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.043 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.044 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "telemetry", 2024-05-23T00:10:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_basic": { 2024-05-23T00:10:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_basic", 2024-05-23T00:10:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.045 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-23T00:10:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.046 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_crash": { 2024-05-23T00:10:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_crash", 2024-05-23T00:10:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.047 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-23T00:10:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.048 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_device": { 2024-05-23T00:10:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_device", 2024-05-23T00:10:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.049 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-23T00:10:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_ident": { 2024-05-23T00:10:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_ident", 2024-05-23T00:10:49.050 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.051 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-23T00:10:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_perf": { 2024-05-23T00:10:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_perf", 2024-05-23T00:10:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.052 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-23T00:10:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.053 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "contact": { 2024-05-23T00:10:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "contact", 2024-05-23T00:10:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.054 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.055 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "description": { 2024-05-23T00:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "description", 2024-05-23T00:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.056 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.057 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "device_url": { 2024-05-23T00:10:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_url", 2024-05-23T00:10:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-23T00:10:49.058 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.059 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": { 2024-05-23T00:10:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enabled", 2024-05-23T00:10:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.060 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.061 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "24", 2024-05-23T00:10:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "8", 2024-05-23T00:10:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.062 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "last_opt_revision": { 2024-05-23T00:10:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "last_opt_revision", 2024-05-23T00:10:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.063 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1", 2024-05-23T00:10:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.064 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "leaderboard": { 2024-05-23T00:10:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "leaderboard", 2024-05-23T00:10:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.065 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.066 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "leaderboard_description": { 2024-05-23T00:10:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "leaderboard_description", 2024-05-23T00:10:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.067 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.068 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.069 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.070 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.071 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.072 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.073 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.074 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.075 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.076 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.077 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "organization": { 2024-05-23T00:10:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "organization", 2024-05-23T00:10:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.078 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "proxy": { 2024-05-23T00:10:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "proxy", 2024-05-23T00:10:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.080 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "url": { 2024-05-23T00:10:49.081 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "url", 2024-05-23T00:10:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-23T00:10:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.082 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.083 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "test_orchestrator", 2024-05-23T00:10:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.084 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.085 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.086 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.087 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.089 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.090 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.091 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.092 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.093 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "volumes", 2024-05-23T00:10:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.094 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.095 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.096 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.097 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.098 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.099 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.100 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.101 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.102 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "max_concurrent_clones": { 2024-05-23T00:10:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_concurrent_clones", 2024-05-23T00:10:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.103 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "4", 2024-05-23T00:10:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-23T00:10:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "periodic_async_work": { 2024-05-23T00:10:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "periodic_async_work", 2024-05-23T00:10:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.105 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Periodically check for async work", 2024-05-23T00:10:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.106 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "snapshot_clone_delay": { 2024-05-23T00:10:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snapshot_clone_delay", 2024-05-23T00:10:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.107 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-23T00:10:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.108 INFO:teuthology.orchestra.run.smithi103.stdout: "snapshot_clone_no_wait": { 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snapshot_clone_no_wait", 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.109 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-23T00:10:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.110 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix", 2024-05-23T00:10:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "discovery_interval": { 2024-05-23T00:10:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "discovery_interval", 2024-05-23T00:10:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.111 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "100", 2024-05-23T00:10:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.112 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "identifier": { 2024-05-23T00:10:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "identifier", 2024-05-23T00:10:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.113 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.114 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:49.115 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.117 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.118 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.119 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.120 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.121 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.122 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.123 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.124 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.125 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.126 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_host": { 2024-05-23T00:10:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_host", 2024-05-23T00:10:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.127 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_port": { 2024-05-23T00:10:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_port", 2024-05-23T00:10:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10051", 2024-05-23T00:10:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.129 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.130 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_sender": { 2024-05-23T00:10:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_sender", 2024-05-23T00:10:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-23T00:10:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.131 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.132 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "gid": 25387, 2024-05-23T00:10:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "y", 2024-05-23T00:10:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "mgr_features": 4540701547738038271, 2024-05-23T00:10:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: "available_modules": [ 2024-05-23T00:10:49.133 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "alerts", 2024-05-23T00:10:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.134 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How frequently to reexamine health status", 2024-05-23T00:10:49.135 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.136 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.137 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.138 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.139 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.140 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.141 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.142 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.143 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.144 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_destination": { 2024-05-23T00:10:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_destination", 2024-05-23T00:10:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.145 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Email address to send alerts to", 2024-05-23T00:10:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_from_name": { 2024-05-23T00:10:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_from_name", 2024-05-23T00:10:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.147 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "Ceph", 2024-05-23T00:10:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Email From: name", 2024-05-23T00:10:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.148 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.149 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_host": { 2024-05-23T00:10:49.149 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_host", 2024-05-23T00:10:49.149 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.149 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.149 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.149 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.149 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.149 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.150 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.150 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP server", 2024-05-23T00:10:49.150 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.150 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.150 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.150 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.150 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_password": { 2024-05-23T00:10:49.151 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_password", 2024-05-23T00:10:49.151 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.151 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.151 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.151 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.151 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.151 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.151 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.152 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Password to authenticate with", 2024-05-23T00:10:49.152 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.152 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.152 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.152 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.152 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_port": { 2024-05-23T00:10:49.152 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_port", 2024-05-23T00:10:49.152 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.153 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.153 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.153 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "465", 2024-05-23T00:10:49.153 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.153 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.153 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.153 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP port", 2024-05-23T00:10:49.154 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.154 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.154 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.154 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.154 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_sender": { 2024-05-23T00:10:49.154 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_sender", 2024-05-23T00:10:49.154 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.154 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.155 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.155 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.155 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.155 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.155 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.155 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP envelope sender", 2024-05-23T00:10:49.155 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.156 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.156 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.156 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_ssl": { 2024-05-23T00:10:49.156 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_ssl", 2024-05-23T00:10:49.156 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.156 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.156 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.157 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.157 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.157 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.157 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.157 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-23T00:10:49.157 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.157 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.157 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.158 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_user": { 2024-05-23T00:10:49.158 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_user", 2024-05-23T00:10:49.158 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.158 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.158 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.158 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.159 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.159 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.159 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.159 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "User to authenticate as", 2024-05-23T00:10:49.159 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.159 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.159 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.159 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.160 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.160 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.160 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "balancer", 2024-05-23T00:10:49.160 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.160 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.160 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.161 INFO:teuthology.orchestra.run.smithi103.stdout: "active": { 2024-05-23T00:10:49.161 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "active", 2024-05-23T00:10:49.161 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.161 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.161 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.161 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.161 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.162 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.162 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.162 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "automatically balance PGs across cluster", 2024-05-23T00:10:49.162 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.162 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.162 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.162 INFO:teuthology.orchestra.run.smithi103.stdout: "begin_time": { 2024-05-23T00:10:49.163 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "begin_time", 2024-05-23T00:10:49.163 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.163 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.163 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.163 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0000", 2024-05-23T00:10:49.163 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.163 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "beginning time of day to automatically balance", 2024-05-23T00:10:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-23T00:10:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: "begin_weekday": { 2024-05-23T00:10:49.164 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "begin_weekday", 2024-05-23T00:10:49.165 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.165 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.165 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.165 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.165 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0", 2024-05-23T00:10:49.165 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "6", 2024-05-23T00:10:49.165 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.166 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-23T00:10:49.166 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-23T00:10:49.166 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.166 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.166 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.166 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_max_iterations": { 2024-05-23T00:10:49.166 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_max_iterations", 2024-05-23T00:10:49.167 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.167 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.167 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.167 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "25", 2024-05-23T00:10:49.167 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:49.167 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "250", 2024-05-23T00:10:49.167 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.167 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_metrics": { 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_metrics", 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.168 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.169 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "pgs,objects,bytes", 2024-05-23T00:10:49.169 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.169 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.169 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.169 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-23T00:10:49.169 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-23T00:10:49.169 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.170 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.170 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.170 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_step": { 2024-05-23T00:10:49.170 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_step", 2024-05-23T00:10:49.170 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.170 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.170 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.170 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0.5", 2024-05-23T00:10:49.171 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0.001", 2024-05-23T00:10:49.171 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "0.999", 2024-05-23T00:10:49.171 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.171 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "aggressiveness of optimization", 2024-05-23T00:10:49.171 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-23T00:10:49.171 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.171 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.172 INFO:teuthology.orchestra.run.smithi103.stdout: "end_time": { 2024-05-23T00:10:49.172 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "end_time", 2024-05-23T00:10:49.172 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.172 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.172 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.172 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "2359", 2024-05-23T00:10:49.172 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.172 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.173 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.173 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "ending time of day to automatically balance", 2024-05-23T00:10:49.173 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-23T00:10:49.173 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.173 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.173 INFO:teuthology.orchestra.run.smithi103.stdout: "end_weekday": { 2024-05-23T00:10:49.173 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "end_weekday", 2024-05-23T00:10:49.174 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.174 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.174 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.174 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.174 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0", 2024-05-23T00:10:49.174 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "6", 2024-05-23T00:10:49.174 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.174 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-23T00:10:49.175 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-23T00:10:49.175 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.175 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.175 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.175 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.175 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.175 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.176 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.176 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.176 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.176 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.176 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.176 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.176 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.176 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.177 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.177 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.177 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.177 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.177 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.177 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.177 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.177 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.178 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.178 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.178 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.178 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.178 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.178 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.178 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.179 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.179 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.179 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.179 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.179 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.179 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.179 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.180 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.180 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.180 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.180 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.180 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.180 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.180 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.180 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.181 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.181 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.181 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.181 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.181 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.181 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.181 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.181 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.182 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.182 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.182 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.182 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.182 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.182 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.183 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.184 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.184 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.184 INFO:teuthology.orchestra.run.smithi103.stdout: "min_score": { 2024-05-23T00:10:49.184 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "min_score", 2024-05-23T00:10:49.184 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.184 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.184 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.185 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.185 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.185 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.185 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.185 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-23T00:10:49.185 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.185 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.186 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.186 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": { 2024-05-23T00:10:49.186 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mode", 2024-05-23T00:10:49.186 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.186 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.186 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.187 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "upmap", 2024-05-23T00:10:49.187 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.187 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.187 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.187 INFO:teuthology.orchestra.run.smithi103.stdout: "crush-compat", 2024-05-23T00:10:49.187 INFO:teuthology.orchestra.run.smithi103.stdout: "none", 2024-05-23T00:10:49.187 INFO:teuthology.orchestra.run.smithi103.stdout: "read", 2024-05-23T00:10:49.187 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap", 2024-05-23T00:10:49.188 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap-read" 2024-05-23T00:10:49.188 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.188 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Balancer mode", 2024-05-23T00:10:49.188 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.188 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.188 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.188 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_ids": { 2024-05-23T00:10:49.189 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pool_ids", 2024-05-23T00:10:49.189 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.189 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.189 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.189 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.189 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.189 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.190 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.190 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-23T00:10:49.190 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.190 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.190 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.190 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.190 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:49.190 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:49.191 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.191 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.191 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.191 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:49.191 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.191 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.191 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.192 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-23T00:10:49.192 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.192 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.192 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.192 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.192 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap_max_deviation": { 2024-05-23T00:10:49.192 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "upmap_max_deviation", 2024-05-23T00:10:49.192 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.193 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.193 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.193 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:49.193 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:49.193 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.193 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.193 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-23T00:10:49.193 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-23T00:10:49.194 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.194 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.194 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.194 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap_max_optimizations": { 2024-05-23T00:10:49.194 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "upmap_max_optimizations", 2024-05-23T00:10:49.194 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.194 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.195 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.195 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.195 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.195 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.195 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.195 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-23T00:10:49.195 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.195 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cephadm", 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.196 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.197 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_down_multiplier": { 2024-05-23T00:10:49.197 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_down_multiplier", 2024-05-23T00:10:49.197 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.197 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.197 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.197 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3.0", 2024-05-23T00:10:49.197 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.197 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.198 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.198 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-23T00:10:49.198 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.198 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.198 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.198 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.198 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_refresh_rate": { 2024-05-23T00:10:49.198 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_refresh_rate", 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "20", 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-23T00:10:49.199 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_starting_port": { 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_starting_port", 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.200 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "4721", 2024-05-23T00:10:49.201 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.201 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.201 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.201 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-23T00:10:49.201 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.201 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.201 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.202 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_ptrace": { 2024-05-23T00:10:49.202 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_ptrace", 2024-05-23T00:10:49.202 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.202 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.202 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.202 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.202 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.202 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.203 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.203 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-23T00:10:49.203 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-23T00:10:49.203 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.203 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.203 INFO:teuthology.orchestra.run.smithi103.stdout: "autotune_interval": { 2024-05-23T00:10:49.204 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "autotune_interval", 2024-05-23T00:10:49.204 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.204 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.204 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.204 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:49.204 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.204 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.204 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.205 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-23T00:10:49.205 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.205 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.205 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.205 INFO:teuthology.orchestra.run.smithi103.stdout: "autotune_memory_target_ratio": { 2024-05-23T00:10:49.205 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "autotune_memory_target_ratio", 2024-05-23T00:10:49.205 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.206 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.206 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.206 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0.7", 2024-05-23T00:10:49.206 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.206 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.206 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.206 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-23T00:10:49.206 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.207 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.207 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.207 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.207 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm_log_destination": { 2024-05-23T00:10:49.207 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cephadm_log_destination", 2024-05-23T00:10:49.207 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.207 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.208 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.208 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.208 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.208 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.208 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.208 INFO:teuthology.orchestra.run.smithi103.stdout: "file", 2024-05-23T00:10:49.208 INFO:teuthology.orchestra.run.smithi103.stdout: "file,syslog", 2024-05-23T00:10:49.208 INFO:teuthology.orchestra.run.smithi103.stdout: "syslog" 2024-05-23T00:10:49.209 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.209 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-23T00:10:49.209 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.209 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.209 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.209 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.209 INFO:teuthology.orchestra.run.smithi103.stdout: "cgroups_split": { 2024-05-23T00:10:49.209 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cgroups_split", 2024-05-23T00:10:49.210 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.210 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.210 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.210 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.210 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.210 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.210 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.211 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-23T00:10:49.211 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.211 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.211 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.211 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.211 INFO:teuthology.orchestra.run.smithi103.stdout: "config_checks_enabled": { 2024-05-23T00:10:49.211 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_checks_enabled", 2024-05-23T00:10:49.212 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.212 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.212 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.212 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.212 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.212 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.212 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.212 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-23T00:10:49.213 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.213 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.213 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.213 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.213 INFO:teuthology.orchestra.run.smithi103.stdout: "config_dashboard": { 2024-05-23T00:10:49.213 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_dashboard", 2024-05-23T00:10:49.213 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.213 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.214 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.214 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.214 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.214 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.214 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.214 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-23T00:10:49.214 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.215 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.215 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.215 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_alertmanager": { 2024-05-23T00:10:49.215 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_alertmanager", 2024-05-23T00:10:49.215 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.215 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.215 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.216 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-23T00:10:49.216 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.216 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.216 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.216 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:49.216 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.216 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.216 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.217 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_base": { 2024-05-23T00:10:49.217 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_base", 2024-05-23T00:10:49.217 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.217 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.217 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.217 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-23T00:10:49.217 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.218 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.218 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.218 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Container image name, without the tag", 2024-05-23T00:10:49.218 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.218 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.218 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.218 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_elasticsearch": { 2024-05-23T00:10:49.219 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_elasticsearch", 2024-05-23T00:10:49.219 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.219 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.219 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.219 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-23T00:10:49.219 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.219 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.219 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.220 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "elasticsearch container image", 2024-05-23T00:10:49.220 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.220 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.220 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.220 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_grafana": { 2024-05-23T00:10:49.220 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_grafana", 2024-05-23T00:10:49.220 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.221 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.221 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.221 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-23T00:10:49.221 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.221 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.221 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.221 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:49.221 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_haproxy": { 2024-05-23T00:10:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_haproxy", 2024-05-23T00:10:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.222 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.223 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-23T00:10:49.223 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.223 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.223 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.223 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "HAproxy container image", 2024-05-23T00:10:49.223 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.223 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.224 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.224 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.224 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_agent": { 2024-05-23T00:10:49.224 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_agent", 2024-05-23T00:10:49.224 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.224 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.224 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.224 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-23T00:10:49.225 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.225 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.225 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.225 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger agent container image", 2024-05-23T00:10:49.225 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.225 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.225 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.226 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.226 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_collector": { 2024-05-23T00:10:49.226 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_collector", 2024-05-23T00:10:49.226 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.226 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.226 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.226 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-23T00:10:49.226 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger collector container image", 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_query": { 2024-05-23T00:10:49.227 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_query", 2024-05-23T00:10:49.228 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.228 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.228 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.228 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-23T00:10:49.228 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.228 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.228 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.228 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger query container image", 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_keepalived": { 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_keepalived", 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.229 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.230 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-23T00:10:49.230 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.230 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.230 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.230 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Keepalived container image", 2024-05-23T00:10:49.230 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.230 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.230 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.231 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.231 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_loki": { 2024-05-23T00:10:49.231 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_loki", 2024-05-23T00:10:49.231 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.231 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.231 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.231 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-23T00:10:49.231 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Loki container image", 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_node_exporter": { 2024-05-23T00:10:49.232 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_node_exporter", 2024-05-23T00:10:49.233 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.233 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.233 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.233 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-23T00:10:49.233 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.233 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.233 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.233 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:49.234 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.234 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.234 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.234 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.234 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_nvmeof": { 2024-05-23T00:10:49.234 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_nvmeof", 2024-05-23T00:10:49.234 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.234 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.235 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.235 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-23T00:10:49.235 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.235 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.235 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.235 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Nvme-of container image", 2024-05-23T00:10:49.235 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.236 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.236 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.236 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.236 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_prometheus": { 2024-05-23T00:10:49.236 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_prometheus", 2024-05-23T00:10:49.236 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.236 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.236 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.237 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-23T00:10:49.237 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.237 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.237 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.237 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:49.237 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.237 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.237 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.238 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_promtail": { 2024-05-23T00:10:49.238 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_promtail", 2024-05-23T00:10:49.238 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.238 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.238 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.238 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-23T00:10:49.238 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.239 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.239 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.239 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Promtail container image", 2024-05-23T00:10:49.239 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.239 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.239 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.239 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_samba": { 2024-05-23T00:10:49.240 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_samba", 2024-05-23T00:10:49.240 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.240 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.240 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.240 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-23T00:10:49.240 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.240 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.240 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.241 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Samba/SMB container image", 2024-05-23T00:10:49.241 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.241 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.241 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.241 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_snmp_gateway": { 2024-05-23T00:10:49.241 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_snmp_gateway", 2024-05-23T00:10:49.241 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-23T00:10:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SNMP Gateway container image", 2024-05-23T00:10:49.242 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: "container_init": { 2024-05-23T00:10:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_init", 2024-05-23T00:10:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Run podman/docker with `--init`", 2024-05-23T00:10:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.245 INFO:teuthology.orchestra.run.smithi103.stdout: "daemon_cache_timeout": { 2024-05-23T00:10:49.245 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "daemon_cache_timeout", 2024-05-23T00:10:49.245 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.245 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.245 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.245 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:49.246 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.246 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.246 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.246 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-23T00:10:49.246 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.246 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.246 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.246 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: "default_cephadm_command_timeout": { 2024-05-23T00:10:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "default_cephadm_command_timeout", 2024-05-23T00:10:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "900", 2024-05-23T00:10:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.247 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.248 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.248 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-23T00:10:49.248 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.248 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.248 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.248 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.248 INFO:teuthology.orchestra.run.smithi103.stdout: "default_registry": { 2024-05-23T00:10:49.248 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "default_registry", 2024-05-23T00:10:49.249 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.249 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.249 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.249 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io", 2024-05-23T00:10:49.249 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.249 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.249 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.250 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-23T00:10:49.250 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.250 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.250 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.250 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.250 INFO:teuthology.orchestra.run.smithi103.stdout: "device_cache_timeout": { 2024-05-23T00:10:49.250 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_cache_timeout", 2024-05-23T00:10:49.250 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.251 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.251 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.251 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1800", 2024-05-23T00:10:49.251 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.251 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.251 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.251 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache device inventory", 2024-05-23T00:10:49.251 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.252 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.252 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.252 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.252 INFO:teuthology.orchestra.run.smithi103.stdout: "device_enhanced_scan": { 2024-05-23T00:10:49.252 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_enhanced_scan", 2024-05-23T00:10:49.252 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.252 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-23T00:10:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.253 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.254 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.254 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.254 INFO:teuthology.orchestra.run.smithi103.stdout: "facts_cache_timeout": { 2024-05-23T00:10:49.254 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "facts_cache_timeout", 2024-05-23T00:10:49.254 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.254 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.254 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.254 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:49.255 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.255 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.255 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.255 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache host facts data", 2024-05-23T00:10:49.255 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.255 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.255 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.255 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.256 INFO:teuthology.orchestra.run.smithi103.stdout: "grafana_dashboards_path": { 2024-05-23T00:10:49.256 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "grafana_dashboards_path", 2024-05-23T00:10:49.256 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.256 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.256 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.256 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-23T00:10:49.256 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.257 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.257 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.257 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-23T00:10:49.257 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.257 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.257 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.257 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.257 INFO:teuthology.orchestra.run.smithi103.stdout: "host_check_interval": { 2024-05-23T00:10:49.258 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "host_check_interval", 2024-05-23T00:10:49.258 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.258 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.258 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.258 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:49.258 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.258 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.259 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.259 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to perform a host check", 2024-05-23T00:10:49.259 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.259 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.259 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.259 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.259 INFO:teuthology.orchestra.run.smithi103.stdout: "hw_monitoring": { 2024-05-23T00:10:49.259 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "hw_monitoring", 2024-05-23T00:10:49.260 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.260 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.260 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.260 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.260 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.260 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.260 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.260 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-23T00:10:49.261 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.261 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.261 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.261 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.261 INFO:teuthology.orchestra.run.smithi103.stdout: "inventory_list_all": { 2024-05-23T00:10:49.261 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "inventory_list_all", 2024-05-23T00:10:49.261 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.261 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.262 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.262 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.262 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.262 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.262 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.262 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-23T00:10:49.262 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.262 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.263 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.263 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.263 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.263 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.263 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.263 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.263 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.264 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.265 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.265 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.265 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.265 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.265 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.265 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.265 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.265 INFO:teuthology.orchestra.run.smithi103.stdout: "log_refresh_metadata": { 2024-05-23T00:10:49.266 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_refresh_metadata", 2024-05-23T00:10:49.266 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.266 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.266 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.266 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.266 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.266 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.267 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.267 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-23T00:10:49.267 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.267 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.267 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.267 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.267 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.267 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.268 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.268 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.268 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.268 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.268 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.268 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.268 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.268 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-23T00:10:49.269 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.269 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.269 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.269 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.269 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.269 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.269 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.269 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.270 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.270 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.270 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.270 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.270 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.270 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.270 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.271 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.271 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.271 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.271 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.271 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.271 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.271 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.271 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.272 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.272 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.272 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.272 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.272 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.272 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.272 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.273 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.273 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.273 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.273 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.273 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.273 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.273 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.273 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.274 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.274 INFO:teuthology.orchestra.run.smithi103.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-23T00:10:49.274 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-23T00:10:49.274 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.274 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.274 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.274 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.274 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.275 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.275 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.275 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-23T00:10:49.275 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.275 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.275 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.276 INFO:teuthology.orchestra.run.smithi103.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-23T00:10:49.276 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-23T00:10:49.276 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.276 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.276 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.276 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "*", 2024-05-23T00:10:49.276 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.276 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.277 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.277 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-23T00:10:49.277 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.277 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.277 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.277 INFO:teuthology.orchestra.run.smithi103.stdout: "max_count_per_host": { 2024-05-23T00:10:49.278 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_count_per_host", 2024-05-23T00:10:49.278 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.278 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.278 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.278 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.278 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.278 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.278 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.279 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "max number of daemons per service per host", 2024-05-23T00:10:49.279 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.279 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.279 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.279 INFO:teuthology.orchestra.run.smithi103.stdout: "max_osd_draining_count": { 2024-05-23T00:10:49.279 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_osd_draining_count", 2024-05-23T00:10:49.279 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-23T00:10:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.281 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.281 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.281 INFO:teuthology.orchestra.run.smithi103.stdout: "migration_current": { 2024-05-23T00:10:49.281 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "migration_current", 2024-05-23T00:10:49.281 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.281 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.282 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.282 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.282 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.282 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.282 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.282 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "internal - do not modify", 2024-05-23T00:10:49.282 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.282 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": { 2024-05-23T00:10:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mode", 2024-05-23T00:10:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.283 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "root", 2024-05-23T00:10:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm-package", 2024-05-23T00:10:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: "root" 2024-05-23T00:10:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "mode for remote execution of cephadm", 2024-05-23T00:10:49.284 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: "oob_default_addr": { 2024-05-23T00:10:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "oob_default_addr", 2024-05-23T00:10:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.285 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "169.254.1.1", 2024-05-23T00:10:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-23T00:10:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.286 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: "prometheus_alerts_path": { 2024-05-23T00:10:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prometheus_alerts_path", 2024-05-23T00:10:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.287 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-23T00:10:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-23T00:10:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_insecure": { 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_insecure", 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.289 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-23T00:10:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_password": { 2024-05-23T00:10:49.290 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_password", 2024-05-23T00:10:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.291 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-23T00:10:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_url": { 2024-05-23T00:10:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_url", 2024-05-23T00:10:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.292 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-23T00:10:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.293 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_username": { 2024-05-23T00:10:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_username", 2024-05-23T00:10:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.294 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.296 INFO:teuthology.orchestra.run.smithi103.stdout: "secure_monitoring_stack": { 2024-05-23T00:10:49.296 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "secure_monitoring_stack", 2024-05-23T00:10:49.296 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.296 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.296 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.296 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.296 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.296 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-23T00:10:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: "service_discovery_port": { 2024-05-23T00:10:49.297 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "service_discovery_port", 2024-05-23T00:10:49.298 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.298 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.298 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.298 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8765", 2024-05-23T00:10:49.298 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.298 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.298 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.298 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "cephadm service discovery port", 2024-05-23T00:10:49.299 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.299 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.299 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.299 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.299 INFO:teuthology.orchestra.run.smithi103.stdout: "ssh_config_file": { 2024-05-23T00:10:49.299 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssh_config_file", 2024-05-23T00:10:49.299 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.300 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.301 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.301 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.301 INFO:teuthology.orchestra.run.smithi103.stdout: "use_agent": { 2024-05-23T00:10:49.301 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "use_agent", 2024-05-23T00:10:49.301 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.301 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.301 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.301 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.302 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.302 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.302 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.302 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-23T00:10:49.302 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.302 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.302 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.303 INFO:teuthology.orchestra.run.smithi103.stdout: "use_repo_digest": { 2024-05-23T00:10:49.303 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "use_repo_digest", 2024-05-23T00:10:49.303 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.303 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.303 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.303 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.303 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.303 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-23T00:10:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_failed_host_check": { 2024-05-23T00:10:49.304 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_failed_host_check", 2024-05-23T00:10:49.305 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.305 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.305 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.305 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.305 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.305 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.305 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.305 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if the host check fails", 2024-05-23T00:10:49.306 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.306 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.306 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.306 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.306 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_stray_daemons": { 2024-05-23T00:10:49.306 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_stray_daemons", 2024-05-23T00:10:49.306 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.306 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.307 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.307 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.307 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.307 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.307 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.307 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-23T00:10:49.307 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.307 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.308 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.308 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.308 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_stray_hosts": { 2024-05-23T00:10:49.308 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_stray_hosts", 2024-05-23T00:10:49.308 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.308 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.308 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.308 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.309 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.309 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.309 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.309 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-23T00:10:49.309 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.309 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.309 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.309 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crash", 2024-05-23T00:10:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.310 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.311 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.311 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.311 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.311 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.311 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.311 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.311 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.311 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.312 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.313 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.313 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.313 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.313 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.313 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.313 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.313 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.314 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.314 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.314 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.314 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.314 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.314 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.314 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.315 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.315 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.315 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.315 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.315 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.315 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.315 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.315 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.316 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.316 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.316 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.316 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.316 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.316 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.316 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.316 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.317 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.318 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.318 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.318 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.318 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.318 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.318 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.319 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.319 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.321 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.321 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.322 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.322 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.322 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.322 INFO:teuthology.orchestra.run.smithi103.stdout: "retain_interval": { 2024-05-23T00:10:49.322 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "retain_interval", 2024-05-23T00:10:49.322 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.322 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.323 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.323 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "31536000", 2024-05-23T00:10:49.323 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.323 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.323 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.323 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_recent_interval": { 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_recent_interval", 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.324 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.325 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1209600", 2024-05-23T00:10:49.325 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.325 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.325 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.325 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-23T00:10:49.325 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.325 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.325 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.326 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.326 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.326 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.326 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.326 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "dashboard", 2024-05-23T00:10:49.326 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.326 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.327 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.327 INFO:teuthology.orchestra.run.smithi103.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-23T00:10:49.327 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-23T00:10:49.327 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.327 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.327 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.327 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.327 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.328 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.328 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.328 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.328 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.328 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.328 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.328 INFO:teuthology.orchestra.run.smithi103.stdout: "ALERTMANAGER_API_HOST": { 2024-05-23T00:10:49.329 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-23T00:10:49.329 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.329 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.329 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.329 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.329 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.329 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.329 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.330 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.330 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.330 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.330 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.330 INFO:teuthology.orchestra.run.smithi103.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-23T00:10:49.330 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-23T00:10:49.330 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.331 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.331 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.331 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.331 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.331 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.331 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.331 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.332 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.332 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.332 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.332 INFO:teuthology.orchestra.run.smithi103.stdout: "AUDIT_API_ENABLED": { 2024-05-23T00:10:49.332 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "AUDIT_API_ENABLED", 2024-05-23T00:10:49.332 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.332 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.333 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.334 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.334 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.334 INFO:teuthology.orchestra.run.smithi103.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-23T00:10:49.334 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-23T00:10:49.334 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.334 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.334 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.335 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: "ENABLE_BROWSABLE_API": { 2024-05-23T00:10:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-23T00:10:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.336 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.337 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.337 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.337 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.337 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.337 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.337 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.337 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-23T00:10:49.338 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-23T00:10:49.338 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.338 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.338 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.338 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.338 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.338 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.338 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.339 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.339 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.339 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.339 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.339 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-23T00:10:49.339 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-23T00:10:49.340 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.340 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.340 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.340 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.340 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.340 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.340 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.340 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.341 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.341 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.341 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.341 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.341 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-23T00:10:49.341 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-23T00:10:49.341 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.342 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.342 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.342 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.342 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.342 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.342 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.342 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.343 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.343 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.343 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.343 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-23T00:10:49.343 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-23T00:10:49.343 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.344 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.344 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.344 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.344 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.344 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.344 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.344 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.344 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.345 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.345 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.345 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-23T00:10:49.345 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-23T00:10:49.345 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.345 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.345 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.346 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.346 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.346 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.346 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.346 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.346 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.346 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-23T00:10:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-23T00:10:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.347 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.348 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-23T00:10:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-23T00:10:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.349 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.350 INFO:teuthology.orchestra.run.smithi103.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-23T00:10:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-23T00:10:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.351 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.352 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.352 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.352 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.352 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.352 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.352 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_PASSWORD": { 2024-05-23T00:10:49.352 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-23T00:10:49.353 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.353 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.353 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.353 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:49.353 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.353 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.353 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.353 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.354 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.354 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.354 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.354 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.354 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-23T00:10:49.354 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-23T00:10:49.354 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.354 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.355 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.355 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.355 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.355 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.355 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.355 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.355 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.356 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.356 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.356 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_URL": { 2024-05-23T00:10:49.356 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_URL", 2024-05-23T00:10:49.356 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.356 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.356 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.357 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.357 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.357 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.357 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.357 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.357 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.357 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.357 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_USERNAME": { 2024-05-23T00:10:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-23T00:10:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.358 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:49.359 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.359 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.359 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.359 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.359 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.359 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.359 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.359 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-23T00:10:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-23T00:10:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.360 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.361 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.361 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.361 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.361 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.361 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.361 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-23T00:10:49.362 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-23T00:10:49.362 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.362 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.362 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.362 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.362 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.362 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.362 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.363 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.363 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.363 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.363 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.363 INFO:teuthology.orchestra.run.smithi103.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-23T00:10:49.363 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-23T00:10:49.363 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.364 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.364 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.364 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.364 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.364 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.364 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.364 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-23T00:10:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-23T00:10:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.365 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.366 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.366 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.366 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.366 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.366 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.366 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.366 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.367 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.367 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.367 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.367 INFO:teuthology.orchestra.run.smithi103.stdout: "PROMETHEUS_API_HOST": { 2024-05-23T00:10:49.367 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-23T00:10:49.367 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.367 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.367 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.368 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.368 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.368 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.368 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.368 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.368 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.368 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.368 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.369 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.369 INFO:teuthology.orchestra.run.smithi103.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-23T00:10:49.369 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-23T00:10:49.369 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.369 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.369 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.369 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.369 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.370 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-23T00:10:49.371 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-23T00:10:49.371 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.371 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.371 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.371 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.371 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.371 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.371 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-23T00:10:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-23T00:10:49.372 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.373 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.373 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.373 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.373 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.373 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.373 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.373 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.374 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.374 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.374 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.374 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.374 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-23T00:10:49.374 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-23T00:10:49.374 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.374 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.375 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.375 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.375 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.375 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.375 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.375 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.375 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.375 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-23T00:10:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-23T00:10:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.376 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.377 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.377 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.377 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.377 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.377 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.377 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.377 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.377 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.378 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-23T00:10:49.378 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-23T00:10:49.378 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.378 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.378 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.378 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.378 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.379 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.379 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.379 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.379 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.379 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.379 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.379 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-23T00:10:49.380 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-23T00:10:49.380 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.380 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.380 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.380 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.380 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.380 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.380 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.381 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.381 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.381 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.381 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.381 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-23T00:10:49.381 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-23T00:10:49.381 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.382 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.382 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.382 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.382 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.382 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.382 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.382 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.382 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.383 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.383 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.383 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_ENABLED": { 2024-05-23T00:10:49.383 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-23T00:10:49.383 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.383 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.383 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.384 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.384 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.384 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.384 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.384 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.384 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.384 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.385 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.385 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-23T00:10:49.385 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-23T00:10:49.385 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.385 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.385 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.385 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-23T00:10:49.386 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.386 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.386 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.386 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.386 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.386 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.386 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.387 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.387 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-23T00:10:49.387 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-23T00:10:49.387 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.387 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.387 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.387 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.387 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.388 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-23T00:10:49.389 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-23T00:10:49.389 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.389 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.389 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.389 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8", 2024-05-23T00:10:49.389 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.389 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.389 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.390 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.390 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.390 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.390 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.390 INFO:teuthology.orchestra.run.smithi103.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-23T00:10:49.390 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-23T00:10:49.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.391 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.391 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "45", 2024-05-23T00:10:49.391 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.391 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.391 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.391 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.391 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.392 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.392 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.392 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_ACCESS_KEY": { 2024-05-23T00:10:49.392 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-23T00:10:49.392 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.392 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.392 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.393 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.393 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.393 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.393 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.393 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.393 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.393 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-23T00:10:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-23T00:10:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.394 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:49.395 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.395 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.395 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.395 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.395 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.395 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.395 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.396 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_SECRET_KEY": { 2024-05-23T00:10:49.396 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-23T00:10:49.396 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.396 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.396 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.396 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.396 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.396 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.397 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.397 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.397 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.397 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.397 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.397 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_SSL_VERIFY": { 2024-05-23T00:10:49.397 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-23T00:10:49.398 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.398 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.398 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.398 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.398 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.398 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.398 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.399 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.399 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.399 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.399 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.399 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-23T00:10:49.399 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-23T00:10:49.399 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.400 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.400 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.400 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.400 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.400 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.400 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.400 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.400 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.401 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.401 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.401 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-23T00:10:49.401 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-23T00:10:49.401 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.401 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.401 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.402 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.402 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.402 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.402 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.402 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.402 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.402 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.403 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.403 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-23T00:10:49.403 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-23T00:10:49.403 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.403 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.403 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.403 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.405 INFO:teuthology.orchestra.run.smithi103.stdout: "cross_origin_url": { 2024-05-23T00:10:49.405 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cross_origin_url", 2024-05-23T00:10:49.405 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.405 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.405 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.405 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.405 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.405 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.406 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.406 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.406 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.406 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.406 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.406 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.406 INFO:teuthology.orchestra.run.smithi103.stdout: "crt_file": { 2024-05-23T00:10:49.406 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crt_file", 2024-05-23T00:10:49.407 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.407 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.407 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.407 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.407 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.407 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.407 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.408 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.408 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.408 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.408 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.408 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.408 INFO:teuthology.orchestra.run.smithi103.stdout: "debug": { 2024-05-23T00:10:49.408 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "debug", 2024-05-23T00:10:49.408 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.409 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.409 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.409 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.409 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.409 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.409 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.409 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable/disable debug options", 2024-05-23T00:10:49.409 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.410 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.410 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.410 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4480a2d00 msgr2=0x7fc4480a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:49.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4480a2d00 0x7fc4480a51c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc4641071c0 tx=0x7fc44c005f90 comp rx=0 tx=0).stop 2024-05-23T00:10:49.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 msgr2=0x7fc46419e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:10:49.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 0x7fc46419e530 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc454006010 tx=0x7fc454004040 comp rx=0 tx=0).stop 2024-05-23T00:10:49.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 shutdown_connections 2024-05-23T00:10:49.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fc4480a2d00 0x7fc4480a51c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:49.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4641a2df0 0x7fc4641a32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:49.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc46410cfa0 0x7fc46419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:49.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 --2- 172.21.15.103:0/2391553977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc464107ee0 0x7fc46419e530 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:10:49.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.323+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 >> 172.21.15.103:0/2391553977 conn(0x7fc4640773f0 msgr2=0x7fc464079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:10:49.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.324+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 shutdown_connections 2024-05-23T00:10:49.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:10:48.324+0000 7fc46a600640 1 -- 172.21.15.103:0/2391553977 wait complete. 2024-05-23T00:10:49.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:48 smithi174 ceph-mon[36179]: pgmap v320: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:49.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:48 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1829128100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:10:49.414 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:48 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2391553977' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-23T00:10:49.414 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:48 smithi105 ceph-mon[36426]: pgmap v320: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:49.414 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:48 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1829128100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:10:49.415 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:48 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2391553977' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-23T00:10:49.417 INFO:teuthology.orchestra.run.smithi103.stdout: "jwt_token_ttl": { 2024-05-23T00:10:49.417 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "jwt_token_ttl", 2024-05-23T00:10:49.417 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.417 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.417 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.418 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "28800", 2024-05-23T00:10:49.418 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.418 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.418 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.418 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.418 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.418 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.419 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.419 INFO:teuthology.orchestra.run.smithi103.stdout: "key_file": { 2024-05-23T00:10:49.419 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "key_file", 2024-05-23T00:10:49.419 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.419 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.419 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.419 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.420 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.420 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.420 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.420 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.420 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.420 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.420 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.421 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.421 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.421 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.421 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.421 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.421 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.421 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.421 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.422 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.422 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.422 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.422 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.422 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.422 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.422 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.423 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.424 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.425 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.425 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.425 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.425 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.425 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.425 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.426 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.426 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.426 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.426 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.426 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.426 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.426 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.426 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.427 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.427 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.427 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.427 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.427 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.427 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.427 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.428 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.428 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.428 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.428 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.428 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.428 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.428 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.429 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.429 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.429 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.429 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.429 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.429 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.429 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.429 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.430 INFO:teuthology.orchestra.run.smithi103.stdout: "motd": { 2024-05-23T00:10:49.430 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "motd", 2024-05-23T00:10:49.430 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.430 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.430 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.430 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.430 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.431 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.431 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.431 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "The message of the day", 2024-05-23T00:10:49.431 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.431 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.431 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.431 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.431 INFO:teuthology.orchestra.run.smithi103.stdout: "redirect_resolve_ip_addr": { 2024-05-23T00:10:49.432 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "redirect_resolve_ip_addr", 2024-05-23T00:10:49.432 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.432 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.432 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.432 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.432 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.432 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.433 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.433 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.433 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.433 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.433 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.433 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:49.433 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:49.434 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.434 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.434 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.434 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "::", 2024-05-23T00:10:49.434 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.434 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.434 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.435 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.435 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.435 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.435 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.435 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.435 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:49.435 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:49.435 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.436 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.436 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.436 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8080", 2024-05-23T00:10:49.436 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.436 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.436 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.436 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.437 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.437 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.437 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.437 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl": { 2024-05-23T00:10:49.437 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl", 2024-05-23T00:10:49.437 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.437 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.438 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl_server_port": { 2024-05-23T00:10:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl_server_port", 2024-05-23T00:10:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.439 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8443", 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.440 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.441 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_behaviour": { 2024-05-23T00:10:49.441 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_behaviour", 2024-05-23T00:10:49.441 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.441 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.441 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.441 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "redirect", 2024-05-23T00:10:49.441 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.442 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.442 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.442 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.442 INFO:teuthology.orchestra.run.smithi103.stdout: "redirect" 2024-05-23T00:10:49.442 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.442 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.442 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.443 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.443 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.443 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_error_status_code": { 2024-05-23T00:10:49.443 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_error_status_code", 2024-05-23T00:10:49.443 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.443 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.443 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.444 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "500", 2024-05-23T00:10:49.444 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "400", 2024-05-23T00:10:49.444 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "599", 2024-05-23T00:10:49.444 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.444 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.444 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.444 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.444 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.445 INFO:teuthology.orchestra.run.smithi103.stdout: "url_prefix": { 2024-05-23T00:10:49.445 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "url_prefix", 2024-05-23T00:10:49.445 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.445 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.445 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.445 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.445 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.446 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.446 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.446 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.446 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.446 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.446 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.446 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.447 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.447 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "devicehealth", 2024-05-23T00:10:49.447 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.447 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.447 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.448 INFO:teuthology.orchestra.run.smithi103.stdout: "enable_monitoring": { 2024-05-23T00:10:49.448 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enable_monitoring", 2024-05-23T00:10:49.448 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.448 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.448 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.448 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.448 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.448 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.449 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.449 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "monitor device health metrics", 2024-05-23T00:10:49.449 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.449 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.449 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.449 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.449 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.449 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.450 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.450 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.450 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.450 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.450 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.450 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.450 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.450 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.451 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.451 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.451 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.451 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.451 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.451 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.451 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.451 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.452 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.452 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.452 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.452 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.452 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.452 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.453 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.453 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.453 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.453 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.453 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.453 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.453 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.453 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.454 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.455 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.455 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.455 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.455 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.455 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.455 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.455 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.455 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.456 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.456 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.456 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.456 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.456 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.456 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.456 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.457 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.457 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.457 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.457 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.457 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.457 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.457 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.457 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: "mark_out_threshold": { 2024-05-23T00:10:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mark_out_threshold", 2024-05-23T00:10:49.458 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.459 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.459 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.459 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "2419200", 2024-05-23T00:10:49.459 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.459 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.459 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.459 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-23T00:10:49.459 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.460 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.460 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.460 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.460 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": { 2024-05-23T00:10:49.460 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pool_name", 2024-05-23T00:10:49.460 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.460 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.461 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.461 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "device_health_metrics", 2024-05-23T00:10:49.461 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.461 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.461 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.461 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-23T00:10:49.461 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.461 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: "retention_period": { 2024-05-23T00:10:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "retention_period", 2024-05-23T00:10:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.462 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15552000", 2024-05-23T00:10:49.463 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.463 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.463 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.463 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long to retain device health metrics", 2024-05-23T00:10:49.463 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.463 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.463 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: "scrape_frequency": { 2024-05-23T00:10:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "scrape_frequency", 2024-05-23T00:10:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "86400", 2024-05-23T00:10:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.464 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.465 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.465 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-23T00:10:49.465 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.465 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.465 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.465 INFO:teuthology.orchestra.run.smithi103.stdout: "self_heal": { 2024-05-23T00:10:49.465 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "self_heal", 2024-05-23T00:10:49.466 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.466 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.466 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.466 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.466 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.466 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.466 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.467 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-23T00:10:49.467 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.467 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.467 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.467 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.467 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:49.467 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:49.468 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.468 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.468 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.468 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:49.468 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.468 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.468 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.468 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to wake up and check device health", 2024-05-23T00:10:49.469 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.469 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.469 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.469 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.469 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_threshold": { 2024-05-23T00:10:49.469 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_threshold", 2024-05-23T00:10:49.469 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.469 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.470 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.470 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "7257600", 2024-05-23T00:10:49.470 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.470 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.470 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.470 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-23T00:10:49.470 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.470 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.471 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.471 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.471 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.471 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.471 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "diskprediction_local", 2024-05-23T00:10:49.471 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.471 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.472 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.472 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.472 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.472 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.472 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.472 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.472 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.473 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.473 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.473 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.473 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.473 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.473 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.473 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.473 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.474 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.475 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.475 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.475 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.475 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.475 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.475 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.475 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.475 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.476 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.476 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.476 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.476 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.476 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.476 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.477 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.477 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.477 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.477 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.477 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.477 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.477 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.477 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.478 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.479 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.480 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.480 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.480 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.480 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.480 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.480 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.480 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.481 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.481 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.481 INFO:teuthology.orchestra.run.smithi103.stdout: "predict_interval": { 2024-05-23T00:10:49.481 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "predict_interval", 2024-05-23T00:10:49.481 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.481 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.481 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.481 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "86400", 2024-05-23T00:10:49.482 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.482 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.482 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.482 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.482 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.482 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.482 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.483 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.483 INFO:teuthology.orchestra.run.smithi103.stdout: "predictor_model": { 2024-05-23T00:10:49.483 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "predictor_model", 2024-05-23T00:10:49.483 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.483 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.483 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.483 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "prophetstor", 2024-05-23T00:10:49.483 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.484 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.484 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.484 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.484 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.484 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.484 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.484 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:49.485 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:49.485 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.485 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.485 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.485 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:49.485 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.485 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.486 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.486 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.486 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.486 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.486 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.486 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.486 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.487 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.487 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "influx", 2024-05-23T00:10:49.487 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": false, 2024-05-23T00:10:49.487 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "influxdb python module not found", 2024-05-23T00:10:49.487 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.487 INFO:teuthology.orchestra.run.smithi103.stdout: "batch_size": { 2024-05-23T00:10:49.487 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "batch_size", 2024-05-23T00:10:49.487 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5000", 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.488 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.489 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.489 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.489 INFO:teuthology.orchestra.run.smithi103.stdout: "database": { 2024-05-23T00:10:49.489 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "database", 2024-05-23T00:10:49.489 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.489 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.489 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.490 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "ceph", 2024-05-23T00:10:49.490 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.490 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.490 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.490 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-23T00:10:49.490 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.490 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.491 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.491 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.491 INFO:teuthology.orchestra.run.smithi103.stdout: "hostname": { 2024-05-23T00:10:49.491 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "hostname", 2024-05-23T00:10:49.491 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.491 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.491 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.491 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.492 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.492 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.492 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.492 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB server hostname", 2024-05-23T00:10:49.492 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.492 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.492 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.493 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.493 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.493 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.493 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.493 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.493 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "30", 2024-05-23T00:10:49.493 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "5", 2024-05-23T00:10:49.493 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.494 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.494 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-23T00:10:49.494 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.494 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.494 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.494 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.494 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.495 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.495 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.495 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.495 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.495 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.495 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.495 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.495 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.496 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.497 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.497 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.497 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.497 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.497 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.497 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.497 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.498 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.498 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.498 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.498 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.498 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.498 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.498 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.498 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.499 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.499 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.499 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.499 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.499 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.499 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.500 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.500 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.500 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.500 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.500 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.500 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.500 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.500 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.501 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.501 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.501 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.501 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.501 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.501 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.501 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.502 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.502 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.502 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.502 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.502 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.502 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.502 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.502 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.503 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.503 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.503 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.503 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.503 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.503 INFO:teuthology.orchestra.run.smithi103.stdout: "password": { 2024-05-23T00:10:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "password", 2024-05-23T00:10:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.504 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.505 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "password of InfluxDB server user", 2024-05-23T00:10:49.505 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.505 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.505 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.505 INFO:teuthology.orchestra.run.smithi103.stdout: "port": { 2024-05-23T00:10:49.505 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "port", 2024-05-23T00:10:49.505 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.506 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.506 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.506 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8086", 2024-05-23T00:10:49.506 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.506 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.506 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.506 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB server port", 2024-05-23T00:10:49.506 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.507 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.507 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.507 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl": { 2024-05-23T00:10:49.507 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl", 2024-05-23T00:10:49.507 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.507 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.507 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.508 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "false", 2024-05-23T00:10:49.508 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.508 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.508 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.508 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-23T00:10:49.508 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.508 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.509 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.509 INFO:teuthology.orchestra.run.smithi103.stdout: "threads": { 2024-05-23T00:10:49.509 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "threads", 2024-05-23T00:10:49.509 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.509 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.509 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.509 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:49.510 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:49.510 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "32", 2024-05-23T00:10:49.510 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.510 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-23T00:10:49.510 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.510 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.510 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.510 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.511 INFO:teuthology.orchestra.run.smithi103.stdout: "username": { 2024-05-23T00:10:49.511 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "username", 2024-05-23T00:10:49.511 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.511 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.511 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.511 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.511 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.511 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "username of InfluxDB server user", 2024-05-23T00:10:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: "verify_ssl": { 2024-05-23T00:10:49.512 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "verify_ssl", 2024-05-23T00:10:49.513 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.513 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.513 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.513 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "true", 2024-05-23T00:10:49.513 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.513 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.513 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.513 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-23T00:10:49.514 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.514 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.514 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.514 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.514 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.514 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.515 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "insights", 2024-05-23T00:10:49.515 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.515 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.515 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.515 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.515 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.515 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.515 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.516 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.517 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.518 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.520 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.521 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.522 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.523 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.525 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "iostat", 2024-05-23T00:10:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.526 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.527 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.528 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.529 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.530 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.531 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.532 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.533 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.534 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.535 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "k8sevents", 2024-05-23T00:10:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.536 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_event_retention_days": { 2024-05-23T00:10:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ceph_event_retention_days", 2024-05-23T00:10:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "7", 2024-05-23T00:10:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.537 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-23T00:10:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.538 INFO:teuthology.orchestra.run.smithi103.stdout: "config_check_secs": { 2024-05-23T00:10:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_check_secs", 2024-05-23T00:10:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "10", 2024-05-23T00:10:49.539 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-23T00:10:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.540 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.541 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.542 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.543 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.544 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.545 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.546 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.547 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.548 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "localpool", 2024-05-23T00:10:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "failure_domain": { 2024-05-23T00:10:49.550 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "failure_domain", 2024-05-23T00:10:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "host", 2024-05-23T00:10:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.551 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "failure domain for any created local pool", 2024-05-23T00:10:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-23T00:10:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.552 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.553 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.554 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.555 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.556 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.557 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.558 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.559 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.560 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": { 2024-05-23T00:10:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "min_size", 2024-05-23T00:10:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.561 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default min_size for any created local pool", 2024-05-23T00:10:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-23T00:10:49.562 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "num_rep": { 2024-05-23T00:10:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "num_rep", 2024-05-23T00:10:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.563 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3", 2024-05-23T00:10:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default replica count for any created local pool", 2024-05-23T00:10:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": { 2024-05-23T00:10:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pg_num", 2024-05-23T00:10:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "128", 2024-05-23T00:10:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.565 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default pg_num for any created local pool", 2024-05-23T00:10:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.566 INFO:teuthology.orchestra.run.smithi103.stdout: "prefix": { 2024-05-23T00:10:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prefix", 2024-05-23T00:10:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.567 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "name prefix for any created local pool", 2024-05-23T00:10:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "subtree": { 2024-05-23T00:10:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "subtree", 2024-05-23T00:10:49.568 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "rack", 2024-05-23T00:10:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-23T00:10:49.569 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-23T00:10:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mds_autoscaler", 2024-05-23T00:10:49.570 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.571 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.572 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.573 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.574 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.576 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.577 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.578 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.579 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.580 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mirroring", 2024-05-23T00:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.581 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.582 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.583 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.584 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.585 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.586 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.587 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.588 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "nfs", 2024-05-23T00:10:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.590 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.591 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.592 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.593 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.594 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.595 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.596 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.597 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.598 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "orchestrator", 2024-05-23T00:10:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "fail_fs": { 2024-05-23T00:10:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "fail_fs", 2024-05-23T00:10:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.600 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-23T00:10:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.601 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.602 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.602 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.602 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.602 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.602 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.602 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.602 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.603 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.603 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.603 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.603 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.603 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.603 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.603 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.603 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.604 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.604 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.604 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.604 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.604 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.604 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.604 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.605 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.605 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.605 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.605 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.605 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.605 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.605 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.605 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.606 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.606 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.606 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.606 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.606 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.606 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.607 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.607 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.607 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.607 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.607 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.607 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.607 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.608 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.608 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.608 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.608 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.608 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.608 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.608 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.608 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.609 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.609 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.609 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.609 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.609 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.609 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.609 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.610 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.610 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.610 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.610 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.610 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.610 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.610 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.610 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.611 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.611 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.611 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.611 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.611 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.611 INFO:teuthology.orchestra.run.smithi103.stdout: "orchestrator": { 2024-05-23T00:10:49.611 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "orchestrator", 2024-05-23T00:10:49.611 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.612 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.612 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.612 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.612 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.612 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.612 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.612 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm", 2024-05-23T00:10:49.613 INFO:teuthology.orchestra.run.smithi103.stdout: "rook", 2024-05-23T00:10:49.613 INFO:teuthology.orchestra.run.smithi103.stdout: "test_orchestrator" 2024-05-23T00:10:49.613 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.613 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Orchestrator backend", 2024-05-23T00:10:49.613 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.613 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.613 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.613 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.614 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.614 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.614 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.614 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "osd_perf_query", 2024-05-23T00:10:49.614 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.614 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.614 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.614 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.615 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.615 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.615 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.615 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.615 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.615 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.615 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.616 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.616 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.616 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.616 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.616 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.616 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.616 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.617 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.617 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.617 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.617 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.617 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.617 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.617 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.617 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.618 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.618 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.618 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.618 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.618 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.618 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.618 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.619 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.619 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.619 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.619 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.619 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.619 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.619 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.619 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.620 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.620 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.620 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.620 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.620 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.620 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.620 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.620 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.621 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.621 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.621 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.621 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.621 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.621 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.621 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.621 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.622 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.622 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.622 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.622 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.622 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.622 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.623 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.623 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.623 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.623 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.623 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.623 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.623 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.623 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.624 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.624 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.624 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.624 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "osd_support", 2024-05-23T00:10:49.624 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.624 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.624 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.625 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.625 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.625 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.625 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.625 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.625 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.625 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.625 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.626 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.626 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.626 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.626 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.626 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.626 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.626 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.627 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.627 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.627 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.627 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.627 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.627 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.627 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.628 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.628 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.628 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.628 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.628 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.628 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.628 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.629 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.629 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.629 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.629 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.629 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.629 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.629 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.630 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.630 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.630 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.630 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.630 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.630 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.630 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.630 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.631 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.631 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.631 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.631 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.631 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.631 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.631 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.631 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.632 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.632 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.632 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.632 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.632 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.632 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.632 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.633 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.633 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.633 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.633 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.633 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.633 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.633 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.634 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.634 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.634 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.634 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pg_autoscaler", 2024-05-23T00:10:49.634 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.634 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.634 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.635 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.635 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.635 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.635 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.635 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.635 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.635 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.636 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.636 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.636 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.636 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.636 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.636 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.636 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.636 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.637 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.637 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.637 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.637 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.637 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.637 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.637 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.637 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.638 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.638 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.638 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.638 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.638 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.638 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.638 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.639 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.639 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.639 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.639 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.639 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.639 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.639 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.640 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.640 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.640 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.640 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.640 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.640 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.640 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.640 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.641 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.641 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.641 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.641 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.641 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.641 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.641 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.642 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.642 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.642 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.642 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.642 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.642 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.643 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.643 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.643 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.643 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.643 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.643 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.643 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.643 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.644 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.644 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.644 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:49.644 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:49.644 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.644 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.644 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.644 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:49.645 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.645 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.645 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.645 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.645 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.645 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.645 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.646 INFO:teuthology.orchestra.run.smithi103.stdout: "threshold": { 2024-05-23T00:10:49.646 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "threshold", 2024-05-23T00:10:49.646 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.646 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.646 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.646 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3.0", 2024-05-23T00:10:49.646 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1.0", 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "scaling threshold", 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.648 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.648 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "progress", 2024-05-23T00:10:49.648 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.648 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.648 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.648 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_pg_recovery_event": { 2024-05-23T00:10:49.648 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_pg_recovery_event", 2024-05-23T00:10:49.648 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.649 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.649 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.649 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.649 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.649 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.649 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.649 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-23T00:10:49.650 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.650 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.650 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.650 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": { 2024-05-23T00:10:49.650 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enabled", 2024-05-23T00:10:49.650 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.650 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.651 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.652 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.653 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.653 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.653 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.653 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.653 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.653 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.653 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.653 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.654 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.654 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.654 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.654 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.654 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.654 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.654 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.655 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.655 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.655 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.655 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.655 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.655 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.655 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.656 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.656 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.656 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.656 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.656 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.656 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.656 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.656 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.657 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.657 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.657 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.657 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.657 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.657 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.657 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.657 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.658 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.659 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.659 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.659 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.659 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.659 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.659 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.659 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.659 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.660 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.660 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.660 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.660 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.660 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.660 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.660 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.660 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.661 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.661 INFO:teuthology.orchestra.run.smithi103.stdout: "max_completed_events": { 2024-05-23T00:10:49.661 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_completed_events", 2024-05-23T00:10:49.661 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.661 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.661 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.661 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "50", 2024-05-23T00:10:49.662 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.662 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.662 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.662 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "number of past completed events to remember", 2024-05-23T00:10:49.662 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.662 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.662 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.662 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.663 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:49.663 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:49.663 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.663 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.663 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.663 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:49.663 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.663 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.664 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.664 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long the module is going to sleep", 2024-05-23T00:10:49.664 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.664 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.664 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.664 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.664 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.665 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.665 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prometheus", 2024-05-23T00:10:49.665 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.665 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.665 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.665 INFO:teuthology.orchestra.run.smithi103.stdout: "cache": { 2024-05-23T00:10:49.665 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cache", 2024-05-23T00:10:49.666 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.666 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.666 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.666 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.666 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.666 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.666 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.667 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.667 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.667 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.667 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.667 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.667 INFO:teuthology.orchestra.run.smithi103.stdout: "exclude_perf_counters": { 2024-05-23T00:10:49.667 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "exclude_perf_counters", 2024-05-23T00:10:49.667 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.668 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.668 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.668 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.668 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.668 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.668 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.668 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-23T00:10:49.668 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-23T00:10:49.669 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.669 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.669 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.669 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.669 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.669 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.670 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.670 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.670 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.670 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.670 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.670 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.670 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.670 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.671 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.671 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.671 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.671 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.671 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.671 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.671 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.671 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.672 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.672 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.672 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.672 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.672 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.672 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.672 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.672 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.673 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.673 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.673 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.673 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.673 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.673 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.674 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.674 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.674 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.674 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.674 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.674 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.674 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.675 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.675 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.675 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.675 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.675 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.675 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.675 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.675 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.676 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.676 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.676 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.676 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.676 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.676 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.676 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.676 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.677 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.677 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.677 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.677 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.677 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.677 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.677 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.677 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.678 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.678 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.678 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.678 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.678 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_stats_pools": { 2024-05-23T00:10:49.678 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_stats_pools", 2024-05-23T00:10:49.678 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.679 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.679 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.679 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.679 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.679 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.679 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.679 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.680 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.680 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.680 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.680 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.680 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-23T00:10:49.680 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-23T00:10:49.680 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.680 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.681 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.681 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "300", 2024-05-23T00:10:49.681 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.681 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.681 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.681 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.681 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.682 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.682 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.682 INFO:teuthology.orchestra.run.smithi103.stdout: "scrape_interval": { 2024-05-23T00:10:49.682 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "scrape_interval", 2024-05-23T00:10:49.682 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.682 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.682 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.683 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15.0", 2024-05-23T00:10:49.683 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.683 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.683 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.683 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.683 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.683 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.684 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.684 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.684 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:49.684 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:49.684 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.684 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.684 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.684 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "::", 2024-05-23T00:10:49.685 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.685 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.685 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.685 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-23T00:10:49.685 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.685 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.685 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.686 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:49.686 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:49.686 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.686 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.686 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.686 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "9283", 2024-05-23T00:10:49.686 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.687 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.687 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.687 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-23T00:10:49.687 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.687 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.687 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.687 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "stale_cache_strategy": { 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "stale_cache_strategy", 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "log", 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.688 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.689 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.689 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.689 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.689 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.689 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_behaviour": { 2024-05-23T00:10:49.689 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_behaviour", 2024-05-23T00:10:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "default", 2024-05-23T00:10:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.690 INFO:teuthology.orchestra.run.smithi103.stdout: "default", 2024-05-23T00:10:49.691 INFO:teuthology.orchestra.run.smithi103.stdout: "error" 2024-05-23T00:10:49.691 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.691 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.691 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.691 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.691 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.691 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_error_status_code": { 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_error_status_code", 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "500", 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "400", 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "599", 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.692 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_support", 2024-05-23T00:10:49.693 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.694 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.694 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.694 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.694 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.694 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.694 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.694 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.694 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.695 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.695 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.695 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.695 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.695 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.695 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.695 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.695 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.696 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.696 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.696 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.696 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.696 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.696 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.696 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.696 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.697 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.697 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.697 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.697 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.697 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.697 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.697 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.697 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.698 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.698 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.698 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.698 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.698 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.698 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.698 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.698 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.699 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.700 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.700 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.700 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.700 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.700 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.700 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.700 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.700 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.701 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.701 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.701 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.701 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.701 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.701 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.701 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.701 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.702 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.702 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.702 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.702 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.702 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.702 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.702 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.702 INFO:teuthology.orchestra.run.smithi103.stdout: "max_concurrent_snap_create": { 2024-05-23T00:10:49.703 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_concurrent_snap_create", 2024-05-23T00:10:49.703 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.703 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.703 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.703 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.703 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.703 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.703 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.704 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.704 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.704 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.704 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.704 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.704 INFO:teuthology.orchestra.run.smithi103.stdout: "mirror_snapshot_schedule": { 2024-05-23T00:10:49.704 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mirror_snapshot_schedule", 2024-05-23T00:10:49.704 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.705 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.706 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.706 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.706 INFO:teuthology.orchestra.run.smithi103.stdout: "trash_purge_schedule": { 2024-05-23T00:10:49.706 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "trash_purge_schedule", 2024-05-23T00:10:49.706 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.706 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.706 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.707 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.707 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.707 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.707 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.707 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.707 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.707 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.707 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "restful", 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.708 INFO:teuthology.orchestra.run.smithi103.stdout: "enable_auth": { 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enable_auth", 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.709 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.710 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.710 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.710 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.710 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.710 INFO:teuthology.orchestra.run.smithi103.stdout: "key_file": { 2024-05-23T00:10:49.710 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "key_file", 2024-05-23T00:10:49.710 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.711 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.711 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.711 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.711 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.711 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.711 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.711 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.711 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.712 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.712 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.712 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.712 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.712 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.712 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.712 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.713 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.713 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.713 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.713 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.713 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.713 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.713 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.714 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.714 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.714 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.714 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.714 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.714 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.714 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.715 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.715 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.715 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.715 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.715 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.715 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.715 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.716 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.716 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.716 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.716 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.716 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.716 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.716 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.716 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.717 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.717 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.717 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.717 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.717 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.717 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.717 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.717 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.718 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.718 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.718 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.718 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.718 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.718 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.718 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.719 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.719 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.719 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.719 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.719 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.719 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.719 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.719 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.720 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.720 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.720 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.720 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.720 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.720 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.720 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.721 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.721 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.721 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.721 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.721 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.721 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.721 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:49.721 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:49.722 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.722 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.722 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.722 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.722 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.722 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.722 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.722 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.723 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.723 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.723 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.723 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.723 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:49.723 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:49.723 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.724 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.724 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.724 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.724 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.724 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.724 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.724 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.724 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.725 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.725 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.725 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.725 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.725 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.725 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.725 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rgw", 2024-05-23T00:10:49.725 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.726 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.726 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.726 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.726 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.726 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.726 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.726 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.726 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.727 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.727 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.727 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.727 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.727 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.727 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.727 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.728 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.728 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.728 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.728 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.728 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.728 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.728 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.728 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.729 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.729 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.729 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.729 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.729 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.729 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.729 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.730 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.730 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.730 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.730 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.730 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.730 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.730 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.730 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.731 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.731 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.731 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.731 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.731 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.731 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.731 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.731 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.732 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.732 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.732 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.732 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.732 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.732 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.732 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.732 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.733 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.734 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.734 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.734 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.734 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.734 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.734 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.734 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.735 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.735 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.735 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.735 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.735 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.735 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.735 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rook", 2024-05-23T00:10:49.735 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.736 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.736 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.736 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.736 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.736 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.736 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.736 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.736 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.737 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.738 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.738 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.738 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.738 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.738 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.738 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.738 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.738 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.739 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.739 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.739 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.739 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.739 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.739 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.739 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.740 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.740 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.740 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.740 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.740 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.740 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.740 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.740 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.741 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.741 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.741 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.741 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.741 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.741 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.741 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.742 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.742 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.742 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.742 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.742 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.742 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.742 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.742 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.743 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.743 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.743 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.743 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.743 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.743 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.743 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.744 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.744 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.744 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.744 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.744 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.744 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.744 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.744 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.745 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.745 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.745 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.745 INFO:teuthology.orchestra.run.smithi103.stdout: "storage_class": { 2024-05-23T00:10:49.745 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "storage_class", 2024-05-23T00:10:49.745 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.745 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.745 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.746 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "local", 2024-05-23T00:10:49.746 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.746 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.746 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.746 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-23T00:10:49.746 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.746 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.746 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.747 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.747 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.747 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.747 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.747 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "selftest", 2024-05-23T00:10:49.747 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.747 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.747 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.748 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.748 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.748 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.748 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.748 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.748 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.748 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.748 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.749 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.749 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.749 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.749 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.749 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.749 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.749 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.749 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.750 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.750 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.750 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.750 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.750 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.750 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.750 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.751 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.751 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.751 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.751 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.751 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.751 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.751 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.751 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.752 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.752 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.752 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.752 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.752 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.752 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.752 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.752 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.753 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.753 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.753 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.753 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.753 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.753 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.753 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.753 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.754 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.754 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.754 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.754 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.754 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.754 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.754 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.755 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.755 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.755 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.755 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.755 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.755 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.755 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.755 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.756 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.756 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.756 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.756 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.756 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.756 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.756 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.756 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.757 INFO:teuthology.orchestra.run.smithi103.stdout: "roption1": { 2024-05-23T00:10:49.757 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "roption1", 2024-05-23T00:10:49.757 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.757 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.757 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.757 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.757 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.757 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.758 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.758 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.758 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.758 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.758 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.758 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.758 INFO:teuthology.orchestra.run.smithi103.stdout: "roption2": { 2024-05-23T00:10:49.759 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "roption2", 2024-05-23T00:10:49.759 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.759 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.759 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.759 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "xyz", 2024-05-23T00:10:49.759 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.759 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.759 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.760 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.760 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.760 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.760 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.760 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.760 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption1": { 2024-05-23T00:10:49.760 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption1", 2024-05-23T00:10:49.760 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.761 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.761 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.761 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.761 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.761 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.761 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.761 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.761 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.762 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.762 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.762 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.762 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption2": { 2024-05-23T00:10:49.762 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption2", 2024-05-23T00:10:49.762 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.762 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.762 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.763 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.763 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.763 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.763 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.763 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.763 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.763 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.763 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.764 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.764 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption3": { 2024-05-23T00:10:49.764 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption3", 2024-05-23T00:10:49.764 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.764 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.764 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.764 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.765 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.765 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.765 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.765 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.765 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.765 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.765 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.765 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.766 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption4": { 2024-05-23T00:10:49.766 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption4", 2024-05-23T00:10:49.766 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.766 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.766 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.766 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.766 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.767 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.767 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.767 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.767 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.767 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.767 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.767 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.767 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption5": { 2024-05-23T00:10:49.768 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption5", 2024-05-23T00:10:49.768 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.768 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.768 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.768 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.768 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.768 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.768 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.769 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.769 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.769 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.769 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.769 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.769 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption6": { 2024-05-23T00:10:49.769 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption6", 2024-05-23T00:10:49.770 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.770 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.770 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.770 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.770 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.770 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.770 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.770 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.771 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.771 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.771 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.771 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.771 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption7": { 2024-05-23T00:10:49.771 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption7", 2024-05-23T00:10:49.771 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.772 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.772 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.772 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.772 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:49.772 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "42", 2024-05-23T00:10:49.772 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.772 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.772 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.773 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.773 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.773 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.773 INFO:teuthology.orchestra.run.smithi103.stdout: "testkey": { 2024-05-23T00:10:49.773 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testkey", 2024-05-23T00:10:49.773 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.773 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.774 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.774 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.774 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.774 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.774 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.774 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.774 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.774 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.775 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.775 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.775 INFO:teuthology.orchestra.run.smithi103.stdout: "testlkey": { 2024-05-23T00:10:49.775 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testlkey", 2024-05-23T00:10:49.775 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.775 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.775 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.776 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.776 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.776 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.776 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.776 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.776 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.776 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.776 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.777 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.777 INFO:teuthology.orchestra.run.smithi103.stdout: "testnewline": { 2024-05-23T00:10:49.777 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testnewline", 2024-05-23T00:10:49.777 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.777 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.777 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.777 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.778 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.778 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.778 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.778 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.778 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.778 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.778 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.778 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.779 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.779 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.779 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.779 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snap_schedule", 2024-05-23T00:10:49.779 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.779 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.779 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.780 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_m_granularity": { 2024-05-23T00:10:49.780 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_m_granularity", 2024-05-23T00:10:49.780 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.780 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.780 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.780 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.780 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.780 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.781 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.781 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow minute scheduled snapshots", 2024-05-23T00:10:49.781 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.781 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.781 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.781 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.781 INFO:teuthology.orchestra.run.smithi103.stdout: "dump_on_update": { 2024-05-23T00:10:49.781 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "dump_on_update", 2024-05-23T00:10:49.782 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.782 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.782 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.782 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.782 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.782 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.782 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.783 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "dump database to debug log on update", 2024-05-23T00:10:49.783 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.783 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.783 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.783 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.783 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.783 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.783 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.784 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.784 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.784 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.784 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.784 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.784 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.784 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.784 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.785 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.785 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.785 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.785 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.785 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.785 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.785 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.786 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.786 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.786 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.786 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.786 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.786 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.786 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.786 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.787 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.787 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.787 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.787 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.787 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.787 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.787 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.787 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.788 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.789 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.790 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.790 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.790 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.790 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.790 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.790 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.791 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.792 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.792 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.792 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.792 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.792 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.792 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.792 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.793 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.793 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "stats", 2024-05-23T00:10:49.793 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.793 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.793 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.793 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.793 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.794 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.794 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.794 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.794 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.794 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.794 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.794 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.794 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.795 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.795 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.795 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.795 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.795 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.795 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.795 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.796 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.796 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.796 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.796 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.796 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.796 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.797 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.797 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.797 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.797 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.797 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.797 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.797 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.797 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.798 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.798 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.798 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.798 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.798 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.798 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.799 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.799 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.799 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.799 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.799 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.799 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.799 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.800 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.800 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.800 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.800 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.800 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.800 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.800 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.801 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.801 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.801 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.801 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.801 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.801 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.801 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.802 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.802 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.802 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.802 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.802 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.802 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.802 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.803 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.803 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.803 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.803 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.803 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.803 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "status", 2024-05-23T00:10:49.803 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.804 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.804 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.804 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.804 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.804 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.804 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.804 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.804 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.805 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.805 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.805 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.805 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.805 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.805 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.805 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.806 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.806 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.806 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.806 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.806 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.806 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.806 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.807 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.808 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.808 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.808 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.808 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.808 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.808 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.808 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.809 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.809 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.809 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.809 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.809 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.809 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.809 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.810 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.810 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.810 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.810 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.810 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.810 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.810 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.810 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.811 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.811 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.811 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.811 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.811 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.811 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.812 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.812 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.812 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.812 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.812 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.812 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.812 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.813 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.813 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.813 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.813 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.813 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.813 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.813 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.814 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "telegraf", 2024-05-23T00:10:49.814 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.814 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.814 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.814 INFO:teuthology.orchestra.run.smithi103.stdout: "address": { 2024-05-23T00:10:49.814 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "address", 2024-05-23T00:10:49.814 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.815 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.815 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.815 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-23T00:10:49.815 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.815 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.815 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.815 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.815 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.816 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.816 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.816 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.816 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.816 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.816 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.817 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.817 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15", 2024-05-23T00:10:49.817 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.817 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.817 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.817 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.817 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.818 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.818 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.818 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.818 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.818 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.818 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.818 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.819 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.819 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.819 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.819 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.819 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.819 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.819 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.820 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.820 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.820 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.820 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.820 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.820 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.820 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.820 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.821 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.821 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.821 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.821 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.821 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.821 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.822 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.822 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.822 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.822 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.822 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.822 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.822 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.822 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.823 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.823 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.823 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.823 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.823 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.823 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.823 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.823 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.824 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.824 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.824 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.824 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.824 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.824 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.824 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.824 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.825 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.825 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.825 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.825 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.825 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.825 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.825 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.826 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.826 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.826 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.826 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.826 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.826 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.826 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.827 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.827 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.827 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.827 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.827 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.827 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.827 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.828 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "telemetry", 2024-05-23T00:10:49.828 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.828 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.828 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.828 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_basic": { 2024-05-23T00:10:49.828 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_basic", 2024-05-23T00:10:49.828 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.828 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.829 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.829 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.829 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.829 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.829 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.829 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-23T00:10:49.829 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.830 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.830 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.830 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_crash": { 2024-05-23T00:10:49.830 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_crash", 2024-05-23T00:10:49.830 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.830 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.830 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.831 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.831 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.831 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.831 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.831 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-23T00:10:49.831 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.831 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.831 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.832 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.832 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_device": { 2024-05-23T00:10:49.832 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_device", 2024-05-23T00:10:49.832 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.832 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.832 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.832 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.832 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.833 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.833 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.833 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-23T00:10:49.833 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.833 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.833 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.834 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_ident": { 2024-05-23T00:10:49.834 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_ident", 2024-05-23T00:10:49.834 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.834 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.834 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.834 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.834 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.834 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.835 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.835 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-23T00:10:49.835 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.835 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.835 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.835 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.835 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_perf": { 2024-05-23T00:10:49.835 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_perf", 2024-05-23T00:10:49.836 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.836 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.836 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.836 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.836 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.836 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.836 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.836 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-23T00:10:49.837 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.837 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.837 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.837 INFO:teuthology.orchestra.run.smithi103.stdout: "contact": { 2024-05-23T00:10:49.837 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "contact", 2024-05-23T00:10:49.837 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.838 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.838 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.838 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.838 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.838 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.838 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.838 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.838 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.839 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.839 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.839 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.839 INFO:teuthology.orchestra.run.smithi103.stdout: "description": { 2024-05-23T00:10:49.839 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "description", 2024-05-23T00:10:49.839 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.839 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.840 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.840 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.840 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.840 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.840 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.840 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.840 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.840 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.841 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.841 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.841 INFO:teuthology.orchestra.run.smithi103.stdout: "device_url": { 2024-05-23T00:10:49.841 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_url", 2024-05-23T00:10:49.841 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.841 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.841 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.842 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-23T00:10:49.842 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.842 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.842 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.842 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.842 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.842 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.842 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.843 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.843 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": { 2024-05-23T00:10:49.843 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enabled", 2024-05-23T00:10:49.843 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.843 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.843 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.843 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.844 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.844 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.844 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.844 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.844 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.844 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.844 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.845 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.845 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.845 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.845 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.845 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.845 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "24", 2024-05-23T00:10:49.845 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "8", 2024-05-23T00:10:49.846 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.846 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.846 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.846 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.846 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.846 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.847 INFO:teuthology.orchestra.run.smithi103.stdout: "last_opt_revision": { 2024-05-23T00:10:49.847 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "last_opt_revision", 2024-05-23T00:10:49.847 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.847 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.847 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.847 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1", 2024-05-23T00:10:49.847 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.848 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.848 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.848 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.848 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.848 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.848 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.848 INFO:teuthology.orchestra.run.smithi103.stdout: "leaderboard": { 2024-05-23T00:10:49.849 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "leaderboard", 2024-05-23T00:10:49.849 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.849 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.849 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.849 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.849 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.849 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.850 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.850 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.850 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.850 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.850 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.850 INFO:teuthology.orchestra.run.smithi103.stdout: "leaderboard_description": { 2024-05-23T00:10:49.850 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "leaderboard_description", 2024-05-23T00:10:49.851 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.851 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.851 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.851 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.851 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.851 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.851 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.852 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.852 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.852 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.852 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.852 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.852 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.852 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.852 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.853 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.853 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.853 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.853 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.853 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.853 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.853 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.853 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.854 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.854 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.854 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.854 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.854 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.854 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.854 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.855 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.855 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.855 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.855 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.855 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.855 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.855 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.856 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.856 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.856 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.856 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.856 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.856 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.856 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.856 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.857 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.857 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.857 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.857 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.857 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.857 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.857 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.858 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.858 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.858 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.858 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.858 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.858 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.858 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.859 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.859 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.859 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.859 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.859 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.859 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.859 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.860 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.860 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.860 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.860 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.860 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.860 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.860 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.860 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.861 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.861 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.861 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.861 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.861 INFO:teuthology.orchestra.run.smithi103.stdout: "organization": { 2024-05-23T00:10:49.861 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "organization", 2024-05-23T00:10:49.861 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.862 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.862 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.862 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.862 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.862 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.862 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.862 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.862 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: "proxy": { 2024-05-23T00:10:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "proxy", 2024-05-23T00:10:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.863 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.864 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.865 INFO:teuthology.orchestra.run.smithi103.stdout: "url": { 2024-05-23T00:10:49.865 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "url", 2024-05-23T00:10:49.865 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.865 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.865 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.865 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-23T00:10:49.866 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.866 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.866 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.866 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.866 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.866 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.866 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.867 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.867 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.867 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.867 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "test_orchestrator", 2024-05-23T00:10:49.867 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.867 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.867 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.868 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.868 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.868 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.868 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.868 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.868 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.868 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.868 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.869 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.869 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.869 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.869 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.869 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.869 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.869 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.869 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.870 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.870 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.870 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.870 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.870 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.870 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.871 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.871 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.871 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.871 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.871 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.871 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.871 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.871 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.872 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.872 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.872 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.872 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.872 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.872 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.872 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.873 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.874 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.874 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.874 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.874 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.874 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.874 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.874 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.875 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.875 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.875 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.875 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.875 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.875 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.875 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.876 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.876 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.876 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.876 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.876 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.876 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.876 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.876 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.877 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.877 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.877 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.877 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.877 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "volumes", 2024-05-23T00:10:49.877 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.877 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.878 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.878 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.878 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.878 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.878 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.878 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.878 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.878 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.879 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.879 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.879 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.879 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.879 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.879 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.879 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.880 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.881 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.881 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.881 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.881 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.881 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.881 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.881 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.881 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.882 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.882 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.882 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.882 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.882 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.882 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.882 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.882 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.883 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.883 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.883 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.883 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.883 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.883 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.883 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.883 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.884 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.884 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.884 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.884 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.884 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.884 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.884 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.885 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.885 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.885 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.885 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.885 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.885 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.885 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.886 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.886 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.886 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.886 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.886 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.886 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.886 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.886 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.887 INFO:teuthology.orchestra.run.smithi103.stdout: "max_concurrent_clones": { 2024-05-23T00:10:49.887 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_concurrent_clones", 2024-05-23T00:10:49.887 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.887 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.887 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.887 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "4", 2024-05-23T00:10:49.887 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.888 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.888 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.888 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-23T00:10:49.888 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.888 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.888 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.888 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.888 INFO:teuthology.orchestra.run.smithi103.stdout: "periodic_async_work": { 2024-05-23T00:10:49.889 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "periodic_async_work", 2024-05-23T00:10:49.889 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.889 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.889 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.889 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.889 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.889 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.889 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.890 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Periodically check for async work", 2024-05-23T00:10:49.890 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.890 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.890 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.890 INFO:teuthology.orchestra.run.smithi103.stdout: "snapshot_clone_delay": { 2024-05-23T00:10:49.890 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snapshot_clone_delay", 2024-05-23T00:10:49.891 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.891 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.891 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.891 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.891 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.891 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.891 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.891 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-23T00:10:49.892 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.892 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.892 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.892 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.892 INFO:teuthology.orchestra.run.smithi103.stdout: "snapshot_clone_no_wait": { 2024-05-23T00:10:49.892 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snapshot_clone_no_wait", 2024-05-23T00:10:49.892 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.892 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.893 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.893 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.893 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.893 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.893 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.893 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-23T00:10:49.893 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.894 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.894 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.894 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.894 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.894 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.894 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.894 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix", 2024-05-23T00:10:49.894 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.895 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.895 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.895 INFO:teuthology.orchestra.run.smithi103.stdout: "discovery_interval": { 2024-05-23T00:10:49.895 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "discovery_interval", 2024-05-23T00:10:49.895 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.895 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.895 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.895 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "100", 2024-05-23T00:10:49.896 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.896 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.896 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.896 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.896 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.896 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.896 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.896 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.897 INFO:teuthology.orchestra.run.smithi103.stdout: "identifier": { 2024-05-23T00:10:49.897 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "identifier", 2024-05-23T00:10:49.897 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.897 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.897 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.897 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.897 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.897 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.898 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.898 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.898 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.898 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.898 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.898 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.898 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.899 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.899 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.899 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.899 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.899 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:49.899 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.899 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.899 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.900 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.900 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.900 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.900 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.900 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.900 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.900 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.900 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.901 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.901 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.901 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.901 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.901 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.901 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.901 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.901 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.902 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.902 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.902 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.902 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.902 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.902 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.902 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.903 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.903 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.903 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.903 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.903 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.903 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.903 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.903 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.904 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.904 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.904 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.904 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.904 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.904 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.904 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.904 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.905 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.905 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.905 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.905 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.905 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.905 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.905 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.905 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.906 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.906 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.906 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.906 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.906 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.906 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.906 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.907 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.907 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.907 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.907 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.907 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.907 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.907 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.907 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.908 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.908 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.908 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.908 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.908 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.908 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.908 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.908 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.909 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.909 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.909 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.909 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.909 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.909 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_host": { 2024-05-23T00:10:49.909 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_host", 2024-05-23T00:10:49.909 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.910 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.910 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.910 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.910 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.910 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.910 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.910 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.910 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.911 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.911 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.911 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.911 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_port": { 2024-05-23T00:10:49.911 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_port", 2024-05-23T00:10:49.911 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.911 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10051", 2024-05-23T00:10:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.912 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.913 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.913 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.913 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_sender": { 2024-05-23T00:10:49.913 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_sender", 2024-05-23T00:10:49.913 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.913 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.913 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.913 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-23T00:10:49.914 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.914 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.914 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.914 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.914 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.914 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.914 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.915 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:49.915 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.915 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.915 INFO:teuthology.orchestra.run.smithi103.stdout: "modules": [ 2024-05-23T00:10:49.915 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm", 2024-05-23T00:10:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: "dashboard", 2024-05-23T00:10:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: "iostat", 2024-05-23T00:10:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: "nfs", 2024-05-23T00:10:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: "restful", 2024-05-23T00:10:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: "snap_schedule" 2024-05-23T00:10:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: "available_modules": [ 2024-05-23T00:10:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "alerts", 2024-05-23T00:10:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How frequently to reexamine health status", 2024-05-23T00:10:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.928 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.928 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.928 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.928 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.928 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_destination": { 2024-05-23T00:10:49.928 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_destination", 2024-05-23T00:10:49.928 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.928 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.929 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.929 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.929 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.929 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.929 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.929 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Email address to send alerts to", 2024-05-23T00:10:49.929 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.929 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.930 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.930 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.930 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_from_name": { 2024-05-23T00:10:49.930 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_from_name", 2024-05-23T00:10:49.930 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.930 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.930 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.930 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "Ceph", 2024-05-23T00:10:49.931 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.931 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.931 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.931 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Email From: name", 2024-05-23T00:10:49.931 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.931 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.931 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.933 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.933 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_host": { 2024-05-23T00:10:49.933 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_host", 2024-05-23T00:10:49.933 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.933 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.933 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.934 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.934 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.934 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.934 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.934 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP server", 2024-05-23T00:10:49.934 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.934 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.935 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.935 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.935 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_password": { 2024-05-23T00:10:49.935 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_password", 2024-05-23T00:10:49.935 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.935 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.935 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.935 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.936 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.936 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.936 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.936 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Password to authenticate with", 2024-05-23T00:10:49.936 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.936 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.936 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.937 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.937 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_port": { 2024-05-23T00:10:49.937 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_port", 2024-05-23T00:10:49.937 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.937 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.937 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.937 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "465", 2024-05-23T00:10:49.937 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.938 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.938 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.938 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP port", 2024-05-23T00:10:49.938 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.938 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.938 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.938 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.939 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_sender": { 2024-05-23T00:10:49.939 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_sender", 2024-05-23T00:10:49.939 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.939 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.939 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.939 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.939 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.940 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.940 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.940 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SMTP envelope sender", 2024-05-23T00:10:49.940 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.940 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.940 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.940 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.940 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_ssl": { 2024-05-23T00:10:49.941 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_ssl", 2024-05-23T00:10:49.941 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.941 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.941 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.941 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.941 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.941 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.942 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.942 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-23T00:10:49.942 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.942 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.942 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.942 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.942 INFO:teuthology.orchestra.run.smithi103.stdout: "smtp_user": { 2024-05-23T00:10:49.942 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "smtp_user", 2024-05-23T00:10:49.943 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.943 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.943 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.943 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.943 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.943 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.943 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.943 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "User to authenticate as", 2024-05-23T00:10:49.944 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.944 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.944 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.944 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.944 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.944 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.944 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.945 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "balancer", 2024-05-23T00:10:49.945 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.945 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.945 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.945 INFO:teuthology.orchestra.run.smithi103.stdout: "active": { 2024-05-23T00:10:49.945 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "active", 2024-05-23T00:10:49.945 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.945 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.946 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.946 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.946 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.946 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.946 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.946 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "automatically balance PGs across cluster", 2024-05-23T00:10:49.946 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.946 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.947 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.947 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.947 INFO:teuthology.orchestra.run.smithi103.stdout: "begin_time": { 2024-05-23T00:10:49.947 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "begin_time", 2024-05-23T00:10:49.947 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.947 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.947 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.947 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0000", 2024-05-23T00:10:49.948 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.948 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.948 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.948 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "beginning time of day to automatically balance", 2024-05-23T00:10:49.948 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-23T00:10:49.948 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.948 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.948 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.949 INFO:teuthology.orchestra.run.smithi103.stdout: "begin_weekday": { 2024-05-23T00:10:49.949 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "begin_weekday", 2024-05-23T00:10:49.949 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.949 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.949 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.949 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.949 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0", 2024-05-23T00:10:49.949 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "6", 2024-05-23T00:10:49.950 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.950 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-23T00:10:49.950 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-23T00:10:49.950 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.950 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.950 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.950 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_max_iterations": { 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_max_iterations", 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "25", 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "250", 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.951 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-23T00:10:49.952 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.952 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.952 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.952 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.952 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_metrics": { 2024-05-23T00:10:49.952 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_metrics", 2024-05-23T00:10:49.952 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.952 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.953 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.953 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "pgs,objects,bytes", 2024-05-23T00:10:49.953 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.953 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.953 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.953 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-23T00:10:49.953 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-23T00:10:49.954 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.954 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.954 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.954 INFO:teuthology.orchestra.run.smithi103.stdout: "crush_compat_step": { 2024-05-23T00:10:49.954 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crush_compat_step", 2024-05-23T00:10:49.954 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.954 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.954 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.955 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0.5", 2024-05-23T00:10:49.955 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0.001", 2024-05-23T00:10:49.955 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "0.999", 2024-05-23T00:10:49.955 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.955 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "aggressiveness of optimization", 2024-05-23T00:10:49.955 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-23T00:10:49.955 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.955 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.956 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.956 INFO:teuthology.orchestra.run.smithi103.stdout: "end_time": { 2024-05-23T00:10:49.956 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "end_time", 2024-05-23T00:10:49.956 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.956 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.956 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.956 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "2359", 2024-05-23T00:10:49.956 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.957 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.957 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.957 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "ending time of day to automatically balance", 2024-05-23T00:10:49.957 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-23T00:10:49.957 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.957 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.957 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.957 INFO:teuthology.orchestra.run.smithi103.stdout: "end_weekday": { 2024-05-23T00:10:49.958 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "end_weekday", 2024-05-23T00:10:49.958 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.958 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.958 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.958 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.958 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "0", 2024-05-23T00:10:49.958 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "6", 2024-05-23T00:10:49.959 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.959 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-23T00:10:49.959 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-23T00:10:49.959 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.959 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.959 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.959 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:49.959 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:49.960 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.960 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.960 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.960 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.960 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.960 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.960 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.960 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.961 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.961 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.961 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.961 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.961 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.961 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.961 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.962 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.963 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.963 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.963 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.963 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.963 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.963 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.963 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.964 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.964 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.964 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.964 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:49.964 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:49.964 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.964 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.964 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.965 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:49.965 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.965 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.965 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.965 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:49.965 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:49.965 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:49.965 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:49.966 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:49.966 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:49.966 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.966 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.966 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.966 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.966 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.967 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.967 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:49.967 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:49.967 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.967 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.967 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.967 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.967 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.968 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.968 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.968 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:49.968 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.968 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.968 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.968 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.968 INFO:teuthology.orchestra.run.smithi103.stdout: "min_score": { 2024-05-23T00:10:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "min_score", 2024-05-23T00:10:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.969 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-23T00:10:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": { 2024-05-23T00:10:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mode", 2024-05-23T00:10:49.970 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.971 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.971 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.971 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "upmap", 2024-05-23T00:10:49.971 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.971 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.971 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.971 INFO:teuthology.orchestra.run.smithi103.stdout: "crush-compat", 2024-05-23T00:10:49.971 INFO:teuthology.orchestra.run.smithi103.stdout: "none", 2024-05-23T00:10:49.972 INFO:teuthology.orchestra.run.smithi103.stdout: "read", 2024-05-23T00:10:49.972 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap", 2024-05-23T00:10:49.972 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap-read" 2024-05-23T00:10:49.972 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.972 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Balancer mode", 2024-05-23T00:10:49.972 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.972 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.973 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.973 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.973 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_ids": { 2024-05-23T00:10:49.973 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pool_ids", 2024-05-23T00:10:49.973 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.973 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.973 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.973 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.974 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.974 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.974 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.974 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-23T00:10:49.974 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.974 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.974 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.974 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.975 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:49.975 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:49.975 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.975 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.975 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.975 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:49.975 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.975 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.976 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.976 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-23T00:10:49.976 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.976 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.976 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.976 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap_max_deviation": { 2024-05-23T00:10:49.977 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "upmap_max_deviation", 2024-05-23T00:10:49.977 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.977 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.977 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.977 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:49.977 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:49.977 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.977 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.978 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-23T00:10:49.978 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-23T00:10:49.978 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.978 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.978 INFO:teuthology.orchestra.run.smithi103.stdout: "upmap_max_optimizations": { 2024-05-23T00:10:49.978 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "upmap_max_optimizations", 2024-05-23T00:10:49.979 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:49.979 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.979 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:49.979 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:49.979 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.979 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.979 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.979 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-23T00:10:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:49.980 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cephadm", 2024-05-23T00:10:49.981 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:49.981 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:49.981 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:49.981 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_down_multiplier": { 2024-05-23T00:10:49.981 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_down_multiplier", 2024-05-23T00:10:49.981 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.981 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.981 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.982 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3.0", 2024-05-23T00:10:49.982 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.982 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.982 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.982 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-23T00:10:49.982 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.982 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.983 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.983 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_refresh_rate": { 2024-05-23T00:10:49.983 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_refresh_rate", 2024-05-23T00:10:49.983 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.983 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.983 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.983 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "20", 2024-05-23T00:10:49.984 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.984 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.984 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.984 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-23T00:10:49.984 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.984 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.984 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.984 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.985 INFO:teuthology.orchestra.run.smithi103.stdout: "agent_starting_port": { 2024-05-23T00:10:49.985 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "agent_starting_port", 2024-05-23T00:10:49.985 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:49.985 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.985 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.985 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "4721", 2024-05-23T00:10:49.985 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.986 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.986 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.986 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-23T00:10:49.986 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.986 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.986 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.986 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.986 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_ptrace": { 2024-05-23T00:10:49.987 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_ptrace", 2024-05-23T00:10:49.987 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.987 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.987 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.987 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.987 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.987 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.987 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.988 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-23T00:10:49.988 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-23T00:10:49.988 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.988 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.988 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.988 INFO:teuthology.orchestra.run.smithi103.stdout: "autotune_interval": { 2024-05-23T00:10:49.988 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "autotune_interval", 2024-05-23T00:10:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.989 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-23T00:10:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: "autotune_memory_target_ratio": { 2024-05-23T00:10:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "autotune_memory_target_ratio", 2024-05-23T00:10:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:49.990 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.991 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.991 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0.7", 2024-05-23T00:10:49.991 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.991 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.991 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.991 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-23T00:10:49.991 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.991 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.992 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.992 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.992 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm_log_destination": { 2024-05-23T00:10:49.992 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cephadm_log_destination", 2024-05-23T00:10:49.992 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:49.992 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.992 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.992 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: "file", 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: "file,syslog", 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: "syslog" 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-23T00:10:49.993 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.994 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.994 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.994 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.994 INFO:teuthology.orchestra.run.smithi103.stdout: "cgroups_split": { 2024-05-23T00:10:49.994 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cgroups_split", 2024-05-23T00:10:49.994 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.994 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.995 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.996 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.996 INFO:teuthology.orchestra.run.smithi103.stdout: "config_checks_enabled": { 2024-05-23T00:10:49.996 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_checks_enabled", 2024-05-23T00:10:49.996 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.996 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.996 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.996 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:49.996 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.997 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.997 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.997 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-23T00:10:49.997 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.997 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.997 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.997 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.997 INFO:teuthology.orchestra.run.smithi103.stdout: "config_dashboard": { 2024-05-23T00:10:49.998 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_dashboard", 2024-05-23T00:10:49.998 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:49.998 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:49.998 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:49.998 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:49.998 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:49.998 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:49.999 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:49.999 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-23T00:10:49.999 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:49.999 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:49.999 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:49.999 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:49.999 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_alertmanager": { 2024-05-23T00:10:49.999 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_alertmanager", 2024-05-23T00:10:50.000 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.000 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.000 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.000 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-23T00:10:50.000 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.000 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.000 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.000 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:50.001 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.001 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.001 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.001 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.001 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_base": { 2024-05-23T00:10:50.001 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_base", 2024-05-23T00:10:50.001 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.001 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.002 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.002 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-23T00:10:50.002 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.002 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.002 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.002 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Container image name, without the tag", 2024-05-23T00:10:50.002 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.003 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.003 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.003 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.003 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_elasticsearch": { 2024-05-23T00:10:50.003 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_elasticsearch", 2024-05-23T00:10:50.003 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.003 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.003 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.004 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-23T00:10:50.004 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.004 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.004 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.004 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "elasticsearch container image", 2024-05-23T00:10:50.004 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.004 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.005 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.005 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.005 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_grafana": { 2024-05-23T00:10:50.005 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_grafana", 2024-05-23T00:10:50.005 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.005 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.005 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.005 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-23T00:10:50.006 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.006 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.006 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.006 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:50.006 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.006 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.006 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.006 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.007 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_haproxy": { 2024-05-23T00:10:50.007 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_haproxy", 2024-05-23T00:10:50.007 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.007 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.007 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.007 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-23T00:10:50.007 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.008 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.008 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.008 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "HAproxy container image", 2024-05-23T00:10:50.008 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.008 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.008 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.008 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.008 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_agent": { 2024-05-23T00:10:50.009 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_agent", 2024-05-23T00:10:50.009 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.009 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.009 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.009 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-23T00:10:50.009 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.009 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.010 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.010 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger agent container image", 2024-05-23T00:10:50.010 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.010 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.010 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.010 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.010 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_collector": { 2024-05-23T00:10:50.010 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_collector", 2024-05-23T00:10:50.011 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.011 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.011 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.011 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-23T00:10:50.011 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.011 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.011 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.011 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger collector container image", 2024-05-23T00:10:50.012 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.012 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.012 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.012 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.012 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_jaeger_query": { 2024-05-23T00:10:50.012 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_jaeger_query", 2024-05-23T00:10:50.012 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-23T00:10:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Jaeger query container image", 2024-05-23T00:10:50.013 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.014 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.014 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.014 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.014 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_keepalived": { 2024-05-23T00:10:50.014 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_keepalived", 2024-05-23T00:10:50.014 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.014 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.014 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.015 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-23T00:10:50.015 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.015 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.015 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.015 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Keepalived container image", 2024-05-23T00:10:50.015 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.015 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.016 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.016 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.016 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_loki": { 2024-05-23T00:10:50.016 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_loki", 2024-05-23T00:10:50.016 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.016 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.016 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.016 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-23T00:10:50.017 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.017 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.017 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.017 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Loki container image", 2024-05-23T00:10:50.017 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.017 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.017 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.017 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.018 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_node_exporter": { 2024-05-23T00:10:50.018 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_node_exporter", 2024-05-23T00:10:50.018 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.018 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.018 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.018 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-23T00:10:50.018 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.018 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.019 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.019 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:50.019 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.019 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.019 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.019 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.019 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_nvmeof": { 2024-05-23T00:10:50.019 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_nvmeof", 2024-05-23T00:10:50.020 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.020 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.020 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.020 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-23T00:10:50.020 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.020 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.020 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.021 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Nvme-of container image", 2024-05-23T00:10:50.021 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.021 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.021 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.021 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.021 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_prometheus": { 2024-05-23T00:10:50.021 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_prometheus", 2024-05-23T00:10:50.021 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.022 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.022 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.022 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-23T00:10:50.022 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.022 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.022 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.022 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Prometheus container image", 2024-05-23T00:10:50.022 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.023 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.023 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.023 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.023 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_promtail": { 2024-05-23T00:10:50.023 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_promtail", 2024-05-23T00:10:50.023 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.023 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.023 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.024 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-23T00:10:50.024 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.024 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.024 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.024 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Promtail container image", 2024-05-23T00:10:50.024 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.024 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_samba": { 2024-05-23T00:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_samba", 2024-05-23T00:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.025 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-23T00:10:50.026 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.026 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.026 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.026 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Samba/SMB container image", 2024-05-23T00:10:50.026 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.026 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.026 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.026 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.027 INFO:teuthology.orchestra.run.smithi103.stdout: "container_image_snmp_gateway": { 2024-05-23T00:10:50.027 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_image_snmp_gateway", 2024-05-23T00:10:50.027 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.027 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.027 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.027 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-23T00:10:50.027 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.027 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.028 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.028 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "SNMP Gateway container image", 2024-05-23T00:10:50.028 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.028 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.028 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.028 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.029 INFO:teuthology.orchestra.run.smithi103.stdout: "container_init": { 2024-05-23T00:10:50.029 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "container_init", 2024-05-23T00:10:50.029 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.029 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.029 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.029 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.029 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.029 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.030 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.030 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Run podman/docker with `--init`", 2024-05-23T00:10:50.030 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.030 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.030 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.030 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.030 INFO:teuthology.orchestra.run.smithi103.stdout: "daemon_cache_timeout": { 2024-05-23T00:10:50.031 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "daemon_cache_timeout", 2024-05-23T00:10:50.031 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.031 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.031 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.031 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:50.031 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.031 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.032 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.032 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-23T00:10:50.032 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.032 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.032 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.032 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.032 INFO:teuthology.orchestra.run.smithi103.stdout: "default_cephadm_command_timeout": { 2024-05-23T00:10:50.032 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "default_cephadm_command_timeout", 2024-05-23T00:10:50.033 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.033 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.033 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.033 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "900", 2024-05-23T00:10:50.033 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.033 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.033 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.034 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-23T00:10:50.034 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.034 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.034 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.034 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.034 INFO:teuthology.orchestra.run.smithi103.stdout: "default_registry": { 2024-05-23T00:10:50.034 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "default_registry", 2024-05-23T00:10:50.035 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.035 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.035 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.035 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "docker.io", 2024-05-23T00:10:50.035 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.035 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.035 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.035 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-23T00:10:50.036 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.036 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.036 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.036 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.036 INFO:teuthology.orchestra.run.smithi103.stdout: "device_cache_timeout": { 2024-05-23T00:10:50.036 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_cache_timeout", 2024-05-23T00:10:50.036 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.036 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.037 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.037 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1800", 2024-05-23T00:10:50.037 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.037 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.037 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.037 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache device inventory", 2024-05-23T00:10:50.037 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.038 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.038 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.038 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.038 INFO:teuthology.orchestra.run.smithi103.stdout: "device_enhanced_scan": { 2024-05-23T00:10:50.038 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_enhanced_scan", 2024-05-23T00:10:50.038 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.038 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.038 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.039 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.039 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.039 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.039 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.039 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-23T00:10:50.039 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.039 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.040 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.040 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.040 INFO:teuthology.orchestra.run.smithi103.stdout: "facts_cache_timeout": { 2024-05-23T00:10:50.040 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "facts_cache_timeout", 2024-05-23T00:10:50.040 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.040 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.040 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.040 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:50.041 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.041 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.041 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.041 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "seconds to cache host facts data", 2024-05-23T00:10:50.041 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.041 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.041 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.042 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.042 INFO:teuthology.orchestra.run.smithi103.stdout: "grafana_dashboards_path": { 2024-05-23T00:10:50.042 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "grafana_dashboards_path", 2024-05-23T00:10:50.042 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.042 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.042 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.042 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-23T00:10:50.042 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.043 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.043 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.043 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-23T00:10:50.043 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.043 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.043 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.043 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.044 INFO:teuthology.orchestra.run.smithi103.stdout: "host_check_interval": { 2024-05-23T00:10:50.044 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "host_check_interval", 2024-05-23T00:10:50.044 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.044 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.044 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.044 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:50.044 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to perform a host check", 2024-05-23T00:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.045 INFO:teuthology.orchestra.run.smithi103.stdout: "hw_monitoring": { 2024-05-23T00:10:50.046 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "hw_monitoring", 2024-05-23T00:10:50.046 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.046 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.046 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.046 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.046 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.046 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.047 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.047 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-23T00:10:50.047 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.047 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.047 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.047 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.047 INFO:teuthology.orchestra.run.smithi103.stdout: "inventory_list_all": { 2024-05-23T00:10:50.047 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "inventory_list_all", 2024-05-23T00:10:50.048 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.048 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.048 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.048 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.048 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.048 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.048 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.048 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-23T00:10:50.049 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.049 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.049 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.049 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.049 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.049 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.049 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.049 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.050 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.050 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.050 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.050 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.050 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.050 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.050 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.051 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.051 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.051 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.051 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.051 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.051 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.051 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.052 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.052 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.052 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.052 INFO:teuthology.orchestra.run.smithi103.stdout: "log_refresh_metadata": { 2024-05-23T00:10:50.052 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_refresh_metadata", 2024-05-23T00:10:50.052 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.052 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.052 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.053 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.053 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.053 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.053 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.053 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-23T00:10:50.053 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.053 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.053 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.054 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.054 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.054 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.054 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.054 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.054 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.054 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.055 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.055 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.055 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.055 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-23T00:10:50.055 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.055 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.055 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.055 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.056 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.056 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.056 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.056 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.056 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.056 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.056 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.057 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.057 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.057 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.057 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.057 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.057 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.057 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.057 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.058 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.058 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.058 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.058 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.058 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.058 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.058 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.058 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.059 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.059 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.059 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.059 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.059 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.059 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.059 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.060 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.060 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.060 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.060 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.060 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.060 INFO:teuthology.orchestra.run.smithi103.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-23T00:10:50.060 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-23T00:10:50.060 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.061 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.061 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.061 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.061 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.061 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.061 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.061 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-23T00:10:50.061 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.062 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.062 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.062 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.062 INFO:teuthology.orchestra.run.smithi103.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-23T00:10:50.062 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-23T00:10:50.062 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.062 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.063 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.063 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "*", 2024-05-23T00:10:50.063 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.063 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.063 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.063 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-23T00:10:50.063 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.063 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.064 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.064 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.064 INFO:teuthology.orchestra.run.smithi103.stdout: "max_count_per_host": { 2024-05-23T00:10:50.064 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_count_per_host", 2024-05-23T00:10:50.064 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.064 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.064 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.065 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:50.065 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.065 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.065 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.065 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "max number of daemons per service per host", 2024-05-23T00:10:50.065 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.065 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.065 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.066 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.066 INFO:teuthology.orchestra.run.smithi103.stdout: "max_osd_draining_count": { 2024-05-23T00:10:50.066 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_osd_draining_count", 2024-05-23T00:10:50.066 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.066 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.066 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.066 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:50.067 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.067 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.067 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.067 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-23T00:10:50.067 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.067 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.067 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.067 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.068 INFO:teuthology.orchestra.run.smithi103.stdout: "migration_current": { 2024-05-23T00:10:50.068 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "migration_current", 2024-05-23T00:10:50.068 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.068 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.068 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.068 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.068 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.069 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.069 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.069 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "internal - do not modify", 2024-05-23T00:10:50.069 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.069 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.069 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.069 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.070 INFO:teuthology.orchestra.run.smithi103.stdout: "mode": { 2024-05-23T00:10:50.070 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mode", 2024-05-23T00:10:50.070 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.070 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.070 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.070 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "root", 2024-05-23T00:10:50.070 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.070 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.071 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.071 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm-package", 2024-05-23T00:10:50.071 INFO:teuthology.orchestra.run.smithi103.stdout: "root" 2024-05-23T00:10:50.071 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.071 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "mode for remote execution of cephadm", 2024-05-23T00:10:50.071 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.071 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.072 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.072 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.072 INFO:teuthology.orchestra.run.smithi103.stdout: "oob_default_addr": { 2024-05-23T00:10:50.072 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "oob_default_addr", 2024-05-23T00:10:50.072 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.072 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.072 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.072 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "169.254.1.1", 2024-05-23T00:10:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-23T00:10:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.073 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: "prometheus_alerts_path": { 2024-05-23T00:10:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prometheus_alerts_path", 2024-05-23T00:10:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-23T00:10:50.074 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.075 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.075 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.075 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-23T00:10:50.075 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.075 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.075 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.075 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.075 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_insecure": { 2024-05-23T00:10:50.076 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_insecure", 2024-05-23T00:10:50.076 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.076 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.076 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.076 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.076 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.076 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-23T00:10:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_password": { 2024-05-23T00:10:50.077 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_password", 2024-05-23T00:10:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.078 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.079 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-23T00:10:50.079 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.079 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.079 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.079 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.079 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_url": { 2024-05-23T00:10:50.079 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_url", 2024-05-23T00:10:50.079 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.080 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.080 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.080 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.080 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.080 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.080 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.080 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-23T00:10:50.080 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.081 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.081 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.081 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.081 INFO:teuthology.orchestra.run.smithi103.stdout: "registry_username": { 2024-05-23T00:10:50.081 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "registry_username", 2024-05-23T00:10:50.081 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.081 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.082 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.082 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.082 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.082 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.082 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.082 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-23T00:10:50.082 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.082 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.083 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.083 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.083 INFO:teuthology.orchestra.run.smithi103.stdout: "secure_monitoring_stack": { 2024-05-23T00:10:50.083 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "secure_monitoring_stack", 2024-05-23T00:10:50.083 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.083 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.083 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.083 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.084 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.084 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.084 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.084 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-23T00:10:50.084 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.084 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.084 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.085 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.085 INFO:teuthology.orchestra.run.smithi103.stdout: "service_discovery_port": { 2024-05-23T00:10:50.085 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "service_discovery_port", 2024-05-23T00:10:50.085 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.085 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.085 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.085 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8765", 2024-05-23T00:10:50.085 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.086 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.086 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.086 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "cephadm service discovery port", 2024-05-23T00:10:50.086 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.086 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.086 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.086 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.086 INFO:teuthology.orchestra.run.smithi103.stdout: "ssh_config_file": { 2024-05-23T00:10:50.087 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssh_config_file", 2024-05-23T00:10:50.087 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.087 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.087 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.087 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.087 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.087 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-23T00:10:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: "use_agent": { 2024-05-23T00:10:50.088 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "use_agent", 2024-05-23T00:10:50.089 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.089 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.089 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.089 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.089 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.089 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.089 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.089 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-23T00:10:50.090 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.090 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.090 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.090 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.090 INFO:teuthology.orchestra.run.smithi103.stdout: "use_repo_digest": { 2024-05-23T00:10:50.090 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "use_repo_digest", 2024-05-23T00:10:50.090 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.090 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.091 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.091 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.091 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.091 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.091 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.091 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-23T00:10:50.091 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.091 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.092 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.092 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.092 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_failed_host_check": { 2024-05-23T00:10:50.092 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_failed_host_check", 2024-05-23T00:10:50.092 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.092 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.092 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.092 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.093 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.093 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.093 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.093 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if the host check fails", 2024-05-23T00:10:50.093 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.093 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.093 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.093 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.094 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_stray_daemons": { 2024-05-23T00:10:50.094 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_stray_daemons", 2024-05-23T00:10:50.094 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.094 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.094 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.094 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.094 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.095 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.095 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.095 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-23T00:10:50.095 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.095 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.095 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.095 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.095 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_on_stray_hosts": { 2024-05-23T00:10:50.096 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_on_stray_hosts", 2024-05-23T00:10:50.096 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.096 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.096 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.096 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.096 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.096 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.096 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.097 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-23T00:10:50.097 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.097 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.097 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.097 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.097 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.097 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.098 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.098 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crash", 2024-05-23T00:10:50.098 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.098 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.098 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.098 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.098 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.098 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.099 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.099 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.099 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.099 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.099 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.099 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.099 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.099 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.100 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.101 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.102 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.102 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.102 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.102 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.102 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.102 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.102 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.103 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.103 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.103 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.103 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.103 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.103 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.103 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.103 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.104 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.104 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.104 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.104 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.104 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.104 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.104 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.104 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.105 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.105 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.105 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.105 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.105 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.105 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.105 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.105 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.106 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.106 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.106 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.106 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.106 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.106 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.106 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.107 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.107 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.107 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.107 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.107 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.107 INFO:teuthology.orchestra.run.smithi103.stdout: "retain_interval": { 2024-05-23T00:10:50.107 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "retain_interval", 2024-05-23T00:10:50.107 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.108 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.108 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.108 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "31536000", 2024-05-23T00:10:50.108 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.108 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.108 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.108 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-23T00:10:50.108 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.109 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.109 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.109 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.109 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_recent_interval": { 2024-05-23T00:10:50.109 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_recent_interval", 2024-05-23T00:10:50.109 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.109 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.109 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.110 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1209600", 2024-05-23T00:10:50.110 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.110 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.110 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.110 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-23T00:10:50.110 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.110 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.110 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.111 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.111 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.111 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.111 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.111 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "dashboard", 2024-05-23T00:10:50.111 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.111 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.111 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.112 INFO:teuthology.orchestra.run.smithi103.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-23T00:10:50.112 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-23T00:10:50.112 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.112 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.112 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.112 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:50.112 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.113 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.113 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.113 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.113 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.113 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.113 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.113 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.114 INFO:teuthology.orchestra.run.smithi103.stdout: "ALERTMANAGER_API_HOST": { 2024-05-23T00:10:50.114 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-23T00:10:50.114 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.114 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.114 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.114 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.114 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.114 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.115 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.115 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.115 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.115 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.115 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.115 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.115 INFO:teuthology.orchestra.run.smithi103.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-23T00:10:50.116 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-23T00:10:50.116 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.116 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.116 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.116 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.116 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.116 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.116 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.117 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.117 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.117 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.117 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.117 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.117 INFO:teuthology.orchestra.run.smithi103.stdout: "AUDIT_API_ENABLED": { 2024-05-23T00:10:50.117 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "AUDIT_API_ENABLED", 2024-05-23T00:10:50.117 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.118 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.118 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.118 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.118 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.118 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.118 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.118 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.119 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.119 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.119 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.119 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.119 INFO:teuthology.orchestra.run.smithi103.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-23T00:10:50.119 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-23T00:10:50.119 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.119 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.120 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.120 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.120 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.120 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.120 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.120 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.120 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.120 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: "ENABLE_BROWSABLE_API": { 2024-05-23T00:10:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-23T00:10:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.121 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.122 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.122 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.122 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.122 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.122 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.122 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.122 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.123 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.123 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-23T00:10:50.123 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-23T00:10:50.123 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.123 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.123 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.123 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.123 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.124 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.124 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.124 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.124 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.124 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.124 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.124 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.124 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-23T00:10:50.125 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-23T00:10:50.125 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.125 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.125 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.125 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.125 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.125 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.126 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.126 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.126 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.126 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.126 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.126 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.126 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-23T00:10:50.126 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-23T00:10:50.127 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.127 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.127 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.127 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.127 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.127 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.127 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.127 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.128 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.128 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.128 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.128 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.128 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-23T00:10:50.128 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-23T00:10:50.128 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.128 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.129 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.130 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.130 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.130 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-23T00:10:50.130 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-23T00:10:50.130 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.130 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.130 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.131 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.132 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.132 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-23T00:10:50.132 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-23T00:10:50.132 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.132 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.132 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.132 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.132 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.133 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.133 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.133 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.133 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.133 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.133 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.133 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.134 INFO:teuthology.orchestra.run.smithi103.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-23T00:10:50.134 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-23T00:10:50.134 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.134 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.134 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.134 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.134 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.134 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.135 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.135 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.135 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.135 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.135 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.135 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.135 INFO:teuthology.orchestra.run.smithi103.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-23T00:10:50.135 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-23T00:10:50.137 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.137 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.137 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.137 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.137 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.138 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.138 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.138 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.138 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.138 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.138 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.138 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.138 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_PASSWORD": { 2024-05-23T00:10:50.139 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-23T00:10:50.139 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.139 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.139 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.139 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:50.139 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.139 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.139 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.140 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.140 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.140 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.140 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.140 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.140 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-23T00:10:50.140 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-23T00:10:50.140 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.141 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.141 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.141 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.141 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.141 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.141 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.141 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_URL": { 2024-05-23T00:10:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_URL", 2024-05-23T00:10:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.142 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.143 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.143 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.143 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.143 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.143 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.143 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.143 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.144 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.144 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.144 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.144 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_API_USERNAME": { 2024-05-23T00:10:50.144 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-23T00:10:50.144 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.144 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.144 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.145 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:50.145 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.145 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.145 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.145 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.145 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.145 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.145 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.146 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.146 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-23T00:10:50.146 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-23T00:10:50.146 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.146 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.146 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.146 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.147 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.147 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.147 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.147 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.147 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.147 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.147 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.147 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.148 INFO:teuthology.orchestra.run.smithi103.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-23T00:10:50.148 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-23T00:10:50.148 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.148 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.148 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.148 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.148 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.148 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.149 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.149 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.149 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.149 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.149 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.149 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.149 INFO:teuthology.orchestra.run.smithi103.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-23T00:10:50.149 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-23T00:10:50.150 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.150 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.150 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.150 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.150 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.150 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.150 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.151 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.151 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.151 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.151 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.151 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.151 INFO:teuthology.orchestra.run.smithi103.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-23T00:10:50.151 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-23T00:10:50.151 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.152 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.152 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.152 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.152 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.152 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.152 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.152 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.152 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.153 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.153 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.153 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.153 INFO:teuthology.orchestra.run.smithi103.stdout: "PROMETHEUS_API_HOST": { 2024-05-23T00:10:50.153 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-23T00:10:50.153 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.153 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.153 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.154 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.154 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.154 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.154 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.154 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.154 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.154 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.155 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.155 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.155 INFO:teuthology.orchestra.run.smithi103.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-23T00:10:50.155 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-23T00:10:50.155 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.155 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.155 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.155 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.156 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.156 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.156 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.156 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.156 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.156 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.156 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.156 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-23T00:10:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-23T00:10:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.157 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.158 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.158 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.158 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.158 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.158 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.158 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.158 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-23T00:10:50.158 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-23T00:10:50.159 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.159 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.159 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.159 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.159 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.159 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.159 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.159 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.160 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.160 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.160 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.160 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.160 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-23T00:10:50.160 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-23T00:10:50.160 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.161 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.162 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.162 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.162 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.162 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-23T00:10:50.162 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-23T00:10:50.162 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.162 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.162 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.163 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.164 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.164 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-23T00:10:50.164 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-23T00:10:50.164 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.164 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.164 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.164 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.165 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.165 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.165 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.165 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.165 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.165 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.165 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.165 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.166 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-23T00:10:50.166 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-23T00:10:50.166 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.166 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.166 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.166 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.166 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.167 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-23T00:10:50.168 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-23T00:10:50.168 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.168 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.168 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.168 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.168 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.168 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.169 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.169 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.169 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.169 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.169 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.169 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.169 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_ENABLED": { 2024-05-23T00:10:50.169 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-23T00:10:50.170 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.170 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.170 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.170 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.170 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.170 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.170 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.171 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.171 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.171 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.171 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.171 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.171 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-23T00:10:50.171 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-23T00:10:50.172 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.172 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.172 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.172 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-23T00:10:50.172 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.172 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.172 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.172 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.173 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.173 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.173 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.173 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.173 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-23T00:10:50.173 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-23T00:10:50.173 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.174 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.174 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.174 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:50.174 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.174 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.174 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.174 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.175 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.175 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.175 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.175 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.175 INFO:teuthology.orchestra.run.smithi103.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-23T00:10:50.175 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-23T00:10:50.175 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.176 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.176 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.176 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8", 2024-05-23T00:10:50.176 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.176 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.176 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.176 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.176 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.177 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.177 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.177 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.177 INFO:teuthology.orchestra.run.smithi103.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-23T00:10:50.177 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-23T00:10:50.177 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.177 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.178 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.178 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "45", 2024-05-23T00:10:50.178 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.178 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.178 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.178 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.178 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.178 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.179 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.179 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.179 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_ACCESS_KEY": { 2024-05-23T00:10:50.179 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-23T00:10:50.179 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.179 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.179 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.179 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.180 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.180 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.180 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.180 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.180 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.180 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.180 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.180 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-23T00:10:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-23T00:10:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "admin", 2024-05-23T00:10:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.181 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.182 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.182 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.182 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.182 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.182 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.182 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.182 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_SECRET_KEY": { 2024-05-23T00:10:50.183 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-23T00:10:50.183 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.183 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.183 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.183 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.183 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.183 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.183 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.184 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.184 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.184 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.184 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.184 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.184 INFO:teuthology.orchestra.run.smithi103.stdout: "RGW_API_SSL_VERIFY": { 2024-05-23T00:10:50.184 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-23T00:10:50.184 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.185 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.185 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.185 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.185 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.185 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.185 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.185 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.185 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.186 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.186 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.186 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.186 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-23T00:10:50.186 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-23T00:10:50.186 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.186 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.186 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.187 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:50.187 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.187 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.187 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.187 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.187 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.187 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.188 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.188 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.188 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-23T00:10:50.188 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-23T00:10:50.188 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.188 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.188 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.188 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:50.189 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.189 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.189 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.189 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.189 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.189 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.189 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.189 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.190 INFO:teuthology.orchestra.run.smithi103.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-23T00:10:50.190 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-23T00:10:50.190 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.190 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.190 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.190 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:50.190 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.191 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.191 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.191 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.191 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.191 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.191 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.191 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.191 INFO:teuthology.orchestra.run.smithi103.stdout: "cross_origin_url": { 2024-05-23T00:10:50.192 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cross_origin_url", 2024-05-23T00:10:50.192 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.192 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.192 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.192 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.192 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.192 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.193 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.193 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.193 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.193 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.193 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.193 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.193 INFO:teuthology.orchestra.run.smithi103.stdout: "crt_file": { 2024-05-23T00:10:50.193 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "crt_file", 2024-05-23T00:10:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.194 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.195 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.195 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.195 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.195 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.195 INFO:teuthology.orchestra.run.smithi103.stdout: "debug": { 2024-05-23T00:10:50.195 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "debug", 2024-05-23T00:10:50.195 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.195 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.196 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.196 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.196 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.196 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.196 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.196 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Enable/disable debug options", 2024-05-23T00:10:50.196 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.197 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.197 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.197 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.197 INFO:teuthology.orchestra.run.smithi103.stdout: "jwt_token_ttl": { 2024-05-23T00:10:50.197 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "jwt_token_ttl", 2024-05-23T00:10:50.197 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.197 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.198 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.198 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "28800", 2024-05-23T00:10:50.198 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.198 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.198 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.198 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.198 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.198 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.199 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.199 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.199 INFO:teuthology.orchestra.run.smithi103.stdout: "key_file": { 2024-05-23T00:10:50.199 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "key_file", 2024-05-23T00:10:50.199 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.199 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.199 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.199 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.200 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.200 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.200 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.200 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.200 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.200 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.200 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.201 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.201 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.201 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.201 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.201 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.201 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.201 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.201 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.202 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.202 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.202 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.202 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.202 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.202 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.202 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.203 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.203 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.203 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.203 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.203 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.203 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.203 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.203 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.204 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.204 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.204 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.204 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.204 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.204 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.204 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.205 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.205 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.205 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.205 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.205 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.205 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.205 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.205 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.206 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.206 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.206 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.206 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.206 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.206 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.206 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.207 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.207 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.207 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.207 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.207 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.207 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.207 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.208 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.208 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.208 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.208 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.208 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.208 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.208 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.209 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.209 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.209 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.209 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.209 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.209 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.209 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.209 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.210 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.210 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.210 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.210 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.210 INFO:teuthology.orchestra.run.smithi103.stdout: "motd": { 2024-05-23T00:10:50.210 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "motd", 2024-05-23T00:10:50.210 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.211 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.211 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.211 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.211 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.211 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.211 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.211 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "The message of the day", 2024-05-23T00:10:50.211 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.212 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.212 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.212 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.212 INFO:teuthology.orchestra.run.smithi103.stdout: "redirect_resolve_ip_addr": { 2024-05-23T00:10:50.212 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "redirect_resolve_ip_addr", 2024-05-23T00:10:50.212 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.212 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.213 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.214 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.214 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:50.214 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:50.214 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.214 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.214 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.215 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "::", 2024-05-23T00:10:50.215 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.215 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.215 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.215 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.215 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.215 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.216 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.216 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.216 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:50.216 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:50.216 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.216 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.216 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.216 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8080", 2024-05-23T00:10:50.217 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.217 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.217 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.217 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.217 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.217 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.217 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.218 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl": { 2024-05-23T00:10:50.218 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl", 2024-05-23T00:10:50.218 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.218 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.218 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.218 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.218 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.219 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl_server_port": { 2024-05-23T00:10:50.220 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl_server_port", 2024-05-23T00:10:50.220 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.220 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.220 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.220 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8443", 2024-05-23T00:10:50.220 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.220 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.220 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.221 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.221 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.221 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.221 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.221 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_behaviour": { 2024-05-23T00:10:50.221 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_behaviour", 2024-05-23T00:10:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "redirect", 2024-05-23T00:10:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.222 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.223 INFO:teuthology.orchestra.run.smithi103.stdout: "redirect" 2024-05-23T00:10:50.223 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.223 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.223 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.223 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.223 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.223 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.223 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_error_status_code": { 2024-05-23T00:10:50.224 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_error_status_code", 2024-05-23T00:10:50.224 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.224 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.224 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.224 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "500", 2024-05-23T00:10:50.224 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "400", 2024-05-23T00:10:50.224 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "599", 2024-05-23T00:10:50.225 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.225 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.225 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.225 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.225 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.225 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.225 INFO:teuthology.orchestra.run.smithi103.stdout: "url_prefix": { 2024-05-23T00:10:50.226 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "url_prefix", 2024-05-23T00:10:50.226 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.226 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.226 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.226 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.226 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.226 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.226 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.227 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.227 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.227 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.227 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.227 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.227 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.227 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.228 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.228 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "devicehealth", 2024-05-23T00:10:50.228 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.228 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.228 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.228 INFO:teuthology.orchestra.run.smithi103.stdout: "enable_monitoring": { 2024-05-23T00:10:50.228 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enable_monitoring", 2024-05-23T00:10:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.229 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "monitor device health metrics", 2024-05-23T00:10:50.230 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.230 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.230 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.230 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.230 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.230 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.230 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.231 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.231 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.231 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.231 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.231 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.231 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.231 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.231 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.232 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.232 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.232 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.232 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.232 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.232 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.232 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.233 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.233 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.233 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.233 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.233 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.233 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.233 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.234 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.234 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.234 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.234 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.234 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.234 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.234 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.235 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.235 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.235 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.235 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.235 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.235 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.235 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.235 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.236 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.236 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.236 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.236 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.236 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.236 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.236 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.237 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.238 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.238 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.238 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.238 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.238 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.238 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.238 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.239 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.239 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.239 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.239 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.239 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.239 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.239 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.239 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.240 INFO:teuthology.orchestra.run.smithi103.stdout: "mark_out_threshold": { 2024-05-23T00:10:50.240 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mark_out_threshold", 2024-05-23T00:10:50.240 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.240 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.240 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.240 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "2419200", 2024-05-23T00:10:50.240 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.241 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.241 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.241 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-23T00:10:50.241 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.241 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.241 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.241 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.241 INFO:teuthology.orchestra.run.smithi103.stdout: "pool_name": { 2024-05-23T00:10:50.242 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pool_name", 2024-05-23T00:10:50.242 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.242 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.242 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.242 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "device_health_metrics", 2024-05-23T00:10:50.242 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.242 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.243 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.243 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-23T00:10:50.243 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.243 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.243 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.243 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.243 INFO:teuthology.orchestra.run.smithi103.stdout: "retention_period": { 2024-05-23T00:10:50.243 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "retention_period", 2024-05-23T00:10:50.244 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.244 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.244 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.244 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15552000", 2024-05-23T00:10:50.244 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.244 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.244 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.245 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long to retain device health metrics", 2024-05-23T00:10:50.245 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.245 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.245 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.245 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.245 INFO:teuthology.orchestra.run.smithi103.stdout: "scrape_frequency": { 2024-05-23T00:10:50.245 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "scrape_frequency", 2024-05-23T00:10:50.245 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.246 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.246 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.246 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "86400", 2024-05-23T00:10:50.246 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.246 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.246 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.246 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-23T00:10:50.247 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.247 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.247 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.247 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.247 INFO:teuthology.orchestra.run.smithi103.stdout: "self_heal": { 2024-05-23T00:10:50.247 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "self_heal", 2024-05-23T00:10:50.247 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.247 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.248 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.248 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.248 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.248 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.248 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.248 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-23T00:10:50.248 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.248 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.249 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.249 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.249 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:50.249 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:50.249 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.249 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.249 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.250 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:50.250 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.250 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.250 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.250 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how frequently to wake up and check device health", 2024-05-23T00:10:50.250 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.250 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: "warn_threshold": { 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "warn_threshold", 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "7257600", 2024-05-23T00:10:50.251 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-23T00:10:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.252 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.253 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.253 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.253 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "diskprediction_local", 2024-05-23T00:10:50.253 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.253 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.253 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.253 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.253 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.254 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.254 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.254 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.254 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.254 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.254 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.254 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.255 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.255 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.255 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.255 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.255 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.255 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.255 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.255 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.256 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.256 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.256 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.256 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.256 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.256 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.256 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.256 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.257 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.257 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.257 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.257 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.257 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.257 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.257 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.257 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.258 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.258 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.258 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.258 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.258 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.258 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.258 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.259 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.259 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.259 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.259 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.259 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.259 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.259 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.259 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.260 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.261 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.261 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.261 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.261 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.261 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.261 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.261 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.261 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.262 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.262 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.262 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.262 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.262 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.262 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.262 INFO:teuthology.orchestra.run.smithi103.stdout: "predict_interval": { 2024-05-23T00:10:50.262 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "predict_interval", 2024-05-23T00:10:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "86400", 2024-05-23T00:10:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.263 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.264 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.264 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.264 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.264 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.264 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.264 INFO:teuthology.orchestra.run.smithi103.stdout: "predictor_model": { 2024-05-23T00:10:50.264 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "predictor_model", 2024-05-23T00:10:50.264 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.265 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.265 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.265 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "prophetstor", 2024-05-23T00:10:50.265 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.265 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.265 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.265 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.266 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.266 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.266 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.266 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.266 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:50.266 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:50.266 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.267 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.267 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.267 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "600", 2024-05-23T00:10:50.267 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.267 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.267 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.267 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.268 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.268 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.268 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.268 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.268 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.268 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.268 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.269 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "influx", 2024-05-23T00:10:50.269 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": false, 2024-05-23T00:10:50.269 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "influxdb python module not found", 2024-05-23T00:10:50.269 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.269 INFO:teuthology.orchestra.run.smithi103.stdout: "batch_size": { 2024-05-23T00:10:50.269 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "batch_size", 2024-05-23T00:10:50.269 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.270 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.270 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.270 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5000", 2024-05-23T00:10:50.270 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.270 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.270 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.270 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-23T00:10:50.271 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.271 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.271 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.271 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.271 INFO:teuthology.orchestra.run.smithi103.stdout: "database": { 2024-05-23T00:10:50.271 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "database", 2024-05-23T00:10:50.271 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.271 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.272 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.272 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "ceph", 2024-05-23T00:10:50.272 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.272 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.272 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.272 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-23T00:10:50.272 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.273 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.273 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.273 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.273 INFO:teuthology.orchestra.run.smithi103.stdout: "hostname": { 2024-05-23T00:10:50.273 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "hostname", 2024-05-23T00:10:50.273 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.273 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.274 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.274 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.274 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.274 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.274 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.274 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB server hostname", 2024-05-23T00:10:50.274 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.275 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.275 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.275 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.275 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:50.275 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:50.275 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.275 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.276 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.276 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "30", 2024-05-23T00:10:50.276 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "5", 2024-05-23T00:10:50.276 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.276 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.276 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-23T00:10:50.276 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.276 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.277 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.278 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.278 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.278 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.278 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.278 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.278 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.278 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.279 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.280 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.280 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.280 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.280 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.280 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.280 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.280 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.281 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.281 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.281 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.281 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.281 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.281 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.281 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.281 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.282 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.282 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.282 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.282 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.282 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.282 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.282 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.283 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.283 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.283 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.283 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.283 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.283 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.283 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.283 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.284 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.285 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.285 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.285 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.285 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.285 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.285 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.285 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.285 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.286 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.286 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.286 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.286 INFO:teuthology.orchestra.run.smithi103.stdout: "password": { 2024-05-23T00:10:50.286 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "password", 2024-05-23T00:10:50.286 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.286 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.287 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.287 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.287 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.287 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.287 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.287 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "password of InfluxDB server user", 2024-05-23T00:10:50.287 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.287 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: "port": { 2024-05-23T00:10:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "port", 2024-05-23T00:10:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.288 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "8086", 2024-05-23T00:10:50.289 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.289 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.289 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.289 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "InfluxDB server port", 2024-05-23T00:10:50.289 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.289 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.289 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.289 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.290 INFO:teuthology.orchestra.run.smithi103.stdout: "ssl": { 2024-05-23T00:10:50.290 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ssl", 2024-05-23T00:10:50.290 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.290 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.290 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.290 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "false", 2024-05-23T00:10:50.290 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.290 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.291 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.291 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-23T00:10:50.291 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.291 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.291 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.291 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.291 INFO:teuthology.orchestra.run.smithi103.stdout: "threads": { 2024-05-23T00:10:50.291 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "threads", 2024-05-23T00:10:50.292 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.292 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.292 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.292 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:50.292 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:50.292 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "32", 2024-05-23T00:10:50.292 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.292 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-23T00:10:50.293 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.293 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.293 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.293 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.293 INFO:teuthology.orchestra.run.smithi103.stdout: "username": { 2024-05-23T00:10:50.293 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "username", 2024-05-23T00:10:50.293 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.294 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.294 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.294 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.294 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.294 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.294 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.294 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "username of InfluxDB server user", 2024-05-23T00:10:50.294 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.295 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.295 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.295 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.295 INFO:teuthology.orchestra.run.smithi103.stdout: "verify_ssl": { 2024-05-23T00:10:50.295 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "verify_ssl", 2024-05-23T00:10:50.295 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.295 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.295 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.296 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "true", 2024-05-23T00:10:50.296 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.296 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.296 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.296 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-23T00:10:50.296 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.296 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.296 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.297 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.297 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.297 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.297 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.297 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "insights", 2024-05-23T00:10:50.297 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.297 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.298 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.298 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.298 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.298 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.298 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.298 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.298 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.298 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.299 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.299 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.299 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.299 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.299 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.299 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.299 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.299 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.300 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.300 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.300 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.300 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.300 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.300 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.300 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.300 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.301 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.301 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.301 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.301 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.301 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.301 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.301 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.301 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.302 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.302 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.302 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.302 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.302 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.302 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.302 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.302 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.303 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.303 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.303 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.303 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.303 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.303 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.303 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.303 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.304 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.304 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.304 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.304 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.304 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.304 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.304 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.304 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.305 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.305 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.305 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.305 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.305 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.305 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.305 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.305 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.306 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.306 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.306 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.306 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.306 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.306 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.306 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.307 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.307 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.307 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.307 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "iostat", 2024-05-23T00:10:50.307 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.307 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.307 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.307 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.308 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.308 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.308 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.308 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.308 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.308 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.308 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.308 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.309 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.309 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.309 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.309 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.309 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.309 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.309 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.309 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.310 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.311 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.311 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.311 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.311 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.311 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.311 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.311 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.311 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.312 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.312 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.312 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.312 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.312 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.312 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.312 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.313 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.313 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.313 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.313 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.313 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.313 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.313 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.313 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.314 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.314 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.314 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.314 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.314 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.314 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.314 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.314 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.315 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.315 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.315 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.315 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.315 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.315 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.315 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.315 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.316 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.316 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.316 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.316 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.316 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.316 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.316 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.316 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.317 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.317 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "k8sevents", 2024-05-23T00:10:50.317 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.317 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.317 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.317 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_event_retention_days": { 2024-05-23T00:10:50.317 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "ceph_event_retention_days", 2024-05-23T00:10:50.317 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.318 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.318 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.318 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "7", 2024-05-23T00:10:50.318 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.318 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.318 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.318 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-23T00:10:50.318 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.319 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.319 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.319 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.319 INFO:teuthology.orchestra.run.smithi103.stdout: "config_check_secs": { 2024-05-23T00:10:50.319 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "config_check_secs", 2024-05-23T00:10:50.319 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.319 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.319 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.320 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:50.320 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "10", 2024-05-23T00:10:50.320 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.320 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.320 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-23T00:10:50.320 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.320 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.321 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.322 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.322 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.322 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.322 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.322 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.322 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.322 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.322 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.323 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.323 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.323 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.323 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.323 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.323 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.323 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.324 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.324 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.324 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.324 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.324 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.324 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.324 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.324 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.325 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.325 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.325 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.325 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.325 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.325 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.325 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.325 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.326 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.326 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.326 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.326 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.326 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.326 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.326 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.327 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.327 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.327 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.327 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.327 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.327 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.327 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.327 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.328 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.328 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.328 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.328 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.328 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.328 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.328 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.328 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.329 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.329 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.329 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.329 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.329 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.329 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.329 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.329 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.330 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.330 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.330 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.330 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.330 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.330 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "localpool", 2024-05-23T00:10:50.330 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.331 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.331 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.331 INFO:teuthology.orchestra.run.smithi103.stdout: "failure_domain": { 2024-05-23T00:10:50.331 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "failure_domain", 2024-05-23T00:10:50.331 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.331 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.331 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.331 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "host", 2024-05-23T00:10:50.332 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.332 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.332 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.332 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "failure domain for any created local pool", 2024-05-23T00:10:50.332 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-23T00:10:50.332 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.332 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.332 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.333 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.333 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.333 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.333 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.333 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.333 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.333 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.333 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.334 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.334 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.334 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.334 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.334 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.334 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.334 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.334 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.335 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.335 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.335 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.335 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.335 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.335 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.335 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.335 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.336 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.336 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.336 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.336 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.336 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.336 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.336 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.337 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.338 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.338 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.338 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.338 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.338 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.338 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.338 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.339 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.339 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.339 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.341 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.341 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.341 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.341 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.341 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.342 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.342 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.342 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.342 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.342 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.342 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.342 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.343 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.344 INFO:teuthology.orchestra.run.smithi103.stdout: "min_size": { 2024-05-23T00:10:50.344 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "min_size", 2024-05-23T00:10:50.344 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.344 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.344 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.344 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.344 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.345 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.345 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.345 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default min_size for any created local pool", 2024-05-23T00:10:50.345 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-23T00:10:50.345 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.345 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.345 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.345 INFO:teuthology.orchestra.run.smithi103.stdout: "num_rep": { 2024-05-23T00:10:50.346 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "num_rep", 2024-05-23T00:10:50.346 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.346 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.346 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.346 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3", 2024-05-23T00:10:50.346 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.346 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.346 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.347 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default replica count for any created local pool", 2024-05-23T00:10:50.347 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.347 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.347 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.347 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_num": { 2024-05-23T00:10:50.347 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pg_num", 2024-05-23T00:10:50.347 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.348 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.348 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.348 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "128", 2024-05-23T00:10:50.348 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.348 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.348 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.348 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "default pg_num for any created local pool", 2024-05-23T00:10:50.349 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.349 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.349 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.349 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.349 INFO:teuthology.orchestra.run.smithi103.stdout: "prefix": { 2024-05-23T00:10:50.349 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prefix", 2024-05-23T00:10:50.349 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.349 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.350 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.350 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.350 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.350 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.350 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.350 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "name prefix for any created local pool", 2024-05-23T00:10:50.350 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.350 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.351 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.351 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.351 INFO:teuthology.orchestra.run.smithi103.stdout: "subtree": { 2024-05-23T00:10:50.351 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "subtree", 2024-05-23T00:10:50.351 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.351 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.351 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.351 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "rack", 2024-05-23T00:10:50.352 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.352 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.352 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.352 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-23T00:10:50.352 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-23T00:10:50.352 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.352 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.353 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.353 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.353 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.353 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.353 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mds_autoscaler", 2024-05-23T00:10:50.353 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.353 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.353 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.354 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.354 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.354 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.354 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.354 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.354 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.354 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.355 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.355 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.355 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.355 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.355 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.355 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.355 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.355 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.356 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.356 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.356 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.356 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.356 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.356 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.356 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.357 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.358 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.358 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.358 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.358 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.358 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.358 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.358 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.358 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.359 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.360 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.361 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.361 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.361 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.361 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.361 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.361 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.361 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.362 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.363 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.363 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.363 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.363 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.363 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mirroring", 2024-05-23T00:10:50.363 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.363 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.364 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.364 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.364 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.364 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.364 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.364 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.364 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.364 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.365 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.365 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.365 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.365 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.365 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.365 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.365 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.366 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.367 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.367 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.367 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.367 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.367 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.367 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.367 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.367 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.368 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.368 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.368 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.368 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.368 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.368 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.368 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.368 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.369 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.369 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.369 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.369 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.369 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.369 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.369 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.369 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.370 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.370 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.370 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.370 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.370 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.370 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.370 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.371 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.371 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.371 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.371 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.371 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.371 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.371 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.371 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.372 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "nfs", 2024-05-23T00:10:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.373 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.374 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.375 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.376 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.377 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.378 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.379 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.380 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.381 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.382 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "orchestrator", 2024-05-23T00:10:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.383 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "fail_fs": { 2024-05-23T00:10:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "fail_fs", 2024-05-23T00:10:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.384 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-23T00:10:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.385 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.386 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.387 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.388 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.389 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.390 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.391 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.392 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.393 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.394 INFO:teuthology.orchestra.run.smithi103.stdout: "orchestrator": { 2024-05-23T00:10:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "orchestrator", 2024-05-23T00:10:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.395 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "cephadm", 2024-05-23T00:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "rook", 2024-05-23T00:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "test_orchestrator" 2024-05-23T00:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Orchestrator backend", 2024-05-23T00:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.396 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "osd_perf_query", 2024-05-23T00:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.397 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.398 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.399 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.400 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.401 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.402 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.403 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.404 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.405 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.406 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "osd_support", 2024-05-23T00:10:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.407 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.408 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.409 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.410 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.411 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.412 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.413 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.414 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.415 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "pg_autoscaler", 2024-05-23T00:10:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.417 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.418 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.419 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.420 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.421 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.422 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.423 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.424 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.425 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.426 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.427 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "threshold": { 2024-05-23T00:10:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "threshold", 2024-05-23T00:10:50.428 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "3.0", 2024-05-23T00:10:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1.0", 2024-05-23T00:10:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "scaling threshold", 2024-05-23T00:10:50.429 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-23T00:10:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "progress", 2024-05-23T00:10:50.430 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_pg_recovery_event": { 2024-05-23T00:10:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_pg_recovery_event", 2024-05-23T00:10:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.431 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-23T00:10:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.432 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": { 2024-05-23T00:10:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enabled", 2024-05-23T00:10:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.433 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.434 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.435 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.436 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.437 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.438 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.439 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.440 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.441 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.442 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.443 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "max_completed_events": { 2024-05-23T00:10:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_completed_events", 2024-05-23T00:10:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "50", 2024-05-23T00:10:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.444 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "number of past completed events to remember", 2024-05-23T00:10:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "sleep_interval": { 2024-05-23T00:10:50.445 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "sleep_interval", 2024-05-23T00:10:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "5", 2024-05-23T00:10:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.446 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "how long the module is going to sleep", 2024-05-23T00:10:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.447 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "prometheus", 2024-05-23T00:10:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "cache": { 2024-05-23T00:10:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "cache", 2024-05-23T00:10:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.448 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.449 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "exclude_perf_counters": { 2024-05-23T00:10:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "exclude_perf_counters", 2024-05-23T00:10:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.450 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-23T00:10:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-23T00:10:50.451 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.452 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.453 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.454 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.455 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.456 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.457 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.458 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.459 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.460 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_stats_pools": { 2024-05-23T00:10:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_stats_pools", 2024-05-23T00:10:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.461 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.462 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-23T00:10:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-23T00:10:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.463 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "300", 2024-05-23T00:10:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.464 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "scrape_interval": { 2024-05-23T00:10:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "scrape_interval", 2024-05-23T00:10:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.465 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15.0", 2024-05-23T00:10:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.466 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "::", 2024-05-23T00:10:50.467 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-23T00:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.468 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "9283", 2024-05-23T00:10:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.469 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-23T00:10:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "stale_cache_strategy": { 2024-05-23T00:10:50.470 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "stale_cache_strategy", 2024-05-23T00:10:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "log", 2024-05-23T00:10:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.471 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_behaviour": { 2024-05-23T00:10:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_behaviour", 2024-05-23T00:10:50.472 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "default", 2024-05-23T00:10:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "default", 2024-05-23T00:10:50.473 INFO:teuthology.orchestra.run.smithi103.stdout: "error" 2024-05-23T00:10:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "standby_error_status_code": { 2024-05-23T00:10:50.474 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "standby_error_status_code", 2024-05-23T00:10:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "500", 2024-05-23T00:10:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "400", 2024-05-23T00:10:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "599", 2024-05-23T00:10:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.475 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.476 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_support", 2024-05-23T00:10:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.477 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.478 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.479 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.480 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.481 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.482 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.483 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.484 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.485 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "max_concurrent_snap_create": { 2024-05-23T00:10:50.486 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_concurrent_snap_create", 2024-05-23T00:10:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10", 2024-05-23T00:10:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.487 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "mirror_snapshot_schedule": { 2024-05-23T00:10:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "mirror_snapshot_schedule", 2024-05-23T00:10:50.488 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.489 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.490 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.490 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.490 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.490 INFO:teuthology.orchestra.run.smithi103.stdout: "trash_purge_schedule": { 2024-05-23T00:10:50.490 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "trash_purge_schedule", 2024-05-23T00:10:50.490 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.490 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.491 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.491 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.491 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.491 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.491 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.491 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.491 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.491 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.492 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.492 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.492 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.492 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.492 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.492 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "restful", 2024-05-23T00:10:50.492 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.492 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout: "enable_auth": { 2024-05-23T00:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enable_auth", 2024-05-23T00:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.493 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.494 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.494 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.494 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.494 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.494 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.494 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.494 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.495 INFO:teuthology.orchestra.run.smithi103.stdout: "key_file": { 2024-05-23T00:10:50.495 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "key_file", 2024-05-23T00:10:50.495 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.495 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.495 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.495 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.495 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.495 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.496 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.496 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.496 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.496 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.496 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.496 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.496 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.496 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.497 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.498 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.498 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.498 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.498 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.498 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.498 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.498 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.499 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.500 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.500 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.500 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.500 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.500 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.500 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.500 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.500 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.501 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.501 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.501 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.501 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.501 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.501 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.501 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.501 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.502 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.502 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.502 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.502 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.502 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.502 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.502 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.503 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.503 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.503 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.503 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.503 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.503 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.503 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.503 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.504 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.504 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.504 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.504 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.504 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.504 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.504 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.505 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.505 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.505 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.505 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.505 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.505 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.505 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.505 INFO:teuthology.orchestra.run.smithi103.stdout: "server_addr": { 2024-05-23T00:10:50.506 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_addr", 2024-05-23T00:10:50.506 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.506 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.506 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.506 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.506 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.506 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: "server_port": { 2024-05-23T00:10:50.507 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "server_port", 2024-05-23T00:10:50.508 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.508 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.508 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.508 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.508 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.508 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.508 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.508 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.509 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.509 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.509 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.509 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.509 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.509 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.509 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rgw", 2024-05-23T00:10:50.510 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.510 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.510 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.510 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.510 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.510 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.510 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.511 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.511 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.511 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.511 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.511 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.511 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.511 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.511 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.512 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.512 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.512 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.512 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.512 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.512 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.512 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.512 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.513 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.513 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.513 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.513 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.513 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.513 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.513 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.513 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.514 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.514 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.514 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.514 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.514 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.514 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.514 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.514 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.515 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.515 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.515 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.515 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.515 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.515 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.515 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.516 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.516 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.516 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.516 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.516 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.516 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.516 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.516 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.517 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.518 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.518 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.518 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.518 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.518 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.518 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.518 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.519 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rook", 2024-05-23T00:10:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.520 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.521 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.521 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.521 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.521 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.521 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.521 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.521 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.521 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.522 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.522 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.522 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.522 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.522 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.522 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.522 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.522 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.523 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.523 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.523 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.523 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.523 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.523 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.523 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.523 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.524 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.524 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.524 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.524 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.524 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.524 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.524 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.524 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.525 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.525 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.525 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.525 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.525 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.525 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.525 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.525 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.526 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.526 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.526 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.526 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.526 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.526 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.526 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.527 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.527 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.527 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.527 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.527 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.527 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.527 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.527 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.528 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.528 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.528 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.528 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.528 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.528 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.528 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: "storage_class": { 2024-05-23T00:10:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "storage_class", 2024-05-23T00:10:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.529 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.530 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "local", 2024-05-23T00:10:50.530 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.530 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.530 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.530 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-23T00:10:50.530 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.530 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.530 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.531 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.531 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.531 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.531 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.531 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "selftest", 2024-05-23T00:10:50.531 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.531 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.532 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.532 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.532 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.532 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.532 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.532 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.532 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.532 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.533 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.533 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.533 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.533 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.533 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.533 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.533 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.534 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.534 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.534 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.534 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.534 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.534 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.534 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.534 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.535 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.535 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.535 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.535 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.535 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.535 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.535 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.536 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.536 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.536 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.536 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.536 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.536 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.536 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.536 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.537 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.537 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.537 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.537 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.537 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.537 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.537 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.537 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.538 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.538 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.538 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.538 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.538 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.538 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.538 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.538 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.539 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.539 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.539 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.539 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.539 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.539 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.539 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.539 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.540 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.540 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.540 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.540 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.540 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.540 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.540 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.540 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.541 INFO:teuthology.orchestra.run.smithi103.stdout: "roption1": { 2024-05-23T00:10:50.541 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "roption1", 2024-05-23T00:10:50.541 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.541 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.541 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.541 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.542 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.542 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.542 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.542 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.542 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.542 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.542 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.543 INFO:teuthology.orchestra.run.smithi103.stdout: "roption2": { 2024-05-23T00:10:50.543 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "roption2", 2024-05-23T00:10:50.543 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.543 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.545 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.545 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "xyz", 2024-05-23T00:10:50.545 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.545 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.546 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.546 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.546 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.546 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.546 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.546 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption1": { 2024-05-23T00:10:50.547 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption1", 2024-05-23T00:10:50.547 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.547 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.547 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.547 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.547 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.547 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.547 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.548 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.548 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.548 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.548 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.548 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.548 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption2": { 2024-05-23T00:10:50.548 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption2", 2024-05-23T00:10:50.548 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.549 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.549 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.549 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.549 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.549 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.549 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.549 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.549 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.550 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.550 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.550 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.550 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption3": { 2024-05-23T00:10:50.550 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption3", 2024-05-23T00:10:50.550 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "float", 2024-05-23T00:10:50.550 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.551 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.551 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.551 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.551 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.551 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.551 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.551 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.552 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.552 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.552 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption4": { 2024-05-23T00:10:50.552 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption4", 2024-05-23T00:10:50.552 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.552 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.552 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.553 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.554 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption5": { 2024-05-23T00:10:50.554 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption5", 2024-05-23T00:10:50.554 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.554 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.554 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.554 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.555 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.555 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.555 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.555 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.555 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.555 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.555 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.555 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.556 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption6": { 2024-05-23T00:10:50.556 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption6", 2024-05-23T00:10:50.556 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.556 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.556 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.556 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.556 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.556 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: "rwoption7": { 2024-05-23T00:10:50.557 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rwoption7", 2024-05-23T00:10:50.558 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.558 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.558 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.558 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.558 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "1", 2024-05-23T00:10:50.558 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "42", 2024-05-23T00:10:50.558 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.559 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.559 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.559 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.559 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.559 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.559 INFO:teuthology.orchestra.run.smithi103.stdout: "testkey": { 2024-05-23T00:10:50.559 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testkey", 2024-05-23T00:10:50.559 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.560 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.560 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.560 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.560 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.560 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.560 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.560 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.560 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.561 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.561 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.561 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.561 INFO:teuthology.orchestra.run.smithi103.stdout: "testlkey": { 2024-05-23T00:10:50.561 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testlkey", 2024-05-23T00:10:50.561 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.561 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.561 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.562 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.562 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.562 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.562 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.562 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.562 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.562 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.563 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.563 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.563 INFO:teuthology.orchestra.run.smithi103.stdout: "testnewline": { 2024-05-23T00:10:50.563 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "testnewline", 2024-05-23T00:10:50.563 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.563 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.563 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.563 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.564 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.564 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.564 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.564 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.564 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.564 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.564 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.564 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.565 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.565 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.565 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.565 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snap_schedule", 2024-05-23T00:10:50.565 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.565 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.565 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.565 INFO:teuthology.orchestra.run.smithi103.stdout: "allow_m_granularity": { 2024-05-23T00:10:50.566 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "allow_m_granularity", 2024-05-23T00:10:50.566 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.566 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.566 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.566 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.566 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.566 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.566 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.567 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "allow minute scheduled snapshots", 2024-05-23T00:10:50.567 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.567 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.567 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.567 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.567 INFO:teuthology.orchestra.run.smithi103.stdout: "dump_on_update": { 2024-05-23T00:10:50.567 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "dump_on_update", 2024-05-23T00:10:50.567 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.568 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.568 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.568 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.568 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.568 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.568 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.568 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "dump database to debug log on update", 2024-05-23T00:10:50.568 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.569 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.569 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.569 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.569 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.569 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.569 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.569 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.570 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.570 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.570 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.570 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.570 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.570 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.570 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.570 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.571 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.571 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.571 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.571 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.571 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.571 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.571 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.571 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.572 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.572 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.572 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.572 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.572 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.572 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.573 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.573 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.573 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.573 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.573 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.573 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.573 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.574 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.574 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.574 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.574 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.574 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.574 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.574 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.575 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.575 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.575 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.575 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.575 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.575 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.575 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.576 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.576 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.576 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.576 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.576 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.576 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.576 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.576 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.577 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.577 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.577 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.577 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.577 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.577 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.577 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.578 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "stats", 2024-05-23T00:10:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.579 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.580 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.580 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.580 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.580 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.580 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.580 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.580 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.581 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.581 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.581 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.581 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.581 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.581 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.581 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.581 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.582 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.582 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.582 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.582 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.582 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.582 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.582 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.583 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.583 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.583 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.583 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.583 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.583 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.583 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.583 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.584 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.584 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.584 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.584 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.584 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.584 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.585 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.585 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.585 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.585 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.585 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.585 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.585 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.585 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.586 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.586 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.586 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.586 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.586 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.586 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.586 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.587 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.587 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.587 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.587 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.587 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.587 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.587 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.588 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.588 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.588 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.588 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.588 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.588 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.588 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.589 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.589 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.589 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "status", 2024-05-23T00:10:50.589 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.589 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.589 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.589 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.590 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.590 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.590 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.590 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.590 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.590 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.590 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.591 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.591 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.591 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.591 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.591 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.591 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.591 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.591 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.592 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.593 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.593 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.593 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.593 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.593 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.593 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.593 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.593 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.594 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.594 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.594 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.594 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.594 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.594 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.595 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.595 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.595 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.595 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.595 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.595 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.595 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.595 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.596 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.596 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.596 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.596 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.596 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.596 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.596 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.596 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.597 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.597 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.597 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.597 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.597 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.597 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.598 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.598 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.598 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.598 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.598 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.598 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.598 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.598 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.599 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.599 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.599 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "telegraf", 2024-05-23T00:10:50.599 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.599 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.599 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.599 INFO:teuthology.orchestra.run.smithi103.stdout: "address": { 2024-05-23T00:10:50.600 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "address", 2024-05-23T00:10:50.600 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.600 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.600 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.600 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-23T00:10:50.600 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.600 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.600 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.601 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.601 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.601 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.601 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.601 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:50.601 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:50.602 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.602 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.602 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.602 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "15", 2024-05-23T00:10:50.602 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.602 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.602 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.602 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.603 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.604 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.604 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.604 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.604 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.604 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.604 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.604 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.604 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.605 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.605 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.605 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.605 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.605 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.605 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.605 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.605 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.606 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.607 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.608 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.609 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.609 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.609 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.609 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.609 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.609 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.609 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.609 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.610 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.610 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.610 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.610 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.610 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.610 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.610 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.610 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.611 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.611 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.611 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.611 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.611 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.611 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.611 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.611 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.612 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.612 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.612 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.612 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.612 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.612 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "telemetry", 2024-05-23T00:10:50.613 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.613 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.613 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.613 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_basic": { 2024-05-23T00:10:50.613 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_basic", 2024-05-23T00:10:50.613 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.613 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.613 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.614 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.614 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.614 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.614 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.614 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-23T00:10:50.614 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.614 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.615 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.615 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_crash": { 2024-05-23T00:10:50.615 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_crash", 2024-05-23T00:10:50.615 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.615 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.615 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.615 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-23T00:10:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.617 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_device": { 2024-05-23T00:10:50.617 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_device", 2024-05-23T00:10:50.617 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.617 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.617 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.617 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.617 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.617 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.618 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.618 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-23T00:10:50.618 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.618 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.618 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.618 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.618 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_ident": { 2024-05-23T00:10:50.618 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_ident", 2024-05-23T00:10:50.619 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.619 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.619 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.619 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.619 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.619 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.619 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.619 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-23T00:10:50.620 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.620 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.620 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.620 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.620 INFO:teuthology.orchestra.run.smithi103.stdout: "channel_perf": { 2024-05-23T00:10:50.620 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "channel_perf", 2024-05-23T00:10:50.620 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.620 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.621 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.621 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.621 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.621 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.621 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.621 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-23T00:10:50.621 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.622 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.622 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.622 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.622 INFO:teuthology.orchestra.run.smithi103.stdout: "contact": { 2024-05-23T00:10:50.622 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "contact", 2024-05-23T00:10:50.622 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.622 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.622 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.623 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.623 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.623 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.623 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.623 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.623 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.623 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.624 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.624 INFO:teuthology.orchestra.run.smithi103.stdout: "description": { 2024-05-23T00:10:50.624 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "description", 2024-05-23T00:10:50.624 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.624 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.624 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.624 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.625 INFO:teuthology.orchestra.run.smithi103.stdout: "device_url": { 2024-05-23T00:10:50.626 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "device_url", 2024-05-23T00:10:50.626 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.626 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.626 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.626 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-23T00:10:50.626 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.626 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.626 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.627 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.627 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.627 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.627 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.627 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.627 INFO:teuthology.orchestra.run.smithi103.stdout: "enabled": { 2024-05-23T00:10:50.627 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "enabled", 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.628 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.629 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.629 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.629 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.629 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:50.629 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:50.629 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.629 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.629 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "24", 2024-05-23T00:10:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "8", 2024-05-23T00:10:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.630 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.631 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.631 INFO:teuthology.orchestra.run.smithi103.stdout: "last_opt_revision": { 2024-05-23T00:10:50.631 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "last_opt_revision", 2024-05-23T00:10:50.631 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.631 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.631 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.631 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "1", 2024-05-23T00:10:50.631 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "leaderboard": { 2024-05-23T00:10:50.632 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "leaderboard", 2024-05-23T00:10:50.633 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.633 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.633 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.633 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.633 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.633 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.633 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.633 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.634 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.634 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.634 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.634 INFO:teuthology.orchestra.run.smithi103.stdout: "leaderboard_description": { 2024-05-23T00:10:50.634 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "leaderboard_description", 2024-05-23T00:10:50.634 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.635 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.636 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.636 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.636 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.636 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.636 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.636 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.636 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.637 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.637 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.637 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.637 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.637 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.637 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.637 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.637 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.638 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.638 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.638 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.638 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.638 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.638 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.638 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.639 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.640 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.640 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.640 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.640 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.640 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.640 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.640 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.640 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.641 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.641 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.641 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.641 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.641 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.641 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.641 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.641 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.642 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.642 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.642 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.642 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.642 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.642 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.642 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.643 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.643 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.643 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.643 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.643 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.643 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.643 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.643 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.644 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.644 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.644 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.644 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.644 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.644 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.644 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.644 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.645 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.645 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.645 INFO:teuthology.orchestra.run.smithi103.stdout: "organization": { 2024-05-23T00:10:50.645 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "organization", 2024-05-23T00:10:50.645 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.645 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.645 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.645 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.646 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.646 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.646 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.646 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.646 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.646 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.646 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.647 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.647 INFO:teuthology.orchestra.run.smithi103.stdout: "proxy": { 2024-05-23T00:10:50.647 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "proxy", 2024-05-23T00:10:50.647 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.647 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.647 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.647 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.647 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: "url": { 2024-05-23T00:10:50.648 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "url", 2024-05-23T00:10:50.649 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.649 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.649 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.649 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-23T00:10:50.649 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.649 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.649 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.650 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.650 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.650 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.650 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.650 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.650 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.650 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.651 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.651 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "test_orchestrator", 2024-05-23T00:10:50.651 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.651 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.651 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.651 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.651 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.651 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.652 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.653 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.653 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.653 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.653 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.653 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.653 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.653 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.653 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.654 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.654 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.654 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.654 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.654 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.654 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.654 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.655 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.655 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.655 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.655 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.655 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.655 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.655 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.655 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.656 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.656 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.656 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.656 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.656 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.656 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.656 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.656 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.657 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.658 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.658 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.658 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.658 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.658 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.658 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.658 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.659 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.659 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.659 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.659 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.659 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.659 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.659 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.659 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "volumes", 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.660 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.661 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.661 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.661 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.661 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.661 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.661 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.661 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.661 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.662 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.662 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.662 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.662 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.662 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.662 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.662 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.662 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.663 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.664 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.664 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.664 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.664 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.664 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.664 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.664 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.664 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.665 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.665 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.665 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.665 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.665 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.665 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.665 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.665 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.666 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.666 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.666 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.666 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.666 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.666 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.667 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.668 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.668 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.668 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.668 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.668 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.668 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.668 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.668 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: "max_concurrent_clones": { 2024-05-23T00:10:50.669 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "max_concurrent_clones", 2024-05-23T00:10:50.670 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.670 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.670 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.670 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "4", 2024-05-23T00:10:50.670 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.670 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.670 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: "periodic_async_work": { 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "periodic_async_work", 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.671 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.672 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.672 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.672 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.672 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.672 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.672 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Periodically check for async work", 2024-05-23T00:10:50.672 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.672 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.673 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.673 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.673 INFO:teuthology.orchestra.run.smithi103.stdout: "snapshot_clone_delay": { 2024-05-23T00:10:50.673 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snapshot_clone_delay", 2024-05-23T00:10:50.673 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.673 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.673 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.673 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "0", 2024-05-23T00:10:50.674 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.674 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.674 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.674 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-23T00:10:50.674 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.674 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.674 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: "snapshot_clone_no_wait": { 2024-05-23T00:10:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snapshot_clone_no_wait", 2024-05-23T00:10:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "True", 2024-05-23T00:10:50.675 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-23T00:10:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.676 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix", 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: "can_run": true, 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: "error_string": "", 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: "module_options": { 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: "discovery_interval": { 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "discovery_interval", 2024-05-23T00:10:50.677 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "uint", 2024-05-23T00:10:50.678 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.678 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.678 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "100", 2024-05-23T00:10:50.678 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.678 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.678 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.678 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.678 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.679 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.679 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.679 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.679 INFO:teuthology.orchestra.run.smithi103.stdout: "identifier": { 2024-05-23T00:10:50.679 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "identifier", 2024-05-23T00:10:50.679 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.679 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.679 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.680 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: "interval": { 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "interval", 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "secs", 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "60", 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.681 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: "log_level": { 2024-05-23T00:10:50.682 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_level", 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.683 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.684 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.685 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.685 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster": { 2024-05-23T00:10:50.685 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster", 2024-05-23T00:10:50.685 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.685 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.685 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.685 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.686 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.686 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.686 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.686 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.686 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.686 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.686 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.686 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_cluster_level": { 2024-05-23T00:10:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_cluster_level", 2024-05-23T00:10:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "info", 2024-05-23T00:10:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.687 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.688 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [ 2024-05-23T00:10:50.688 INFO:teuthology.orchestra.run.smithi103.stdout: "", 2024-05-23T00:10:50.688 INFO:teuthology.orchestra.run.smithi103.stdout: "critical", 2024-05-23T00:10:50.688 INFO:teuthology.orchestra.run.smithi103.stdout: "debug", 2024-05-23T00:10:50.688 INFO:teuthology.orchestra.run.smithi103.stdout: "error", 2024-05-23T00:10:50.688 INFO:teuthology.orchestra.run.smithi103.stdout: "info", 2024-05-23T00:10:50.688 INFO:teuthology.orchestra.run.smithi103.stdout: "warning" 2024-05-23T00:10:50.688 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.689 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.689 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.689 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.689 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.689 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.689 INFO:teuthology.orchestra.run.smithi103.stdout: "log_to_file": { 2024-05-23T00:10:50.689 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "log_to_file", 2024-05-23T00:10:50.689 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "bool", 2024-05-23T00:10:50.690 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.690 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 1, 2024-05-23T00:10:50.690 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "False", 2024-05-23T00:10:50.690 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.690 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.690 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.690 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.691 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.691 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.691 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.691 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.691 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_host": { 2024-05-23T00:10:50.691 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_host", 2024-05-23T00:10:50.691 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.691 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.692 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.692 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "", 2024-05-23T00:10:50.692 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.692 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.692 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.692 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.692 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.692 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.693 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.693 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.693 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_port": { 2024-05-23T00:10:50.693 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_port", 2024-05-23T00:10:50.693 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "int", 2024-05-23T00:10:50.693 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.693 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.693 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "10051", 2024-05-23T00:10:50.694 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.694 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.694 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.694 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.694 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.694 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.694 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.694 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.695 INFO:teuthology.orchestra.run.smithi103.stdout: "zabbix_sender": { 2024-05-23T00:10:50.695 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "zabbix_sender", 2024-05-23T00:10:50.695 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "str", 2024-05-23T00:10:50.695 INFO:teuthology.orchestra.run.smithi103.stdout: "level": "advanced", 2024-05-23T00:10:50.695 INFO:teuthology.orchestra.run.smithi103.stdout: "flags": 0, 2024-05-23T00:10:50.695 INFO:teuthology.orchestra.run.smithi103.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-23T00:10:50.695 INFO:teuthology.orchestra.run.smithi103.stdout: "min": "", 2024-05-23T00:10:50.695 INFO:teuthology.orchestra.run.smithi103.stdout: "max": "", 2024-05-23T00:10:50.696 INFO:teuthology.orchestra.run.smithi103.stdout: "enum_allowed": [], 2024-05-23T00:10:50.696 INFO:teuthology.orchestra.run.smithi103.stdout: "desc": "", 2024-05-23T00:10:50.696 INFO:teuthology.orchestra.run.smithi103.stdout: "long_desc": "", 2024-05-23T00:10:50.696 INFO:teuthology.orchestra.run.smithi103.stdout: "tags": [], 2024-05-23T00:10:50.696 INFO:teuthology.orchestra.run.smithi103.stdout: "see_also": [] 2024-05-23T00:10:50.696 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.696 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.696 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.697 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.697 INFO:teuthology.orchestra.run.smithi103.stdout: "services": { 2024-05-23T00:10:50.697 INFO:teuthology.orchestra.run.smithi103.stdout: "dashboard": "https://172.21.15.103:8443/" 2024-05-23T00:10:50.697 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.697 INFO:teuthology.orchestra.run.smithi103.stdout: "always_on_modules": { 2024-05-23T00:10:50.697 INFO:teuthology.orchestra.run.smithi103.stdout: "octopus": [ 2024-05-23T00:10:50.697 INFO:teuthology.orchestra.run.smithi103.stdout: "balancer", 2024-05-23T00:10:50.697 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-05-23T00:10:50.698 INFO:teuthology.orchestra.run.smithi103.stdout: "devicehealth", 2024-05-23T00:10:50.698 INFO:teuthology.orchestra.run.smithi103.stdout: "orchestrator", 2024-05-23T00:10:50.698 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscaler", 2024-05-23T00:10:50.698 INFO:teuthology.orchestra.run.smithi103.stdout: "progress", 2024-05-23T00:10:50.698 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_support", 2024-05-23T00:10:50.698 INFO:teuthology.orchestra.run.smithi103.stdout: "status", 2024-05-23T00:10:50.698 INFO:teuthology.orchestra.run.smithi103.stdout: "telemetry", 2024-05-23T00:10:50.699 INFO:teuthology.orchestra.run.smithi103.stdout: "volumes" 2024-05-23T00:10:50.699 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.699 INFO:teuthology.orchestra.run.smithi103.stdout: "pacific": [ 2024-05-23T00:10:50.699 INFO:teuthology.orchestra.run.smithi103.stdout: "balancer", 2024-05-23T00:10:50.699 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-05-23T00:10:50.699 INFO:teuthology.orchestra.run.smithi103.stdout: "devicehealth", 2024-05-23T00:10:50.699 INFO:teuthology.orchestra.run.smithi103.stdout: "orchestrator", 2024-05-23T00:10:50.700 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscaler", 2024-05-23T00:10:50.700 INFO:teuthology.orchestra.run.smithi103.stdout: "progress", 2024-05-23T00:10:50.700 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_support", 2024-05-23T00:10:50.700 INFO:teuthology.orchestra.run.smithi103.stdout: "status", 2024-05-23T00:10:50.700 INFO:teuthology.orchestra.run.smithi103.stdout: "telemetry", 2024-05-23T00:10:50.700 INFO:teuthology.orchestra.run.smithi103.stdout: "volumes" 2024-05-23T00:10:50.700 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.700 INFO:teuthology.orchestra.run.smithi103.stdout: "quincy": [ 2024-05-23T00:10:50.701 INFO:teuthology.orchestra.run.smithi103.stdout: "balancer", 2024-05-23T00:10:50.701 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-05-23T00:10:50.701 INFO:teuthology.orchestra.run.smithi103.stdout: "devicehealth", 2024-05-23T00:10:50.701 INFO:teuthology.orchestra.run.smithi103.stdout: "orchestrator", 2024-05-23T00:10:50.701 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscaler", 2024-05-23T00:10:50.701 INFO:teuthology.orchestra.run.smithi103.stdout: "progress", 2024-05-23T00:10:50.701 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_support", 2024-05-23T00:10:50.701 INFO:teuthology.orchestra.run.smithi103.stdout: "status", 2024-05-23T00:10:50.702 INFO:teuthology.orchestra.run.smithi103.stdout: "telemetry", 2024-05-23T00:10:50.702 INFO:teuthology.orchestra.run.smithi103.stdout: "volumes" 2024-05-23T00:10:50.702 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.702 INFO:teuthology.orchestra.run.smithi103.stdout: "reef": [ 2024-05-23T00:10:50.702 INFO:teuthology.orchestra.run.smithi103.stdout: "balancer", 2024-05-23T00:10:50.702 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-05-23T00:10:50.702 INFO:teuthology.orchestra.run.smithi103.stdout: "devicehealth", 2024-05-23T00:10:50.703 INFO:teuthology.orchestra.run.smithi103.stdout: "orchestrator", 2024-05-23T00:10:50.703 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscaler", 2024-05-23T00:10:50.703 INFO:teuthology.orchestra.run.smithi103.stdout: "progress", 2024-05-23T00:10:50.703 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_support", 2024-05-23T00:10:50.703 INFO:teuthology.orchestra.run.smithi103.stdout: "status", 2024-05-23T00:10:50.703 INFO:teuthology.orchestra.run.smithi103.stdout: "telemetry", 2024-05-23T00:10:50.703 INFO:teuthology.orchestra.run.smithi103.stdout: "volumes" 2024-05-23T00:10:50.703 INFO:teuthology.orchestra.run.smithi103.stdout: ], 2024-05-23T00:10:50.704 INFO:teuthology.orchestra.run.smithi103.stdout: "squid": [ 2024-05-23T00:10:50.704 INFO:teuthology.orchestra.run.smithi103.stdout: "balancer", 2024-05-23T00:10:50.704 INFO:teuthology.orchestra.run.smithi103.stdout: "crash", 2024-05-23T00:10:50.704 INFO:teuthology.orchestra.run.smithi103.stdout: "devicehealth", 2024-05-23T00:10:50.704 INFO:teuthology.orchestra.run.smithi103.stdout: "orchestrator", 2024-05-23T00:10:50.704 INFO:teuthology.orchestra.run.smithi103.stdout: "pg_autoscaler", 2024-05-23T00:10:50.704 INFO:teuthology.orchestra.run.smithi103.stdout: "progress", 2024-05-23T00:10:50.705 INFO:teuthology.orchestra.run.smithi103.stdout: "rbd_support", 2024-05-23T00:10:50.705 INFO:teuthology.orchestra.run.smithi103.stdout: "status", 2024-05-23T00:10:50.705 INFO:teuthology.orchestra.run.smithi103.stdout: "telemetry", 2024-05-23T00:10:50.705 INFO:teuthology.orchestra.run.smithi103.stdout: "volumes" 2024-05-23T00:10:50.705 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:50.705 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.705 INFO:teuthology.orchestra.run.smithi103.stdout: "last_failure_osd_epoch": 94, 2024-05-23T00:10:50.705 INFO:teuthology.orchestra.run.smithi103.stdout: "active_clients": [ 2024-05-23T00:10:50.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.706 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "devicehealth", 2024-05-23T00:10:50.706 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-05-23T00:10:50.706 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.706 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v2", 2024-05-23T00:10:50.706 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:0", 2024-05-23T00:10:50.706 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1464341397 2024-05-23T00:10:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "libcephsqlite", 2024-05-23T00:10:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-05-23T00:10:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.707 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v2", 2024-05-23T00:10:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:0", 2024-05-23T00:10:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 2641485865 2024-05-23T00:10:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "rbd_support", 2024-05-23T00:10:50.708 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-05-23T00:10:50.709 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.709 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v2", 2024-05-23T00:10:50.709 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:0", 2024-05-23T00:10:50.709 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 2297897003 2024-05-23T00:10:50.709 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.709 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:50.709 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.710 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snap_schedule", 2024-05-23T00:10:50.710 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-05-23T00:10:50.710 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.710 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v2", 2024-05-23T00:10:50.710 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:0", 2024-05-23T00:10:50.710 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 3509173697 2024-05-23T00:10:50.711 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.711 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:50.711 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.711 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "snap_schedule", 2024-05-23T00:10:50.711 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-05-23T00:10:50.711 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.712 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v2", 2024-05-23T00:10:50.712 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:0", 2024-05-23T00:10:50.712 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 2789922122 2024-05-23T00:10:50.712 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.712 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:50.712 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:50.712 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.712 INFO:teuthology.orchestra.run.smithi103.stdout: "name": "volumes", 2024-05-23T00:10:50.713 INFO:teuthology.orchestra.run.smithi103.stdout: "addrvec": [ 2024-05-23T00:10:50.713 INFO:teuthology.orchestra.run.smithi103.stdout: { 2024-05-23T00:10:50.713 INFO:teuthology.orchestra.run.smithi103.stdout: "type": "v2", 2024-05-23T00:10:50.713 INFO:teuthology.orchestra.run.smithi103.stdout: "addr": "172.21.15.103:0", 2024-05-23T00:10:50.713 INFO:teuthology.orchestra.run.smithi103.stdout: "nonce": 1692528918 2024-05-23T00:10:50.713 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.713 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:50.713 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:50.714 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-23T00:10:50.714 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:10:50.720 DEBUG:tasks.check_counter:Getting stats from a 2024-05-23T00:10:50.720 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-23T00:10:50.721 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:48 smithi103 ceph-mon[31370]: pgmap v320: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:10:50.721 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:48 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1829128100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:10:50.721 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:48 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2391553977' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-23T00:10:50.799 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:50 smithi103 ceph-mon[31370]: pgmap v321: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:10:50.890 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:50 smithi174 ceph-mon[36179]: pgmap v321: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:10:50.891 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:50 smithi105 ceph-mon[36426]: pgmap v321: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:10:51.165 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:10:52.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:52 smithi105 ceph-mon[36426]: pgmap v322: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:52.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:52 smithi103 ceph-mon[31370]: pgmap v322: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:52.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:52 smithi174 ceph-mon[36179]: pgmap v322: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:53.537 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:10:53.538 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:10:53.538 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 2765, 2024-05-23T00:10:53.538 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 3948, 2024-05-23T00:10:53.538 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 4988427, 2024-05-23T00:10:53.538 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 4500247, 2024-05-23T00:10:53.538 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 9, 2024-05-23T00:10:53.539 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 3, 2024-05-23T00:10:53.539 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.550619013, 2024-05-23T00:10:53.539 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.247945807, 2024-05-23T00:10:53.539 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.360892946, 2024-05-23T00:10:53.539 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.015946641, 2024-05-23T00:10:53.540 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:10:53.540 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3949, 2024-05-23T00:10:53.540 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.384464675, 2024-05-23T00:10:53.540 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000097357 2024-05-23T00:10:53.540 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.541 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:10:53.541 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 6117, 2024-05-23T00:10:53.541 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.002764237, 2024-05-23T00:10:53.541 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000451 2024-05-23T00:10:53.541 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.542 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-23T00:10:53.542 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 448 2024-05-23T00:10:53.542 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.542 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:10:53.542 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 1086, 2024-05-23T00:10:53.542 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 3160, 2024-05-23T00:10:53.543 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 400200, 2024-05-23T00:10:53.543 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 1760324, 2024-05-23T00:10:53.543 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 23, 2024-05-23T00:10:53.543 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 7, 2024-05-23T00:10:53.543 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.423193938, 2024-05-23T00:10:53.543 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.267896571, 2024-05-23T00:10:53.543 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.180557254, 2024-05-23T00:10:53.543 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.004924905, 2024-05-23T00:10:53.544 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:10:53.544 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3157, 2024-05-23T00:10:53.544 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.680041110, 2024-05-23T00:10:53.544 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000215407 2024-05-23T00:10:53.544 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.544 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:10:53.545 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2313, 2024-05-23T00:10:53.545 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.002149257, 2024-05-23T00:10:53.545 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000929 2024-05-23T00:10:53.545 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.545 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-05-23T00:10:53.546 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 143168 2024-05-23T00:10:53.546 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.546 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:10:53.546 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 407, 2024-05-23T00:10:53.547 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 503, 2024-05-23T00:10:53.547 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1858487, 2024-05-23T00:10:53.547 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 532024, 2024-05-23T00:10:53.547 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 8, 2024-05-23T00:10:53.547 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 7, 2024-05-23T00:10:53.548 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.141575211, 2024-05-23T00:10:53.548 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.052969546, 2024-05-23T00:10:53.548 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.093317874, 2024-05-23T00:10:53.548 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.016079331, 2024-05-23T00:10:53.549 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:10:53.549 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 500, 2024-05-23T00:10:53.549 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.086906545, 2024-05-23T00:10:53.549 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000173813 2024-05-23T00:10:53.549 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.550 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:10:53.550 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.550 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.550 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.551 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 1819664, 2024-05-23T00:10:53.551 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 451296 2024-05-23T00:10:53.551 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.551 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-05-23T00:10:53.551 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-05-23T00:10:53.551 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-05-23T00:10:53.552 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.552 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-05-23T00:10:53.552 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-05-23T00:10:53.552 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-05-23T00:10:53.552 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 240, 2024-05-23T00:10:53.552 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.507679440, 2024-05-23T00:10:53.553 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002115331 2024-05-23T00:10:53.553 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.553 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.553 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-05-23T00:10:53.553 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-05-23T00:10:53.554 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-05-23T00:10:53.554 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 145, 2024-05-23T00:10:53.554 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.028126651, 2024-05-23T00:10:53.554 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000193976 2024-05-23T00:10:53.554 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.554 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.555 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-05-23T00:10:53.555 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 416, 2024-05-23T00:10:53.555 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 274, 2024-05-23T00:10:53.555 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-05-23T00:10:53.555 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 274, 2024-05-23T00:10:53.555 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.786680198, 2024-05-23T00:10:53.555 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002871095 2024-05-23T00:10:53.556 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.556 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-05-23T00:10:53.556 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 142, 2024-05-23T00:10:53.556 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-05-23T00:10:53.556 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-05-23T00:10:53.556 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 11, 2024-05-23T00:10:53.557 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-05-23T00:10:53.557 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 0, 2024-05-23T00:10:53.557 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 5398, 2024-05-23T00:10:53.557 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 1325, 2024-05-23T00:10:53.558 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 4073, 2024-05-23T00:10:53.558 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 0, 2024-05-23T00:10:53.558 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 3181, 2024-05-23T00:10:53.558 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 8, 2024-05-23T00:10:53.558 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 44, 2024-05-23T00:10:53.558 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 44, 2024-05-23T00:10:53.558 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 7, 2024-05-23T00:10:53.558 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 884, 2024-05-23T00:10:53.559 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 521, 2024-05-23T00:10:53.559 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 142, 2024-05-23T00:10:53.559 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 37, 2024-05-23T00:10:53.559 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-05-23T00:10:53.559 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-05-23T00:10:53.559 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 0, 2024-05-23T00:10:53.559 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 1000, 2024-05-23T00:10:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 1, 2024-05-23T00:10:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 0, 2024-05-23T00:10:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 0, 2024-05-23T00:10:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 80, 2024-05-23T00:10:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 5095, 2024-05-23T00:10:53.560 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-05-23T00:10:53.561 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 0, 2024-05-23T00:10:53.561 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-05-23T00:10:53.561 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 4, 2024-05-23T00:10:53.561 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 234, 2024-05-23T00:10:53.561 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 1, 2024-05-23T00:10:53.562 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 0, 2024-05-23T00:10:53.562 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-05-23T00:10:53.562 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-05-23T00:10:53.562 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-05-23T00:10:53.562 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-05-23T00:10:53.562 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-05-23T00:10:53.563 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-05-23T00:10:53.563 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 0, 2024-05-23T00:10:53.563 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 0, 2024-05-23T00:10:53.563 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 0, 2024-05-23T00:10:53.563 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 184, 2024-05-23T00:10:53.563 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-05-23T00:10:53.563 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-23T00:10:53.563 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-23T00:10:53.564 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 538, 2024-05-23T00:10:53.564 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 534, 2024-05-23T00:10:53.564 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 20, 2024-05-23T00:10:53.564 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 180 2024-05-23T00:10:53.564 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.564 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-05-23T00:10:53.565 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 20, 2024-05-23T00:10:53.565 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 342, 2024-05-23T00:10:53.565 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 185, 2024-05-23T00:10:53.565 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 234, 2024-05-23T00:10:53.565 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 20, 2024-05-23T00:10:53.565 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 0, 2024-05-23T00:10:53.565 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-05-23T00:10:53.565 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-05-23T00:10:53.566 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 90, 2024-05-23T00:10:53.566 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 90, 2024-05-23T00:10:53.566 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-05-23T00:10:53.566 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-05-23T00:10:53.566 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-05-23T00:10:53.566 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-05-23T00:10:53.566 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-05-23T00:10:53.567 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-05-23T00:10:53.567 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-05-23T00:10:53.567 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-05-23T00:10:53.567 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-05-23T00:10:53.567 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-05-23T00:10:53.567 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 0, 2024-05-23T00:10:53.567 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-05-23T00:10:53.567 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 0, 2024-05-23T00:10:53.568 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-05-23T00:10:53.568 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-05-23T00:10:53.568 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.568 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-05-23T00:10:53.568 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 1, 2024-05-23T00:10:53.568 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 1108, 2024-05-23T00:10:53.569 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-05-23T00:10:53.569 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-05-23T00:10:53.569 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 1108, 2024-05-23T00:10:53.569 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-05-23T00:10:53.569 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-05-23T00:10:53.570 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 3, 2024-05-23T00:10:53.570 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-05-23T00:10:53.570 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-05-23T00:10:53.570 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 3, 2024-05-23T00:10:53.570 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 2, 2024-05-23T00:10:53.570 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-05-23T00:10:53.571 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-05-23T00:10:53.571 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 4194304, 2024-05-23T00:10:53.571 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 8306893, 2024-05-23T00:10:53.571 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 0, 2024-05-23T00:10:53.571 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-05-23T00:10:53.571 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 97, 2024-05-23T00:10:53.571 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.261257063, 2024-05-23T00:10:53.572 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002693371 2024-05-23T00:10:53.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.572 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 0 2024-05-23T00:10:53.572 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.572 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-05-23T00:10:53.573 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 4390, 2024-05-23T00:10:53.573 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 4485, 2024-05-23T00:10:53.573 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 95, 2024-05-23T00:10:53.573 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 195, 2024-05-23T00:10:53.573 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 230, 2024-05-23T00:10:53.574 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 35, 2024-05-23T00:10:53.574 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 5398, 2024-05-23T00:10:53.574 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 5496, 2024-05-23T00:10:53.574 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 98, 2024-05-23T00:10:53.574 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 44, 2024-05-23T00:10:53.574 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 1758, 2024-05-23T00:10:53.575 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 1714, 2024-05-23T00:10:53.575 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 77836, 2024-05-23T00:10:53.575 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 223612 2024-05-23T00:10:53.575 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.575 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-05-23T00:10:53.575 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 600, 2024-05-23T00:10:53.575 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 58, 2024-05-23T00:10:53.575 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 416, 2024-05-23T00:10:53.576 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 17, 2024-05-23T00:10:53.576 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 296, 2024-05-23T00:10:53.576 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-05-23T00:10:53.576 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 90, 2024-05-23T00:10:53.576 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.277705803, 2024-05-23T00:10:53.576 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.003085620 2024-05-23T00:10:53.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.577 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-05-23T00:10:53.577 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1, 2024-05-23T00:10:53.577 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000380643, 2024-05-23T00:10:53.577 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000380643 2024-05-23T00:10:53.577 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.577 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-05-23T00:10:53.577 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.578 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.578 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.578 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-05-23T00:10:53.578 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.578 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.578 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.578 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.579 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-05-23T00:10:53.579 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 93, 2024-05-23T00:10:53.579 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.093535188, 2024-05-23T00:10:53.579 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001005754 2024-05-23T00:10:53.579 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.579 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-05-23T00:10:53.579 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.580 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.580 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.580 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.580 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-05-23T00:10:53.580 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.580 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.581 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.581 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-05-23T00:10:53.581 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.581 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.581 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.581 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.581 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-05-23T00:10:53.582 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.582 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.582 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.582 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.582 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-05-23T00:10:53.582 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.582 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.582 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.583 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.583 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-05-23T00:10:53.583 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.583 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.584 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.584 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.585 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-05-23T00:10:53.585 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.585 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.585 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.585 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.585 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-05-23T00:10:53.585 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.586 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.586 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.586 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-05-23T00:10:53.586 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.586 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.586 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.586 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.587 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-05-23T00:10:53.587 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.587 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.587 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.587 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.587 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-05-23T00:10:53.588 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.588 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.588 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.588 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.588 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-05-23T00:10:53.588 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.589 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.589 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.589 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.589 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-05-23T00:10:53.589 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.590 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.590 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.590 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.590 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-05-23T00:10:53.591 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.591 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.591 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.591 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.591 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-05-23T00:10:53.592 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.592 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.592 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.592 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.592 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-05-23T00:10:53.592 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.593 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.593 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.593 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.593 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-05-23T00:10:53.593 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.594 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.594 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.594 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.594 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-05-23T00:10:53.594 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.595 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.595 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.595 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.595 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-05-23T00:10:53.595 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.596 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.596 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.596 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.596 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-05-23T00:10:53.597 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.597 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.597 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.597 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.597 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-05-23T00:10:53.597 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.598 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.598 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.598 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.598 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-05-23T00:10:53.598 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.598 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.598 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.599 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-05-23T00:10:53.599 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.599 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.599 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.599 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.599 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-05-23T00:10:53.599 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 90, 2024-05-23T00:10:53.600 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.415058564, 2024-05-23T00:10:53.600 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.004611761 2024-05-23T00:10:53.600 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.600 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-05-23T00:10:53.600 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-05-23T00:10:53.600 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-05-23T00:10:53.600 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.600 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.601 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.601 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.601 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.601 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-05-23T00:10:53.601 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 1, 2024-05-23T00:10:53.602 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 3, 2024-05-23T00:10:53.602 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 2, 2024-05-23T00:10:53.602 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 1, 2024-05-23T00:10:53.602 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-05-23T00:10:53.602 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 266, 2024-05-23T00:10:53.602 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 266, 2024-05-23T00:10:53.602 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 619, 2024-05-23T00:10:53.603 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-05-23T00:10:53.603 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.603 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-05-23T00:10:53.603 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:10:53.603 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-05-23T00:10:53.603 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:10:53.603 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:10:53.603 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:10:53.604 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:10:53.604 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:10:53.604 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:10:53.604 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:10:53.604 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:10:53.604 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:10:53.604 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:10:53.605 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:10:53.605 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:10:53.605 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:10:53.605 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-05-23T00:10:53.605 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:10:53.605 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-05-23T00:10:53.606 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:10:53.606 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:10:53.606 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:10:53.606 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:10:53.606 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:10:53.606 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:10:53.606 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:10:53.607 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-05-23T00:10:53.607 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:10:53.607 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:10:53.607 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:10:53.607 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-05-23T00:10:53.607 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-05-23T00:10:53.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-05-23T00:10:53.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:10:53.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:10:53.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:10:53.609 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:10:53.610 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 453378, 2024-05-23T00:10:53.610 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 4250, 2024-05-23T00:10:53.610 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 880, 2024-05-23T00:10:53.610 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 10, 2024-05-23T00:10:53.610 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-05-23T00:10:53.610 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-05-23T00:10:53.611 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:10:53.611 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-05-23T00:10:53.611 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:10:53.611 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-05-23T00:10:53.611 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 9896, 2024-05-23T00:10:53.611 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 190, 2024-05-23T00:10:53.612 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:10:53.612 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:10:53.612 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-05-23T00:10:53.612 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-05-23T00:10:53.612 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 14258890, 2024-05-23T00:10:53.613 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 132288, 2024-05-23T00:10:53.613 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-05-23T00:10:53.613 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-05-23T00:10:53.613 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-05-23T00:10:53.613 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-05-23T00:10:53.613 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.613 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-05-23T00:10:53.613 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-05-23T00:10:53.614 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-05-23T00:10:53.614 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 362, 2024-05-23T00:10:53.614 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 4161248, 2024-05-23T00:10:53.614 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 33, 2024-05-23T00:10:53.614 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 362, 2024-05-23T00:10:53.614 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-05-23T00:10:53.614 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 362, 2024-05-23T00:10:53.615 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 1.789965473, 2024-05-23T00:10:53.615 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.004944656 2024-05-23T00:10:53.615 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.615 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-05-23T00:10:53.615 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-05-23T00:10:53.615 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 362, 2024-05-23T00:10:53.616 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 404 2024-05-23T00:10:53.616 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.616 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 362, 2024-05-23T00:10:53.616 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 7, 2024-05-23T00:10:53.616 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 355, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 0, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 10, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 7, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 104, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 16, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-05-23T00:10:53.617 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-05-23T00:10:53.618 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-05-23T00:10:53.618 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-05-23T00:10:53.618 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 192, 2024-05-23T00:10:53.618 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-05-23T00:10:53.618 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-05-23T00:10:53.618 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-05-23T00:10:53.619 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 0, 2024-05-23T00:10:53.619 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 20, 2024-05-23T00:10:53.619 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-05-23T00:10:53.619 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-05-23T00:10:53.619 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-05-23T00:10:53.620 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-05-23T00:10:53.620 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-05-23T00:10:53.620 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-05-23T00:10:53.620 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-05-23T00:10:53.620 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-05-23T00:10:53.620 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-05-23T00:10:53.620 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-05-23T00:10:53.621 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-05-23T00:10:53.621 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-05-23T00:10:53.621 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-05-23T00:10:53.621 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-05-23T00:10:53.621 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-05-23T00:10:53.621 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-05-23T00:10:53.621 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-05-23T00:10:53.621 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-05-23T00:10:53.622 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-05-23T00:10:53.622 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-05-23T00:10:53.622 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-05-23T00:10:53.622 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-05-23T00:10:53.622 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-05-23T00:10:53.622 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-05-23T00:10:53.623 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-05-23T00:10:53.623 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-05-23T00:10:53.623 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 104, 2024-05-23T00:10:53.623 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-05-23T00:10:53.623 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 13, 2024-05-23T00:10:53.623 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 12, 2024-05-23T00:10:53.624 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 12, 2024-05-23T00:10:53.624 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-05-23T00:10:53.624 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-05-23T00:10:53.624 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 47, 2024-05-23T00:10:53.624 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 0, 2024-05-23T00:10:53.624 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 8 2024-05-23T00:10:53.624 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.624 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-05-23T00:10:53.625 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 1, 2024-05-23T00:10:53.625 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-05-23T00:10:53.625 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-05-23T00:10:53.625 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-05-23T00:10:53.625 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.625 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-05-23T00:10:53.625 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-05-23T00:10:53.626 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.626 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-05-23T00:10:53.626 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-05-23T00:10:53.626 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 16, 2024-05-23T00:10:53.626 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-05-23T00:10:53.626 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 8, 2024-05-23T00:10:53.627 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 180, 2024-05-23T00:10:53.627 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 90, 2024-05-23T00:10:53.627 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-05-23T00:10:53.628 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.628 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:10:53.628 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-23T00:10:53.628 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-23T00:10:53.628 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-23T00:10:53.628 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 4254, 2024-05-23T00:10:53.629 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 6845864, 2024-05-23T00:10:53.629 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-23T00:10:53.629 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 4254, 2024-05-23T00:10:53.629 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-23T00:10:53.629 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-23T00:10:53.629 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 4254, 2024-05-23T00:10:53.629 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 6845864, 2024-05-23T00:10:53.630 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-23T00:10:53.630 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.630 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.630 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.630 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.630 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.631 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-05-23T00:10:53.631 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-23T00:10:53.631 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-23T00:10:53.631 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-23T00:10:53.631 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-23T00:10:53.631 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-23T00:10:53.632 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-23T00:10:53.632 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-23T00:10:53.632 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 362, 2024-05-23T00:10:53.632 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 4171058, 2024-05-23T00:10:53.632 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 362, 2024-05-23T00:10:53.633 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 4171058, 2024-05-23T00:10:53.633 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-23T00:10:53.633 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.633 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.634 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.634 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.634 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.634 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-05-23T00:10:53.634 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-23T00:10:53.635 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-05-23T00:10:53.635 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-23T00:10:53.635 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-23T00:10:53.635 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-23T00:10:53.636 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-23T00:10:53.636 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-23T00:10:53.636 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 362, 2024-05-23T00:10:53.636 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 362, 2024-05-23T00:10:53.636 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 362, 2024-05-23T00:10:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 362, 2024-05-23T00:10:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-23T00:10:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.637 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-05-23T00:10:53.638 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-23T00:10:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-05-23T00:10:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-23T00:10:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 90, 2024-05-23T00:10:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 9810, 2024-05-23T00:10:53.639 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-23T00:10:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 90, 2024-05-23T00:10:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-23T00:10:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-23T00:10:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 7, 2024-05-23T00:10:53.640 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 9810, 2024-05-23T00:10:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-23T00:10:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.641 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x5628a4c56500": { 2024-05-23T00:10:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-23T00:10:53.642 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-05-23T00:10:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-23T00:10:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 1108, 2024-05-23T00:10:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 4112589, 2024-05-23T00:10:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-23T00:10:53.643 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 1108, 2024-05-23T00:10:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-23T00:10:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-23T00:10:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 97, 2024-05-23T00:10:53.644 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 4112589, 2024-05-23T00:10:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-23T00:10:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:53.645 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.646 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:53.646 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:10:54.526 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-05-23T00:10:54.527 INFO:tasks.check_counter:Daemon mds.a mds.imported=80 2024-05-23T00:10:54.527 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=20 2024-05-23T00:10:54.527 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=342 2024-05-23T00:10:54.527 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=1 2024-05-23T00:10:54.527 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-05-23T00:10:54.527 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-05-23T00:10:54.527 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-05-23T00:10:54.528 DEBUG:tasks.check_counter:Getting stats from d 2024-05-23T00:10:54.528 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-23T00:10:54.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:54 smithi105 ceph-mon[36426]: pgmap v323: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:54.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:54 smithi103 ceph-mon[31370]: pgmap v323: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:54.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:54 smithi174 ceph-mon[36179]: pgmap v323: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:54.974 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:10:56.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:56 smithi103 ceph-mon[31370]: pgmap v324: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:10:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:56 smithi105 ceph-mon[36426]: pgmap v324: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:10:56.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:56 smithi174 ceph-mon[36179]: pgmap v324: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:10:57.212 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:10:57.212 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:10:57.212 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 235, 2024-05-23T00:10:57.212 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 223, 2024-05-23T00:10:57.212 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1722032, 2024-05-23T00:10:57.213 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 137504, 2024-05-23T00:10:57.213 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 4, 2024-05-23T00:10:57.213 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 0, 2024-05-23T00:10:57.213 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.104497202, 2024-05-23T00:10:57.213 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.035317948, 2024-05-23T00:10:57.213 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.077828121, 2024-05-23T00:10:57.213 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.010261910, 2024-05-23T00:10:57.213 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:10:57.214 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 222, 2024-05-23T00:10:57.214 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.021911009, 2024-05-23T00:10:57.214 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000098698 2024-05-23T00:10:57.214 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.214 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:10:57.214 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:57.214 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:57.214 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:57.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.215 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 1722032, 2024-05-23T00:10:57.215 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 137504 2024-05-23T00:10:57.215 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.215 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:10:57.215 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 2, 2024-05-23T00:10:57.215 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 171, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 320, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 185664, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 5, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 3, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.037090414, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.027061299, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.003741113, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:10:57.216 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:10:57.217 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 168, 2024-05-23T00:10:57.217 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.027613237, 2024-05-23T00:10:57.217 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000164364 2024-05-23T00:10:57.217 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.217 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:10:57.217 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:57.217 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:57.217 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:57.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.218 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-23T00:10:57.218 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 185664 2024-05-23T00:10:57.218 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.218 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:10:57.219 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 4, 2024-05-23T00:10:57.219 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 4, 2024-05-23T00:10:57.219 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 3024, 2024-05-23T00:10:57.219 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 448, 2024-05-23T00:10:57.219 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 2, 2024-05-23T00:10:57.219 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 1, 2024-05-23T00:10:57.219 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.001123875, 2024-05-23T00:10:57.219 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.000060949, 2024-05-23T00:10:57.220 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.000986878, 2024-05-23T00:10:57.220 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:10:57.220 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:10:57.220 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3, 2024-05-23T00:10:57.220 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.001308893, 2024-05-23T00:10:57.220 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000436297 2024-05-23T00:10:57.220 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.220 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 448 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-05-23T00:10:57.221 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 0, 2024-05-23T00:10:57.222 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-05-23T00:10:57.222 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.222 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-05-23T00:10:57.222 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:10:57.222 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-05-23T00:10:57.222 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:10:57.222 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:10:57.222 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:10:57.223 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:10:57.223 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:10:57.223 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:10:57.223 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:10:57.223 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:10:57.223 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:10:57.223 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:10:57.224 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:10:57.225 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:10:57.225 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:10:57.225 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:10:57.225 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:10:57.225 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-05-23T00:10:57.225 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:10:57.225 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:10:57.225 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:10:57.226 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-05-23T00:10:57.226 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-05-23T00:10:57.226 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-05-23T00:10:57.226 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:10:57.226 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:10:57.226 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:10:57.226 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:10:57.226 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 148714, 2024-05-23T00:10:57.227 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 30, 2024-05-23T00:10:57.227 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:10:57.227 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-05-23T00:10:57.227 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-05-23T00:10:57.227 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-05-23T00:10:57.227 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:10:57.227 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-05-23T00:10:57.227 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:10:57.228 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-05-23T00:10:57.228 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 0, 2024-05-23T00:10:57.228 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 0, 2024-05-23T00:10:57.228 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:10:57.228 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:10:57.228 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 928, 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 2, 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:10:57.229 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:10:57.230 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-23T00:10:57.230 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-23T00:10:57.230 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-23T00:10:57.230 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 237, 2024-05-23T00:10:57.230 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 1688919, 2024-05-23T00:10:57.230 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-23T00:10:57.230 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 237, 2024-05-23T00:10:57.230 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-23T00:10:57.231 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-23T00:10:57.231 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 237, 2024-05-23T00:10:57.231 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 1688919, 2024-05-23T00:10:57.231 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-23T00:10:57.231 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:10:57.231 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:10:57.231 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:10:57.231 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:57.232 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:10:57.232 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:10:57.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:57 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:10:57.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:57 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:57 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:57 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:57 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:57 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:10:57.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:57 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:57 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:57 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:57 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:57 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-23T00:10:57.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:57 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:57 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:57 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:57.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:57 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:58.114 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-05-23T00:10:58.114 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-05-23T00:10:58.114 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-05-23T00:10:58.114 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-05-23T00:10:58.114 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-05-23T00:10:58.115 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-05-23T00:10:58.115 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-05-23T00:10:58.115 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-05-23T00:10:58.115 DEBUG:tasks.check_counter:Getting stats from g 2024-05-23T00:10:58.115 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-23T00:10:58.574 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:10:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:58 smithi105 ceph-mon[36426]: pgmap v325: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:58 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:58 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:58.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:58 smithi103 ceph-mon[31370]: pgmap v325: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:58.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:58 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:58.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:58 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:58.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:58 smithi174 ceph-mon[36179]: pgmap v325: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-05-23T00:10:58.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:58 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:10:58.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:58 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: pgmap v326: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:11:00.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.080 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-23T00:11:00.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: Unable to set osd_memory_target on smithi105 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-05-23T00:11:00.081 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.082 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:10:59 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-05-23T00:11:00.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: pgmap v326: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:11:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-23T00:11:00.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: Unable to set osd_memory_target on smithi105 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-05-23T00:11:00.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:10:59 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-05-23T00:11:00.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:00.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: pgmap v326: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-05-23T00:11:00.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.368 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi105 to 176.2M 2024-05-23T00:11:00.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: Unable to set osd_memory_target on smithi105 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-05-23T00:11:00.369 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:00.370 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:10:59 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi174 to 176.2M 2024-05-23T00:11:01.079 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:00 smithi103 ceph-mon[31370]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-23T00:11:01.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:00 smithi105 ceph-mon[36426]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-23T00:11:01.337 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:11:01.337 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:01.337 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 232, 2024-05-23T00:11:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 220, 2024-05-23T00:11:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1721216, 2024-05-23T00:11:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 135584, 2024-05-23T00:11:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 4, 2024-05-23T00:11:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 0, 2024-05-23T00:11:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.099622221, 2024-05-23T00:11:01.338 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.034682629, 2024-05-23T00:11:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.069503267, 2024-05-23T00:11:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.010036944, 2024-05-23T00:11:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 219, 2024-05-23T00:11:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.021020636, 2024-05-23T00:11:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000095984 2024-05-23T00:11:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.339 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 1721216, 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 135584 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:01.340 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 2, 2024-05-23T00:11:01.341 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 168, 2024-05-23T00:11:01.341 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 320, 2024-05-23T00:11:01.341 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 185136, 2024-05-23T00:11:01.341 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 5, 2024-05-23T00:11:01.341 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 3, 2024-05-23T00:11:01.342 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.036517230, 2024-05-23T00:11:01.342 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.026684489, 2024-05-23T00:11:01.342 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.003383810, 2024-05-23T00:11:01.342 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:01.342 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:01.342 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 165, 2024-05-23T00:11:01.342 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.031484329, 2024-05-23T00:11:01.343 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000190814 2024-05-23T00:11:01.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.343 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:01.343 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:01.343 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:01.343 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:01.343 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.343 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 185136 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 4, 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 4, 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 3024, 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 448, 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 2, 2024-05-23T00:11:01.344 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 1, 2024-05-23T00:11:01.345 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.001613094, 2024-05-23T00:11:01.345 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.000081874, 2024-05-23T00:11:01.345 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.001496220, 2024-05-23T00:11:01.345 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:01.345 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:01.345 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3, 2024-05-23T00:11:01.345 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.001649178, 2024-05-23T00:11:01.345 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000549726 2024-05-23T00:11:01.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.346 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:01.346 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:01.346 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:01.346 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:01.346 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.347 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-23T00:11:01.347 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 448 2024-05-23T00:11:01.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.347 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-05-23T00:11:01.347 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 0, 2024-05-23T00:11:01.347 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-05-23T00:11:01.347 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.347 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-05-23T00:11:01.348 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:11:01.348 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-05-23T00:11:01.348 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:01.348 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:01.348 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:01.348 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:01.348 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:01.348 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:01.349 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:01.350 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:01.351 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:01.351 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:01.351 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:01.351 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:01.351 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:01.351 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:01.351 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:01.351 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-05-23T00:11:01.352 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:01.352 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:01.352 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:01.352 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:01.352 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 148714, 2024-05-23T00:11:01.352 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 30, 2024-05-23T00:11:01.352 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:11:01.352 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-05-23T00:11:01.353 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-05-23T00:11:01.353 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-05-23T00:11:01.353 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:01.353 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:01.353 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:01.353 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:01.353 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 0, 2024-05-23T00:11:01.353 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 0, 2024-05-23T00:11:01.354 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:01.354 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:01.354 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:01.354 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-05-23T00:11:01.354 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 928, 2024-05-23T00:11:01.354 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 2, 2024-05-23T00:11:01.354 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:01.354 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-05-23T00:11:01.355 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:01.355 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-05-23T00:11:01.355 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:01.355 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:01.355 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-23T00:11:01.355 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-23T00:11:01.355 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-23T00:11:01.355 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 234, 2024-05-23T00:11:01.356 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 1688529, 2024-05-23T00:11:01.356 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:01.356 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 234, 2024-05-23T00:11:01.356 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-23T00:11:01.356 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-23T00:11:01.356 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 234, 2024-05-23T00:11:01.356 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 1688529, 2024-05-23T00:11:01.356 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-23T00:11:01.357 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:01.357 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:01.357 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:01.357 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:11:01.357 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:11:01.357 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:11:01.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:00 smithi174 ceph-mon[36179]: Unable to set osd_memory_target on smithi174 to 184770252: error parsing value: Value '184770252' is below minimum 939524096 2024-05-23T00:11:02.078 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:01 smithi103 ceph-mon[31370]: pgmap v327: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:11:02.166 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-05-23T00:11:02.167 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-05-23T00:11:02.167 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-05-23T00:11:02.167 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-05-23T00:11:02.167 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-05-23T00:11:02.167 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-05-23T00:11:02.167 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-05-23T00:11:02.167 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-05-23T00:11:02.167 DEBUG:tasks.check_counter:Getting stats from j 2024-05-23T00:11:02.168 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-23T00:11:02.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:01 smithi105 ceph-mon[36426]: pgmap v327: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:11:02.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:01 smithi174 ceph-mon[36179]: pgmap v327: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-23T00:11:02.607 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.a/config 2024-05-23T00:11:03.670 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:03 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:03.670 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:03 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:03 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:03.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:03 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:03.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:03 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:03.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:03 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:04 smithi103 ceph-mon[31370]: pgmap v328: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:04 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:04 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:04 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:04 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.606 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:04 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:11:04.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:04 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:11:04.607 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:04 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:04.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:04 smithi105 ceph-mon[36426]: pgmap v328: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:04.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:04 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:04 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:04 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:04 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:04 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:11:04.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:04 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:11:04.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:04 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:04.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:04 smithi174 ceph-mon[36179]: pgmap v328: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:04.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:04 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:04 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:04 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:04 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-23T00:11:04.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:04 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-23T00:11:04.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:04 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-23T00:11:04.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:04 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' 2024-05-23T00:11:04.973 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-23T00:11:04.973 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:04.973 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 242, 2024-05-23T00:11:04.973 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 228, 2024-05-23T00:11:04.974 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 1725872, 2024-05-23T00:11:04.974 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 139872, 2024-05-23T00:11:04.974 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 4, 2024-05-23T00:11:04.974 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 0, 2024-05-23T00:11:04.974 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.107994783, 2024-05-23T00:11:04.974 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.024865204, 2024-05-23T00:11:04.974 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.101450122, 2024-05-23T00:11:04.974 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.011334117, 2024-05-23T00:11:04.975 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:04.975 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 228, 2024-05-23T00:11:04.975 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.024002014, 2024-05-23T00:11:04.975 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000105271 2024-05-23T00:11:04.975 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.975 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:04.975 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:04.975 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:04.976 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:04.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.976 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 1725872, 2024-05-23T00:11:04.976 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 139872 2024-05-23T00:11:04.976 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.976 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:04.976 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 2, 2024-05-23T00:11:04.977 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 174, 2024-05-23T00:11:04.977 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 320, 2024-05-23T00:11:04.977 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 186192, 2024-05-23T00:11:04.977 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 5, 2024-05-23T00:11:04.977 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 3, 2024-05-23T00:11:04.977 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.037470361, 2024-05-23T00:11:04.977 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.026758007, 2024-05-23T00:11:04.977 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.004390745, 2024-05-23T00:11:04.978 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:04.978 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:04.978 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 171, 2024-05-23T00:11:04.978 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.029836959, 2024-05-23T00:11:04.978 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000174485 2024-05-23T00:11:04.978 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.978 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:04.978 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 186192 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 0, 2024-05-23T00:11:04.979 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 2, 2024-05-23T00:11:04.980 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 0, 2024-05-23T00:11:04.980 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 0, 2024-05-23T00:11:04.980 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 2, 2024-05-23T00:11:04.980 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 1, 2024-05-23T00:11:04.980 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.001277795, 2024-05-23T00:11:04.980 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.000000000, 2024-05-23T00:11:04.980 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.001104610, 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:04.981 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-23T00:11:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-05-23T00:11:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-05-23T00:11:04.982 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 0, 2024-05-23T00:11:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-05-23T00:11:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-05-23T00:11:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:11:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-05-23T00:11:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:04.983 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:04.984 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:04.984 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:04.984 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:04.984 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:04.984 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:04.984 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:04.984 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:04.985 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:04.985 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:04.985 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:04.985 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:04.985 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:04.985 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:04.985 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:04.986 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:04.986 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:04.986 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:04.986 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:04.986 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:04.986 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:04.987 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:04.987 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:04.987 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:04.987 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:04.987 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:04.987 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:04.987 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-05-23T00:11:04.987 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:04.988 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:04.988 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:04.988 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:04.988 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 148731, 2024-05-23T00:11:04.988 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 31, 2024-05-23T00:11:04.988 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:11:04.988 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-05-23T00:11:04.988 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:04.989 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:04.990 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:04.990 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-05-23T00:11:04.990 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 928, 2024-05-23T00:11:04.990 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 2, 2024-05-23T00:11:04.990 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:04.990 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-05-23T00:11:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-05-23T00:11:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-23T00:11:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-23T00:11:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-23T00:11:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-23T00:11:04.991 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 240, 2024-05-23T00:11:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 1689309, 2024-05-23T00:11:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 240, 2024-05-23T00:11:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-23T00:11:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-23T00:11:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 240, 2024-05-23T00:11:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 1689309, 2024-05-23T00:11:04.992 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-23T00:11:04.993 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-23T00:11:04.993 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-23T00:11:04.993 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-23T00:11:04.993 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:11:04.993 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-23T00:11:04.993 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-23T00:11:05.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:05 smithi105 ceph-mon[36426]: Adjusting osd_memory_target on smithi103 to 176.4M 2024-05-23T00:11:05.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:05 smithi105 ceph-mon[36426]: Unable to set osd_memory_target on smithi103 to 185066291: error parsing value: Value '185066291' is below minimum 939524096 2024-05-23T00:11:05.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:05 smithi103 ceph-mon[31370]: Adjusting osd_memory_target on smithi103 to 176.4M 2024-05-23T00:11:05.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:05 smithi103 ceph-mon[31370]: Unable to set osd_memory_target on smithi103 to 185066291: error parsing value: Value '185066291' is below minimum 939524096 2024-05-23T00:11:05.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:05 smithi174 ceph-mon[36179]: Adjusting osd_memory_target on smithi103 to 176.4M 2024-05-23T00:11:05.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:05 smithi174 ceph-mon[36179]: Unable to set osd_memory_target on smithi103 to 185066291: error parsing value: Value '185066291' is below minimum 939524096 2024-05-23T00:11:05.923 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-05-23T00:11:05.923 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-05-23T00:11:05.923 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-05-23T00:11:05.923 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-05-23T00:11:05.923 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-05-23T00:11:05.924 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-05-23T00:11:05.924 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-05-23T00:11:05.924 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-05-23T00:11:05.924 DEBUG:tasks.check_counter:Getting stats from b 2024-05-23T00:11:05.925 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-23T00:11:06.174 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-23T00:11:06.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:06 smithi105 ceph-mon[36426]: pgmap v329: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:11:06.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:06 smithi103 ceph-mon[31370]: pgmap v329: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:11:06.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:06 smithi174 ceph-mon[36179]: pgmap v329: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-05-23T00:11:08.022 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-23T00:11:08.022 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:08.022 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 2951, 2024-05-23T00:11:08.022 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2885, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 950712, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 11664362, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 52, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 5, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.253065998, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.087441064, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.083378793, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.069974954, 2024-05-23T00:11:08.023 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:08.024 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2886, 2024-05-23T00:11:08.024 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.903661166, 2024-05-23T00:11:08.024 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000313118 2024-05-23T00:11:08.024 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.024 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:08.024 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1921, 2024-05-23T00:11:08.024 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000368625, 2024-05-23T00:11:08.024 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000191 2024-05-23T00:11:08.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.025 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 42544, 2024-05-23T00:11:08.025 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 1376 2024-05-23T00:11:08.025 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.025 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:08.025 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 2235, 2024-05-23T00:11:08.025 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2184, 2024-05-23T00:11:08.025 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 2461201, 2024-05-23T00:11:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 2549503, 2024-05-23T00:11:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 14, 2024-05-23T00:11:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 6, 2024-05-23T00:11:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.162900691, 2024-05-23T00:11:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.053361337, 2024-05-23T00:11:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.068293516, 2024-05-23T00:11:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.047732031, 2024-05-23T00:11:08.026 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2182, 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 1.877365214, 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000860387 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 294, 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000055488, 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000188 2024-05-23T00:11:08.027 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.028 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1933392, 2024-05-23T00:11:08.028 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 501696 2024-05-23T00:11:08.028 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.028 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:08.028 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 8792, 2024-05-23T00:11:08.028 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 12113, 2024-05-23T00:11:08.028 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 5521242, 2024-05-23T00:11:08.028 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 6673602, 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 21, 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 8, 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.667154705, 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.232230465, 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.360936812, 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.133437137, 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 12108, 2024-05-23T00:11:08.029 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 1.400812519, 2024-05-23T00:11:08.030 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000115693 2024-05-23T00:11:08.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.030 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:08.030 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 14312, 2024-05-23T00:11:08.030 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.002485946, 2024-05-23T00:11:08.030 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000173 2024-05-23T00:11:08.030 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.030 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 325024, 2024-05-23T00:11:08.031 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 325312 2024-05-23T00:11:08.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.031 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-05-23T00:11:08.031 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 1, 2024-05-23T00:11:08.031 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-05-23T00:11:08.031 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.031 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-MDSRank": { 2024-05-23T00:11:08.031 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-23T00:11:08.032 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-23T00:11:08.032 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1195, 2024-05-23T00:11:08.032 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 1.091327405, 2024-05-23T00:11:08.032 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000913244 2024-05-23T00:11:08.032 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.032 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.032 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-PurgeQueue": { 2024-05-23T00:11:08.032 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-23T00:11:08.033 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-23T00:11:08.033 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 288, 2024-05-23T00:11:08.033 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.014452346, 2024-05-23T00:11:08.033 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000050181 2024-05-23T00:11:08.033 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.033 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.033 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-05-23T00:11:08.033 INFO:teuthology.orchestra.run.smithi105.stdout: "request": 1482, 2024-05-23T00:11:08.034 INFO:teuthology.orchestra.run.smithi105.stdout: "reply": 1417, 2024-05-23T00:11:08.034 INFO:teuthology.orchestra.run.smithi105.stdout: "reply_latency": { 2024-05-23T00:11:08.034 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1417, 2024-05-23T00:11:08.034 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 10.586294100, 2024-05-23T00:11:08.034 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.007470920 2024-05-23T00:11:08.034 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.034 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_reply": 0, 2024-05-23T00:11:08.034 INFO:teuthology.orchestra.run.smithi105.stdout: "forward": 65, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_complete": 2, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_keys": 1, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_commit": 255, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_split": 0, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_merge": 0, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes": 525, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_top": 124, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_bottom": 241, 2024-05-23T00:11:08.035 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pin_tail": 160, 2024-05-23T00:11:08.036 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pinned": 302, 2024-05-23T00:11:08.036 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_expired": 3213, 2024-05-23T00:11:08.036 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_with_caps": 256, 2024-05-23T00:11:08.036 INFO:teuthology.orchestra.run.smithi105.stdout: "caps": 256, 2024-05-23T00:11:08.036 INFO:teuthology.orchestra.run.smithi105.stdout: "subtrees": 15, 2024-05-23T00:11:08.036 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse": 2996, 2024-05-23T00:11:08.036 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_hit": 1691, 2024-05-23T00:11:08.036 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_forward": 65, 2024-05-23T00:11:08.037 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_discover": 26, 2024-05-23T00:11:08.037 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_dir_fetch": 0, 2024-05-23T00:11:08.037 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_remote_ino": 0, 2024-05-23T00:11:08.037 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_lock": 4, 2024-05-23T00:11:08.037 INFO:teuthology.orchestra.run.smithi105.stdout: "load_cent": 93, 2024-05-23T00:11:08.037 INFO:teuthology.orchestra.run.smithi105.stdout: "q": 0, 2024-05-23T00:11:08.037 INFO:teuthology.orchestra.run.smithi105.stdout: "exported": 70, 2024-05-23T00:11:08.037 INFO:teuthology.orchestra.run.smithi105.stdout: "exported_inodes": 2653, 2024-05-23T00:11:08.038 INFO:teuthology.orchestra.run.smithi105.stdout: "imported": 79, 2024-05-23T00:11:08.038 INFO:teuthology.orchestra.run.smithi105.stdout: "imported_inodes": 2852, 2024-05-23T00:11:08.038 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_dir_fetch": 0, 2024-05-23T00:11:08.038 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_backtrace_fetch": 32, 2024-05-23T00:11:08.038 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_peer_discover": 0, 2024-05-23T00:11:08.038 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rfiles": 3454, 2024-05-23T00:11:08.038 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rbytes": 26744693, 2024-05-23T00:11:08.038 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rsnaps": 6, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_fetch": 32, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_set_tag": 0, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_repaired": 0, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_inotable_repaired": 0, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_inodes": 21, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_base_inodes": 3, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dirfrag_rstats": 18, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_file_inodes": 11, 2024-05-23T00:11:08.039 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_inode_file_caps": 0, 2024-05-23T00:11:08.040 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_revoke": 26, 2024-05-23T00:11:08.040 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_grant": 375, 2024-05-23T00:11:08.040 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_trunc": 2, 2024-05-23T00:11:08.040 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-23T00:11:08.040 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-23T00:11:08.040 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps": 1542, 2024-05-23T00:11:08.040 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps_dirty": 1437, 2024-05-23T00:11:08.040 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_cap_release": 29, 2024-05-23T00:11:08.041 INFO:teuthology.orchestra.run.smithi105.stdout: "process_request_cap_release": 435 2024-05-23T00:11:08.041 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.041 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache": { 2024-05-23T00:11:08.041 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update": 40, 2024-05-23T00:11:08.041 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update_receipt": 293, 2024-05-23T00:11:08.041 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_try_discover": 150, 2024-05-23T00:11:08.041 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_send_discover": 188, 2024-05-23T00:11:08.041 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_handle_discover": 67, 2024-05-23T00:11:08.042 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays": 0, 2024-05-23T00:11:08.042 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_delayed": 0, 2024-05-23T00:11:08.042 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_enqueuing": 0, 2024-05-23T00:11:08.042 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_created": 139, 2024-05-23T00:11:08.042 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_enqueued": 139, 2024-05-23T00:11:08.042 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_reintegrated": 0, 2024-05-23T00:11:08.042 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_migrated": 0, 2024-05-23T00:11:08.042 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_processing": 0, 2024-05-23T00:11:08.043 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_enqueued": 0, 2024-05-23T00:11:08.043 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_prioritized": 0, 2024-05-23T00:11:08.043 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_started": 0, 2024-05-23T00:11:08.043 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_completed": 0, 2024-05-23T00:11:08.043 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_path": 0, 2024-05-23T00:11:08.043 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_inode": 0, 2024-05-23T00:11:08.043 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_enqueue_scrub": 3, 2024-05-23T00:11:08.043 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_exportdir": 70, 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_flush": 0, 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragmentdir": 0, 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragstats": 0, 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_inodestats": 0 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log": { 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: "evlrg": 0, 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: "evadd": 2779, 2024-05-23T00:11:08.044 INFO:teuthology.orchestra.run.smithi105.stdout: "evex": 2372, 2024-05-23T00:11:08.045 INFO:teuthology.orchestra.run.smithi105.stdout: "evtrm": 2372, 2024-05-23T00:11:08.045 INFO:teuthology.orchestra.run.smithi105.stdout: "ev": 407, 2024-05-23T00:11:08.045 INFO:teuthology.orchestra.run.smithi105.stdout: "evexg": 0, 2024-05-23T00:11:08.045 INFO:teuthology.orchestra.run.smithi105.stdout: "evexd": 153, 2024-05-23T00:11:08.045 INFO:teuthology.orchestra.run.smithi105.stdout: "segadd": 15, 2024-05-23T00:11:08.045 INFO:teuthology.orchestra.run.smithi105.stdout: "segex": 14, 2024-05-23T00:11:08.045 INFO:teuthology.orchestra.run.smithi105.stdout: "segtrm": 14, 2024-05-23T00:11:08.045 INFO:teuthology.orchestra.run.smithi105.stdout: "seg": 1, 2024-05-23T00:11:08.046 INFO:teuthology.orchestra.run.smithi105.stdout: "segmjr": 1, 2024-05-23T00:11:08.046 INFO:teuthology.orchestra.run.smithi105.stdout: "segexg": 0, 2024-05-23T00:11:08.046 INFO:teuthology.orchestra.run.smithi105.stdout: "segexd": 1, 2024-05-23T00:11:08.046 INFO:teuthology.orchestra.run.smithi105.stdout: "expos": 9956587, 2024-05-23T00:11:08.046 INFO:teuthology.orchestra.run.smithi105.stdout: "wrpos": 11471904, 2024-05-23T00:11:08.046 INFO:teuthology.orchestra.run.smithi105.stdout: "rdpos": 0, 2024-05-23T00:11:08.046 INFO:teuthology.orchestra.run.smithi105.stdout: "jlat": { 2024-05-23T00:11:08.046 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 380, 2024-05-23T00:11:08.047 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 1.459061265, 2024-05-23T00:11:08.047 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.003839634 2024-05-23T00:11:08.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.047 INFO:teuthology.orchestra.run.smithi105.stdout: "replayed": 0 2024-05-23T00:11:08.047 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.047 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_mem": { 2024-05-23T00:11:08.047 INFO:teuthology.orchestra.run.smithi105.stdout: "ino": 426, 2024-05-23T00:11:08.047 INFO:teuthology.orchestra.run.smithi105.stdout: "ino+": 3029, 2024-05-23T00:11:08.048 INFO:teuthology.orchestra.run.smithi105.stdout: "ino-": 2603, 2024-05-23T00:11:08.048 INFO:teuthology.orchestra.run.smithi105.stdout: "dir": 68, 2024-05-23T00:11:08.048 INFO:teuthology.orchestra.run.smithi105.stdout: "dir+": 233, 2024-05-23T00:11:08.048 INFO:teuthology.orchestra.run.smithi105.stdout: "dir-": 165, 2024-05-23T00:11:08.048 INFO:teuthology.orchestra.run.smithi105.stdout: "dn": 525, 2024-05-23T00:11:08.048 INFO:teuthology.orchestra.run.smithi105.stdout: "dn+": 3949, 2024-05-23T00:11:08.048 INFO:teuthology.orchestra.run.smithi105.stdout: "dn-": 3424, 2024-05-23T00:11:08.048 INFO:teuthology.orchestra.run.smithi105.stdout: "cap": 256, 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: "cap+": 2133, 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: "cap-": 1877, 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: "rss": 79956, 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: "heap": 231804 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_server": { 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_client_request": 1962, 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_server_request": 0, 2024-05-23T00:11:08.049 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_request": 1482, 2024-05-23T00:11:08.050 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_session": 77, 2024-05-23T00:11:08.050 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_peer_request": 369, 2024-05-23T00:11:08.050 INFO:teuthology.orchestra.run.smithi105.stdout: "req_create_latency": { 2024-05-23T00:11:08.050 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 153, 2024-05-23T00:11:08.050 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.258413309, 2024-05-23T00:11:08.050 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.001688975 2024-05-23T00:11:08.050 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.050 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getattr_latency": { 2024-05-23T00:11:08.051 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 121, 2024-05-23T00:11:08.051 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.029375266, 2024-05-23T00:11:08.051 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000242770 2024-05-23T00:11:08.051 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.051 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getfilelock_latency": { 2024-05-23T00:11:08.051 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.051 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.051 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.052 INFO:teuthology.orchestra.run.smithi105.stdout: "req_link_latency": { 2024-05-23T00:11:08.052 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.052 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.052 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.052 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.052 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookup_latency": { 2024-05-23T00:11:08.052 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 797, 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.269744612, 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000338449 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookuphash_latency": { 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.053 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupino_latency": { 2024-05-23T00:11:08.054 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.054 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.054 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.054 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.054 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupname_latency": { 2024-05-23T00:11:08.054 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.054 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.054 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupparent_latency": { 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupsnap_latency": { 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.055 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.056 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.056 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.056 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lssnap_latency": { 2024-05-23T00:11:08.057 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 11, 2024-05-23T00:11:08.057 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.002600705, 2024-05-23T00:11:08.057 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000236427 2024-05-23T00:11:08.057 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.057 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mkdir_latency": { 2024-05-23T00:11:08.057 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 10, 2024-05-23T00:11:08.057 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.018418043, 2024-05-23T00:11:08.058 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.001841804 2024-05-23T00:11:08.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.058 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mknod_latency": { 2024-05-23T00:11:08.058 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.058 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.058 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.058 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.058 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mksnap_latency": { 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 11, 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.071303872, 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.006482170 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: "req_open_latency": { 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 107, 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.027495313, 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000256965 2024-05-23T00:11:08.059 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.060 INFO:teuthology.orchestra.run.smithi105.stdout: "req_readdir_latency": { 2024-05-23T00:11:08.060 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 32, 2024-05-23T00:11:08.060 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.024935559, 2024-05-23T00:11:08.060 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000779236 2024-05-23T00:11:08.060 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.060 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rename_latency": { 2024-05-23T00:11:08.060 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 19, 2024-05-23T00:11:08.060 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 9.568095623, 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.503583980 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: "req_renamesnap_latency": { 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: "req_snapdiff_latency": { 2024-05-23T00:11:08.061 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.062 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.062 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.062 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.062 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmdir_latency": { 2024-05-23T00:11:08.062 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.062 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.062 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.063 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmsnap_latency": { 2024-05-23T00:11:08.063 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 5, 2024-05-23T00:11:08.063 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.026536205, 2024-05-23T00:11:08.063 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.005307241 2024-05-23T00:11:08.063 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.063 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmxattr_latency": { 2024-05-23T00:11:08.063 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.064 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.064 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.064 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setattr_latency": { 2024-05-23T00:11:08.064 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 4, 2024-05-23T00:11:08.064 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.008782659, 2024-05-23T00:11:08.064 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.002195664 2024-05-23T00:11:08.064 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setdirlayout_latency": { 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setfilelock_latency": { 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.065 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.066 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setlayout_latency": { 2024-05-23T00:11:08.066 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.066 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.066 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.066 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.066 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setxattr_latency": { 2024-05-23T00:11:08.066 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 8, 2024-05-23T00:11:08.067 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.022595280, 2024-05-23T00:11:08.067 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.002824410 2024-05-23T00:11:08.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.067 INFO:teuthology.orchestra.run.smithi105.stdout: "req_symlink_latency": { 2024-05-23T00:11:08.067 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2, 2024-05-23T00:11:08.067 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.001652734, 2024-05-23T00:11:08.067 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000826367 2024-05-23T00:11:08.067 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.068 INFO:teuthology.orchestra.run.smithi105.stdout: "req_unlink_latency": { 2024-05-23T00:11:08.068 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 137, 2024-05-23T00:11:08.068 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.256344920, 2024-05-23T00:11:08.068 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.001871130 2024-05-23T00:11:08.068 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.068 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_revoke_eviction": 0, 2024-05-23T00:11:08.068 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_acquisition_throttle": 0, 2024-05-23T00:11:08.068 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getvxattr_latency": { 2024-05-23T00:11:08.069 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.069 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.069 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.069 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.069 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.069 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sessions": { 2024-05-23T00:11:08.069 INFO:teuthology.orchestra.run.smithi105.stdout: "session_count": 2, 2024-05-23T00:11:08.069 INFO:teuthology.orchestra.run.smithi105.stdout: "session_add": 6, 2024-05-23T00:11:08.070 INFO:teuthology.orchestra.run.smithi105.stdout: "session_remove": 4, 2024-05-23T00:11:08.070 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_open": 2, 2024-05-23T00:11:08.070 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_stale": 0, 2024-05-23T00:11:08.070 INFO:teuthology.orchestra.run.smithi105.stdout: "total_load": 32, 2024-05-23T00:11:08.070 INFO:teuthology.orchestra.run.smithi105.stdout: "average_load": 16, 2024-05-23T00:11:08.070 INFO:teuthology.orchestra.run.smithi105.stdout: "avg_session_uptime": 621, 2024-05-23T00:11:08.070 INFO:teuthology.orchestra.run.smithi105.stdout: "mdthresh_evicted": 0 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 360, 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 360, 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:08.071 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:08.072 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:08.072 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:08.072 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:08.072 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:08.072 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:08.072 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:08.072 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:08.072 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:08.073 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:08.073 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:08.073 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:08.073 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:08.073 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:08.073 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:08.073 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:08.074 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:08.074 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:08.074 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:08.074 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:08.074 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:08.074 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:08.074 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:08.074 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:08.075 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:08.075 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-05-23T00:11:08.075 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:08.075 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:08.075 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:08.075 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:08.075 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 350113, 2024-05-23T00:11:08.075 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 518, 2024-05-23T00:11:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 528, 2024-05-23T00:11:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 6, 2024-05-23T00:11:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-05-23T00:11:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-05-23T00:11:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:08.076 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:08.077 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 9896, 2024-05-23T00:11:08.077 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 190, 2024-05-23T00:11:08.077 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:08.077 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:08.077 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:08.077 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-05-23T00:11:08.077 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 1602242, 2024-05-23T00:11:08.077 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 18138, 2024-05-23T00:11:08.078 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:08.078 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-05-23T00:11:08.078 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:08.078 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-05-23T00:11:08.078 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.078 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter": { 2024-05-23T00:11:08.078 INFO:teuthology.orchestra.run.smithi105.stdout: "op_active": 0, 2024-05-23T00:11:08.078 INFO:teuthology.orchestra.run.smithi105.stdout: "op_laggy": 0, 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send": 5268, 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send_bytes": 11062976, 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: "op_resend": 31, 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: "op_reply": 5268, 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: "op_latency": { 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 5268, 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 585.443051507, 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.111131938 2024-05-23T00:11:08.079 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.080 INFO:teuthology.orchestra.run.smithi105.stdout: "op_inflight": 0, 2024-05-23T00:11:08.080 INFO:teuthology.orchestra.run.smithi105.stdout: "oplen_avg": { 2024-05-23T00:11:08.080 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 5268, 2024-05-23T00:11:08.080 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 11303 2024-05-23T00:11:08.080 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.080 INFO:teuthology.orchestra.run.smithi105.stdout: "op": 5268, 2024-05-23T00:11:08.080 INFO:teuthology.orchestra.run.smithi105.stdout: "op_r": 51, 2024-05-23T00:11:08.080 INFO:teuthology.orchestra.run.smithi105.stdout: "op_w": 5217, 2024-05-23T00:11:08.081 INFO:teuthology.orchestra.run.smithi105.stdout: "op_rmw": 0, 2024-05-23T00:11:08.081 INFO:teuthology.orchestra.run.smithi105.stdout: "op_pg": 0, 2024-05-23T00:11:08.081 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_stat": 186, 2024-05-23T00:11:08.081 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_create": 3630, 2024-05-23T00:11:08.081 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_read": 13, 2024-05-23T00:11:08.081 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_write": 390, 2024-05-23T00:11:08.081 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writefull": 74, 2024-05-23T00:11:08.081 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writesame": 0, 2024-05-23T00:11:08.082 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_append": 0, 2024-05-23T00:11:08.082 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_zero": 0, 2024-05-23T00:11:08.082 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_truncate": 0, 2024-05-23T00:11:08.082 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_delete": 792, 2024-05-23T00:11:08.082 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_mapext": 0, 2024-05-23T00:11:08.082 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_sparse_read": 0, 2024-05-23T00:11:08.082 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_clonerange": 0, 2024-05-23T00:11:08.082 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_getxattr": 38, 2024-05-23T00:11:08.083 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_setxattr": 5478, 2024-05-23T00:11:08.083 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_cmpxattr": 0, 2024-05-23T00:11:08.083 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_rmxattr": 0, 2024-05-23T00:11:08.083 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_resetxattrs": 0, 2024-05-23T00:11:08.083 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_call": 0, 2024-05-23T00:11:08.083 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_watch": 0, 2024-05-23T00:11:08.083 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_notify": 0, 2024-05-23T00:11:08.083 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_src_cmpxattr": 0, 2024-05-23T00:11:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls": 0, 2024-05-23T00:11:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls_filter": 0, 2024-05-23T00:11:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_other": 2, 2024-05-23T00:11:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_active": 0, 2024-05-23T00:11:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_send": 0, 2024-05-23T00:11:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_resend": 0, 2024-05-23T00:11:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_ping": 0, 2024-05-23T00:11:08.084 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_active": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_send": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_resend": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_active": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_send": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_resend": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_active": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_send": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_resend": 0, 2024-05-23T00:11:08.085 INFO:teuthology.orchestra.run.smithi105.stdout: "command_active": 0, 2024-05-23T00:11:08.086 INFO:teuthology.orchestra.run.smithi105.stdout: "command_send": 0, 2024-05-23T00:11:08.086 INFO:teuthology.orchestra.run.smithi105.stdout: "command_resend": 0, 2024-05-23T00:11:08.086 INFO:teuthology.orchestra.run.smithi105.stdout: "map_epoch": 105, 2024-05-23T00:11:08.086 INFO:teuthology.orchestra.run.smithi105.stdout: "map_full": 0, 2024-05-23T00:11:08.086 INFO:teuthology.orchestra.run.smithi105.stdout: "map_inc": 19, 2024-05-23T00:11:08.086 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_sessions": 12, 2024-05-23T00:11:08.086 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_open": 12, 2024-05-23T00:11:08.086 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_close": 0, 2024-05-23T00:11:08.087 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_laggy": 0, 2024-05-23T00:11:08.087 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_wr": 585, 2024-05-23T00:11:08.087 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_rd": 6, 2024-05-23T00:11:08.087 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_del": 109 2024-05-23T00:11:08.087 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.087 INFO:teuthology.orchestra.run.smithi105.stdout: "oft": { 2024-05-23T00:11:08.087 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_objs": 1, 2024-05-23T00:11:08.087 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_kv_pairs": 0, 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_updates": 0, 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_removes": 0 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: "osd-slow-ops": { 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_ops_count": 0 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: "purge_queue": { 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops": 0, 2024-05-23T00:11:08.088 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops_high_water": 16, 2024-05-23T00:11:08.089 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing": 0, 2024-05-23T00:11:08.089 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_high_water": 8, 2024-05-23T00:11:08.089 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed_ops": 278, 2024-05-23T00:11:08.089 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed": 139, 2024-05-23T00:11:08.089 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_item_in_journal": 0 2024-05-23T00:11:08.089 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.089 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:08.089 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 13974, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 7607934, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 13974, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-23T00:11:08.090 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 13974, 2024-05-23T00:11:08.091 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 7607934, 2024-05-23T00:11:08.091 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:08.091 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.091 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.091 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.091 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.091 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.091 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_bytes": { 2024-05-23T00:11:08.092 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:08.092 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-23T00:11:08.092 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:08.092 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-23T00:11:08.092 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-23T00:11:08.092 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:08.092 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:08.092 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 5268, 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 11078091, 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 5268, 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 11078091, 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.093 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.094 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_ops": { 2024-05-23T00:11:08.094 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:08.094 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 1024, 2024-05-23T00:11:08.094 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:08.094 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-23T00:11:08.094 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-23T00:11:08.094 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:08.094 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:08.095 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 5268, 2024-05-23T00:11:08.095 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 5268, 2024-05-23T00:11:08.095 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 5268, 2024-05-23T00:11:08.095 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 5268, 2024-05-23T00:11:08.095 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:08.095 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.095 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.095 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle": { 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 139, 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 15151, 2024-05-23T00:11:08.096 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:08.097 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 139, 2024-05-23T00:11:08.097 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-23T00:11:08.097 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-23T00:11:08.097 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 10, 2024-05-23T00:11:08.097 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 15151, 2024-05-23T00:11:08.097 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:08.097 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.097 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.098 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.098 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.098 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:08.098 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle-0x55d498b1e500": { 2024-05-23T00:11:08.098 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 136, 2024-05-23T00:11:08.098 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-23T00:11:08.098 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:08.098 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 2779, 2024-05-23T00:11:08.099 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 7277600, 2024-05-23T00:11:08.099 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:08.099 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 2779, 2024-05-23T00:11:08.099 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-23T00:11:08.099 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-23T00:11:08.099 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 380, 2024-05-23T00:11:08.099 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 7277464, 2024-05-23T00:11:08.099 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:08.100 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:08.100 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:08.100 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:08.100 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.100 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:08.100 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-23T00:11:08.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:08 smithi103 ceph-mon[31370]: pgmap v330: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:11:08.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:08 smithi103 ceph-mon[31370]: osdmap e105: 12 total, 12 up, 12 in 2024-05-23T00:11:08.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:08 smithi105 ceph-mon[36426]: pgmap v330: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:11:08.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:08 smithi105 ceph-mon[36426]: osdmap e105: 12 total, 12 up, 12 in 2024-05-23T00:11:08.842 INFO:tasks.check_counter:Daemon mds.b mds.exported=70 2024-05-23T00:11:08.842 INFO:tasks.check_counter:Daemon mds.b mds.imported=79 2024-05-23T00:11:08.842 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=40 2024-05-23T00:11:08.843 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=293 2024-05-23T00:11:08.843 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-05-23T00:11:08.843 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-05-23T00:11:08.843 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-05-23T00:11:08.843 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=0 2024-05-23T00:11:08.843 DEBUG:tasks.check_counter:Getting stats from e 2024-05-23T00:11:08.844 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-23T00:11:08.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:08 smithi174 ceph-mon[36179]: pgmap v330: 129 pgs: 129 active+clean; 84 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-05-23T00:11:08.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:08 smithi174 ceph-mon[36179]: osdmap e105: 12 total, 12 up, 12 in 2024-05-23T00:11:09.084 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-23T00:11:10.626 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:10 smithi105 ceph-mon[36426]: pgmap v332: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:10.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:10 smithi103 ceph-mon[31370]: pgmap v332: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:10.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:10 smithi174 ceph-mon[36179]: pgmap v332: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:10.868 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-23T00:11:10.869 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:10.869 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 242, 2024-05-23T00:11:10.869 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 228, 2024-05-23T00:11:10.869 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1725872, 2024-05-23T00:11:10.869 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 139872, 2024-05-23T00:11:10.869 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-05-23T00:11:10.869 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-05-23T00:11:10.870 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.048787357, 2024-05-23T00:11:10.870 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.014295213, 2024-05-23T00:11:10.870 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.038131980, 2024-05-23T00:11:10.870 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.005685024, 2024-05-23T00:11:10.870 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:10.870 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 228, 2024-05-23T00:11:10.870 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.012631187, 2024-05-23T00:11:10.870 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000055399 2024-05-23T00:11:10.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.871 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:10.871 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:10.871 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:10.871 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:10.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.871 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1725872, 2024-05-23T00:11:10.871 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 139872 2024-05-23T00:11:10.872 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.872 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:10.872 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 2, 2024-05-23T00:11:10.872 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 173, 2024-05-23T00:11:10.872 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 320, 2024-05-23T00:11:10.872 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 186016, 2024-05-23T00:11:10.872 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-05-23T00:11:10.872 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-05-23T00:11:10.873 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.012077460, 2024-05-23T00:11:10.873 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.008148036, 2024-05-23T00:11:10.873 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.001455876, 2024-05-23T00:11:10.873 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:10.873 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:10.873 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 170, 2024-05-23T00:11:10.873 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.018378663, 2024-05-23T00:11:10.873 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000108109 2024-05-23T00:11:10.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.874 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:10.874 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:10.874 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:10.874 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:10.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.874 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-23T00:11:10.875 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 186016 2024-05-23T00:11:10.875 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.875 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:10.875 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 0, 2024-05-23T00:11:10.875 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 2, 2024-05-23T00:11:10.875 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 0, 2024-05-23T00:11:10.875 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 0, 2024-05-23T00:11:10.875 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-05-23T00:11:10.876 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-05-23T00:11:10.876 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.000384397, 2024-05-23T00:11:10.876 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000000000, 2024-05-23T00:11:10.876 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000320943, 2024-05-23T00:11:10.876 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:10.876 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:10.876 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:10.877 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:10.877 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:10.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.877 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:10.877 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:10.877 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:10.877 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:10.877 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.878 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-23T00:11:10.878 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 0 2024-05-23T00:11:10.878 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.878 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-05-23T00:11:10.878 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-05-23T00:11:10.879 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-05-23T00:11:10.879 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.879 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-05-23T00:11:10.879 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:11:10.879 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-05-23T00:11:10.879 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:10.879 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:10.880 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:10.880 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:10.880 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:10.880 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:10.880 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:10.880 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:10.880 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:10.880 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:10.881 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:10.881 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:10.881 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:10.881 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:10.881 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:10.881 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:10.881 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:10.881 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:10.882 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:10.882 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:10.882 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:10.882 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:10.882 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:10.882 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:10.882 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:10.882 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:10.883 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:10.883 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:10.883 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:10.883 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-05-23T00:11:10.883 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:10.883 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:10.883 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:10.884 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:10.884 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 148732, 2024-05-23T00:11:10.884 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 31, 2024-05-23T00:11:10.884 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:11:10.884 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-05-23T00:11:10.884 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-05-23T00:11:10.884 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-05-23T00:11:10.884 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:10.885 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:10.885 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:10.885 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:10.885 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-05-23T00:11:10.885 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-05-23T00:11:10.885 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:10.885 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:10.885 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:10.886 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-05-23T00:11:10.886 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-05-23T00:11:10.886 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-05-23T00:11:10.886 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:10.886 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-05-23T00:11:10.886 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:10.886 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-05-23T00:11:10.887 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:10.887 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:10.887 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:10.887 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-23T00:11:10.887 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:10.887 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 240, 2024-05-23T00:11:10.887 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1689309, 2024-05-23T00:11:10.887 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:10.888 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 240, 2024-05-23T00:11:10.888 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-23T00:11:10.888 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-23T00:11:10.888 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 240, 2024-05-23T00:11:10.888 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1689309, 2024-05-23T00:11:10.888 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:10.888 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:10.889 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:10.889 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:10.889 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:10.889 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:10.889 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-23T00:11:11.805 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-05-23T00:11:11.805 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-05-23T00:11:11.805 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.e 2024-05-23T00:11:11.806 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.e 2024-05-23T00:11:11.806 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-05-23T00:11:11.806 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-05-23T00:11:11.806 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-05-23T00:11:11.806 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-05-23T00:11:11.806 DEBUG:tasks.check_counter:Getting stats from h 2024-05-23T00:11:11.806 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-23T00:11:12.045 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-23T00:11:12.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:12 smithi105 ceph-mon[36426]: pgmap v333: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:12.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:12 smithi103 ceph-mon[31370]: pgmap v333: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:12.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:12 smithi174 ceph-mon[36179]: pgmap v333: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:13.814 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-23T00:11:13.814 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:13.814 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 757, 2024-05-23T00:11:13.815 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 731, 2024-05-23T00:11:13.815 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 6126180, 2024-05-23T00:11:13.815 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 296810, 2024-05-23T00:11:13.815 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 6, 2024-05-23T00:11:13.815 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 2, 2024-05-23T00:11:13.815 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.105236658, 2024-05-23T00:11:13.815 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.028619862, 2024-05-23T00:11:13.816 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.052673763, 2024-05-23T00:11:13.816 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.028390086, 2024-05-23T00:11:13.816 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:13.816 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 730, 2024-05-23T00:11:13.816 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.029440302, 2024-05-23T00:11:13.816 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000040329 2024-05-23T00:11:13.816 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.816 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:13.817 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.817 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.817 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.817 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1809504, 2024-05-23T00:11:13.817 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 142256 2024-05-23T00:11:13.817 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.817 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:13.818 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 817, 2024-05-23T00:11:13.818 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 987, 2024-05-23T00:11:13.818 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 27343800, 2024-05-23T00:11:13.818 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 715671, 2024-05-23T00:11:13.818 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 8, 2024-05-23T00:11:13.818 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 6, 2024-05-23T00:11:13.818 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.112110013, 2024-05-23T00:11:13.818 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.031940986, 2024-05-23T00:11:13.819 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.040896990, 2024-05-23T00:11:13.819 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.028805690, 2024-05-23T00:11:13.819 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:13.819 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 984, 2024-05-23T00:11:13.819 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.054784809, 2024-05-23T00:11:13.819 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000055675 2024-05-23T00:11:13.819 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.819 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:13.820 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.820 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.820 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.820 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-23T00:11:13.820 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 456496 2024-05-23T00:11:13.820 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.820 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:13.821 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 1119, 2024-05-23T00:11:13.821 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 1119, 2024-05-23T00:11:13.821 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 21329310, 2024-05-23T00:11:13.821 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 355076, 2024-05-23T00:11:13.821 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 5, 2024-05-23T00:11:13.821 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 4, 2024-05-23T00:11:13.821 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.139688160, 2024-05-23T00:11:13.821 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.042869376, 2024-05-23T00:11:13.822 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.044217359, 2024-05-23T00:11:13.822 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.040412691, 2024-05-23T00:11:13.822 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:13.822 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 1118, 2024-05-23T00:11:13.822 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.056662359, 2024-05-23T00:11:13.822 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000050681 2024-05-23T00:11:13.822 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.823 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:13.823 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.823 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.823 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.823 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.823 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-23T00:11:13.824 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 448 2024-05-23T00:11:13.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.824 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-05-23T00:11:13.824 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 1, 2024-05-23T00:11:13.824 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-05-23T00:11:13.824 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.824 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-MDSRank": { 2024-05-23T00:11:13.824 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-23T00:11:13.825 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-23T00:11:13.825 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 4046, 2024-05-23T00:11:13.825 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.125765405, 2024-05-23T00:11:13.825 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000031083 2024-05-23T00:11:13.825 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.825 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.825 INFO:teuthology.orchestra.run.smithi105.stdout: "finisher-PurgeQueue": { 2024-05-23T00:11:13.825 INFO:teuthology.orchestra.run.smithi105.stdout: "queue_len": 0, 2024-05-23T00:11:13.826 INFO:teuthology.orchestra.run.smithi105.stdout: "complete_latency": { 2024-05-23T00:11:13.826 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.826 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.826 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.826 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.826 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.826 INFO:teuthology.orchestra.run.smithi105.stdout: "mds": { 2024-05-23T00:11:13.826 INFO:teuthology.orchestra.run.smithi105.stdout: "request": 0, 2024-05-23T00:11:13.827 INFO:teuthology.orchestra.run.smithi105.stdout: "reply": 0, 2024-05-23T00:11:13.827 INFO:teuthology.orchestra.run.smithi105.stdout: "reply_latency": { 2024-05-23T00:11:13.827 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.827 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.827 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.827 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.827 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_reply": 0, 2024-05-23T00:11:13.827 INFO:teuthology.orchestra.run.smithi105.stdout: "forward": 0, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_complete": 0, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_fetch_keys": 0, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_commit": 0, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_split": 0, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_merge": 0, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes": 3591, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_top": 270, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_bottom": 3321, 2024-05-23T00:11:13.828 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pin_tail": 0, 2024-05-23T00:11:13.829 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_pinned": 3210, 2024-05-23T00:11:13.829 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_expired": 0, 2024-05-23T00:11:13.829 INFO:teuthology.orchestra.run.smithi105.stdout: "inodes_with_caps": 0, 2024-05-23T00:11:13.829 INFO:teuthology.orchestra.run.smithi105.stdout: "caps": 0, 2024-05-23T00:11:13.829 INFO:teuthology.orchestra.run.smithi105.stdout: "subtrees": 14, 2024-05-23T00:11:13.829 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse": 0, 2024-05-23T00:11:13.829 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_hit": 0, 2024-05-23T00:11:13.829 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_forward": 0, 2024-05-23T00:11:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_discover": 0, 2024-05-23T00:11:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_dir_fetch": 0, 2024-05-23T00:11:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_remote_ino": 0, 2024-05-23T00:11:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: "traverse_lock": 0, 2024-05-23T00:11:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: "load_cent": 0, 2024-05-23T00:11:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: "q": 0, 2024-05-23T00:11:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: "exported": 0, 2024-05-23T00:11:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: "exported_inodes": 0, 2024-05-23T00:11:13.831 INFO:teuthology.orchestra.run.smithi105.stdout: "imported": 0, 2024-05-23T00:11:13.831 INFO:teuthology.orchestra.run.smithi105.stdout: "imported_inodes": 0, 2024-05-23T00:11:13.831 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_dir_fetch": 0, 2024-05-23T00:11:13.831 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_backtrace_fetch": 0, 2024-05-23T00:11:13.831 INFO:teuthology.orchestra.run.smithi105.stdout: "openino_peer_discover": 0, 2024-05-23T00:11:13.831 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rfiles": 0, 2024-05-23T00:11:13.831 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rbytes": 0, 2024-05-23T00:11:13.831 INFO:teuthology.orchestra.run.smithi105.stdout: "root_rsnaps": 0, 2024-05-23T00:11:13.832 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_fetch": 0, 2024-05-23T00:11:13.832 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_set_tag": 0, 2024-05-23T00:11:13.832 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_backtrace_repaired": 0, 2024-05-23T00:11:13.832 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_inotable_repaired": 0, 2024-05-23T00:11:13.832 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_inodes": 0, 2024-05-23T00:11:13.832 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dir_base_inodes": 0, 2024-05-23T00:11:13.832 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_dirfrag_rstats": 0, 2024-05-23T00:11:13.832 INFO:teuthology.orchestra.run.smithi105.stdout: "scrub_file_inodes": 0, 2024-05-23T00:11:13.833 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_inode_file_caps": 0, 2024-05-23T00:11:13.833 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_revoke": 0, 2024-05-23T00:11:13.833 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_grant": 0, 2024-05-23T00:11:13.833 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_trunc": 0, 2024-05-23T00:11:13.833 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-23T00:11:13.833 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-23T00:11:13.833 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps": 0, 2024-05-23T00:11:13.833 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_caps_dirty": 0, 2024-05-23T00:11:13.834 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_cap_release": 0, 2024-05-23T00:11:13.834 INFO:teuthology.orchestra.run.smithi105.stdout: "process_request_cap_release": 0 2024-05-23T00:11:13.834 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.834 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_cache": { 2024-05-23T00:11:13.834 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update": 0, 2024-05-23T00:11:13.834 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_update_receipt": 0, 2024-05-23T00:11:13.834 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_try_discover": 0, 2024-05-23T00:11:13.834 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_send_discover": 0, 2024-05-23T00:11:13.835 INFO:teuthology.orchestra.run.smithi105.stdout: "dir_handle_discover": 0, 2024-05-23T00:11:13.835 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays": 0, 2024-05-23T00:11:13.835 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_delayed": 0, 2024-05-23T00:11:13.835 INFO:teuthology.orchestra.run.smithi105.stdout: "num_strays_enqueuing": 0, 2024-05-23T00:11:13.835 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_created": 0, 2024-05-23T00:11:13.835 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_enqueued": 0, 2024-05-23T00:11:13.835 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_reintegrated": 0, 2024-05-23T00:11:13.835 INFO:teuthology.orchestra.run.smithi105.stdout: "strays_migrated": 0, 2024-05-23T00:11:13.836 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_processing": 0, 2024-05-23T00:11:13.836 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_enqueued": 0, 2024-05-23T00:11:13.836 INFO:teuthology.orchestra.run.smithi105.stdout: "num_recovering_prioritized": 0, 2024-05-23T00:11:13.836 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_started": 0, 2024-05-23T00:11:13.836 INFO:teuthology.orchestra.run.smithi105.stdout: "recovery_completed": 0, 2024-05-23T00:11:13.836 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_path": 0, 2024-05-23T00:11:13.836 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_quiesce_inode": 0, 2024-05-23T00:11:13.836 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_enqueue_scrub": 0, 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_exportdir": 0, 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_flush": 0, 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragmentdir": 0, 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_fragstats": 0, 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: "ireq_inodestats": 0 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_log": { 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: "evlrg": 0, 2024-05-23T00:11:13.837 INFO:teuthology.orchestra.run.smithi105.stdout: "evadd": 0, 2024-05-23T00:11:13.838 INFO:teuthology.orchestra.run.smithi105.stdout: "evex": 0, 2024-05-23T00:11:13.838 INFO:teuthology.orchestra.run.smithi105.stdout: "evtrm": 0, 2024-05-23T00:11:13.838 INFO:teuthology.orchestra.run.smithi105.stdout: "ev": 22485, 2024-05-23T00:11:13.838 INFO:teuthology.orchestra.run.smithi105.stdout: "evexg": 0, 2024-05-23T00:11:13.838 INFO:teuthology.orchestra.run.smithi105.stdout: "evexd": 0, 2024-05-23T00:11:13.838 INFO:teuthology.orchestra.run.smithi105.stdout: "segadd": 0, 2024-05-23T00:11:13.838 INFO:teuthology.orchestra.run.smithi105.stdout: "segex": 0, 2024-05-23T00:11:13.838 INFO:teuthology.orchestra.run.smithi105.stdout: "segtrm": 0, 2024-05-23T00:11:13.839 INFO:teuthology.orchestra.run.smithi105.stdout: "seg": 28, 2024-05-23T00:11:13.839 INFO:teuthology.orchestra.run.smithi105.stdout: "segmjr": 3, 2024-05-23T00:11:13.839 INFO:teuthology.orchestra.run.smithi105.stdout: "segexg": 0, 2024-05-23T00:11:13.839 INFO:teuthology.orchestra.run.smithi105.stdout: "segexd": 0, 2024-05-23T00:11:13.839 INFO:teuthology.orchestra.run.smithi105.stdout: "expos": 4194304, 2024-05-23T00:11:13.839 INFO:teuthology.orchestra.run.smithi105.stdout: "wrpos": 56490353, 2024-05-23T00:11:13.839 INFO:teuthology.orchestra.run.smithi105.stdout: "rdpos": 56489779, 2024-05-23T00:11:13.839 INFO:teuthology.orchestra.run.smithi105.stdout: "jlat": { 2024-05-23T00:11:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: "replayed": 22485 2024-05-23T00:11:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_mem": { 2024-05-23T00:11:13.840 INFO:teuthology.orchestra.run.smithi105.stdout: "ino": 318, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "ino+": 7778, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "ino-": 7460, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "dir": 75, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "dir+": 311, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "dir-": 236, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "dn": 3591, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "dn+": 9103, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "dn-": 5512, 2024-05-23T00:11:13.841 INFO:teuthology.orchestra.run.smithi105.stdout: "cap": 0, 2024-05-23T00:11:13.842 INFO:teuthology.orchestra.run.smithi105.stdout: "cap+": 0, 2024-05-23T00:11:13.842 INFO:teuthology.orchestra.run.smithi105.stdout: "cap-": 0, 2024-05-23T00:11:13.842 INFO:teuthology.orchestra.run.smithi105.stdout: "rss": 67556, 2024-05-23T00:11:13.842 INFO:teuthology.orchestra.run.smithi105.stdout: "heap": 215420 2024-05-23T00:11:13.842 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.842 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_server": { 2024-05-23T00:11:13.842 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_client_request": 0, 2024-05-23T00:11:13.842 INFO:teuthology.orchestra.run.smithi105.stdout: "dispatch_server_request": 0, 2024-05-23T00:11:13.843 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_request": 0, 2024-05-23T00:11:13.843 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_client_session": 0, 2024-05-23T00:11:13.843 INFO:teuthology.orchestra.run.smithi105.stdout: "handle_peer_request": 0, 2024-05-23T00:11:13.843 INFO:teuthology.orchestra.run.smithi105.stdout: "req_create_latency": { 2024-05-23T00:11:13.843 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.843 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.843 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.843 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.844 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getattr_latency": { 2024-05-23T00:11:13.844 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.844 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.844 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.844 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.844 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getfilelock_latency": { 2024-05-23T00:11:13.844 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.844 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.845 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.845 INFO:teuthology.orchestra.run.smithi105.stdout: "req_link_latency": { 2024-05-23T00:11:13.845 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.845 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.845 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.845 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.845 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookup_latency": { 2024-05-23T00:11:13.846 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.846 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.846 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.846 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.846 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookuphash_latency": { 2024-05-23T00:11:13.846 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.846 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.846 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.847 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupino_latency": { 2024-05-23T00:11:13.847 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.847 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.847 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.847 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.847 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupname_latency": { 2024-05-23T00:11:13.847 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.848 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.848 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.848 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupparent_latency": { 2024-05-23T00:11:13.848 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.848 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.848 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.848 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.849 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lookupsnap_latency": { 2024-05-23T00:11:13.849 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.849 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.849 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.849 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.849 INFO:teuthology.orchestra.run.smithi105.stdout: "req_lssnap_latency": { 2024-05-23T00:11:13.849 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.849 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.850 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.850 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.851 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mkdir_latency": { 2024-05-23T00:11:13.851 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.851 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.851 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.851 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.851 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mknod_latency": { 2024-05-23T00:11:13.851 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.851 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.852 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.852 INFO:teuthology.orchestra.run.smithi105.stdout: "req_mksnap_latency": { 2024-05-23T00:11:13.852 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.852 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.852 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.852 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.852 INFO:teuthology.orchestra.run.smithi105.stdout: "req_open_latency": { 2024-05-23T00:11:13.853 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.853 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.853 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.853 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.853 INFO:teuthology.orchestra.run.smithi105.stdout: "req_readdir_latency": { 2024-05-23T00:11:13.853 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.853 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.853 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.854 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rename_latency": { 2024-05-23T00:11:13.854 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.854 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.854 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.854 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.854 INFO:teuthology.orchestra.run.smithi105.stdout: "req_renamesnap_latency": { 2024-05-23T00:11:13.854 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.855 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.855 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.855 INFO:teuthology.orchestra.run.smithi105.stdout: "req_snapdiff_latency": { 2024-05-23T00:11:13.855 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.855 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.855 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.855 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.856 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmdir_latency": { 2024-05-23T00:11:13.856 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.856 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.856 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.856 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.856 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmsnap_latency": { 2024-05-23T00:11:13.856 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.856 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: "req_rmxattr_latency": { 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setattr_latency": { 2024-05-23T00:11:13.857 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.858 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.858 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.858 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.858 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setdirlayout_latency": { 2024-05-23T00:11:13.858 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.858 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.858 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.859 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setfilelock_latency": { 2024-05-23T00:11:13.859 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.859 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.859 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.859 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.859 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setlayout_latency": { 2024-05-23T00:11:13.860 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.860 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.860 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.860 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.860 INFO:teuthology.orchestra.run.smithi105.stdout: "req_setxattr_latency": { 2024-05-23T00:11:13.860 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.860 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.860 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: "req_symlink_latency": { 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: "req_unlink_latency": { 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.861 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_revoke_eviction": 0, 2024-05-23T00:11:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: "cap_acquisition_throttle": 0, 2024-05-23T00:11:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: "req_getvxattr_latency": { 2024-05-23T00:11:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.862 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_sessions": { 2024-05-23T00:11:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: "session_count": 2, 2024-05-23T00:11:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: "session_add": 4, 2024-05-23T00:11:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: "session_remove": 2, 2024-05-23T00:11:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_open": 3, 2024-05-23T00:11:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: "sessions_stale": 0, 2024-05-23T00:11:13.864 INFO:teuthology.orchestra.run.smithi105.stdout: "total_load": 0, 2024-05-23T00:11:13.864 INFO:teuthology.orchestra.run.smithi105.stdout: "average_load": 0, 2024-05-23T00:11:13.864 INFO:teuthology.orchestra.run.smithi105.stdout: "avg_session_uptime": 3079, 2024-05-23T00:11:13.864 INFO:teuthology.orchestra.run.smithi105.stdout: "mdthresh_evicted": 0 2024-05-23T00:11:13.864 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.864 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-05-23T00:11:13.864 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:11:13.864 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-05-23T00:11:13.865 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:13.865 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:13.865 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:13.865 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:13.865 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:13.865 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:13.865 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:13.865 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:13.866 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:13.866 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:13.866 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:13.866 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:13.866 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:13.866 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:13.866 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:13.866 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:13.867 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:13.867 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:13.867 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:13.867 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:13.867 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:13.867 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:13.867 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:13.867 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:13.868 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:13.869 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:13.869 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 227126, 2024-05-23T00:11:13.869 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 409, 2024-05-23T00:11:13.869 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:11:13.869 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-05-23T00:11:13.869 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-05-23T00:11:13.869 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-05-23T00:11:13.869 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:13.870 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:13.870 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:13.870 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:13.870 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 10136, 2024-05-23T00:11:13.870 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 196, 2024-05-23T00:11:13.870 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:13.870 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:13.870 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:13.871 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-05-23T00:11:13.871 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 3409384, 2024-05-23T00:11:13.871 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 24048, 2024-05-23T00:11:13.871 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:13.871 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-05-23T00:11:13.871 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:13.871 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-05-23T00:11:13.871 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.872 INFO:teuthology.orchestra.run.smithi105.stdout: "objecter": { 2024-05-23T00:11:13.872 INFO:teuthology.orchestra.run.smithi105.stdout: "op_active": 0, 2024-05-23T00:11:13.872 INFO:teuthology.orchestra.run.smithi105.stdout: "op_laggy": 0, 2024-05-23T00:11:13.872 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send": 2416, 2024-05-23T00:11:13.872 INFO:teuthology.orchestra.run.smithi105.stdout: "op_send_bytes": 21, 2024-05-23T00:11:13.872 INFO:teuthology.orchestra.run.smithi105.stdout: "op_resend": 3, 2024-05-23T00:11:13.872 INFO:teuthology.orchestra.run.smithi105.stdout: "op_reply": 2416, 2024-05-23T00:11:13.872 INFO:teuthology.orchestra.run.smithi105.stdout: "op_latency": { 2024-05-23T00:11:13.873 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2416, 2024-05-23T00:11:13.873 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 2.749944050, 2024-05-23T00:11:13.873 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.001138221 2024-05-23T00:11:13.873 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.873 INFO:teuthology.orchestra.run.smithi105.stdout: "op_inflight": 0, 2024-05-23T00:11:13.873 INFO:teuthology.orchestra.run.smithi105.stdout: "oplen_avg": { 2024-05-23T00:11:13.873 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 2416, 2024-05-23T00:11:13.873 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 2417 2024-05-23T00:11:13.874 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.874 INFO:teuthology.orchestra.run.smithi105.stdout: "op": 2416, 2024-05-23T00:11:13.874 INFO:teuthology.orchestra.run.smithi105.stdout: "op_r": 2416, 2024-05-23T00:11:13.874 INFO:teuthology.orchestra.run.smithi105.stdout: "op_w": 0, 2024-05-23T00:11:13.874 INFO:teuthology.orchestra.run.smithi105.stdout: "op_rmw": 0, 2024-05-23T00:11:13.874 INFO:teuthology.orchestra.run.smithi105.stdout: "op_pg": 0, 2024-05-23T00:11:13.874 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_stat": 1526, 2024-05-23T00:11:13.874 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_create": 0, 2024-05-23T00:11:13.875 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_read": 888, 2024-05-23T00:11:13.875 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_write": 0, 2024-05-23T00:11:13.875 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writefull": 0, 2024-05-23T00:11:13.875 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_writesame": 0, 2024-05-23T00:11:13.875 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_append": 0, 2024-05-23T00:11:13.875 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_zero": 0, 2024-05-23T00:11:13.875 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_truncate": 0, 2024-05-23T00:11:13.875 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_delete": 0, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_mapext": 0, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_sparse_read": 0, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_clonerange": 0, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_getxattr": 1, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_setxattr": 0, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_cmpxattr": 0, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_rmxattr": 0, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_resetxattrs": 0, 2024-05-23T00:11:13.876 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_call": 0, 2024-05-23T00:11:13.877 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_watch": 0, 2024-05-23T00:11:13.877 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_notify": 0, 2024-05-23T00:11:13.877 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_src_cmpxattr": 0, 2024-05-23T00:11:13.877 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls": 0, 2024-05-23T00:11:13.877 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_pgls_filter": 0, 2024-05-23T00:11:13.877 INFO:teuthology.orchestra.run.smithi105.stdout: "osdop_other": 0, 2024-05-23T00:11:13.877 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_active": 0, 2024-05-23T00:11:13.877 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_send": 0, 2024-05-23T00:11:13.878 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_resend": 0, 2024-05-23T00:11:13.878 INFO:teuthology.orchestra.run.smithi105.stdout: "linger_ping": 0, 2024-05-23T00:11:13.878 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_active": 0, 2024-05-23T00:11:13.878 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_send": 0, 2024-05-23T00:11:13.878 INFO:teuthology.orchestra.run.smithi105.stdout: "poolop_resend": 0, 2024-05-23T00:11:13.878 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_active": 0, 2024-05-23T00:11:13.878 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_send": 0, 2024-05-23T00:11:13.878 INFO:teuthology.orchestra.run.smithi105.stdout: "poolstat_resend": 0, 2024-05-23T00:11:13.879 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_active": 0, 2024-05-23T00:11:13.879 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_send": 0, 2024-05-23T00:11:13.879 INFO:teuthology.orchestra.run.smithi105.stdout: "statfs_resend": 0, 2024-05-23T00:11:13.879 INFO:teuthology.orchestra.run.smithi105.stdout: "command_active": 0, 2024-05-23T00:11:13.879 INFO:teuthology.orchestra.run.smithi105.stdout: "command_send": 0, 2024-05-23T00:11:13.879 INFO:teuthology.orchestra.run.smithi105.stdout: "command_resend": 0, 2024-05-23T00:11:13.879 INFO:teuthology.orchestra.run.smithi105.stdout: "map_epoch": 106, 2024-05-23T00:11:13.879 INFO:teuthology.orchestra.run.smithi105.stdout: "map_full": 0, 2024-05-23T00:11:13.880 INFO:teuthology.orchestra.run.smithi105.stdout: "map_inc": 15, 2024-05-23T00:11:13.880 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_sessions": 8, 2024-05-23T00:11:13.880 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_open": 8, 2024-05-23T00:11:13.880 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_session_close": 0, 2024-05-23T00:11:13.880 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_laggy": 0, 2024-05-23T00:11:13.880 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_wr": 0, 2024-05-23T00:11:13.880 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_rd": 2, 2024-05-23T00:11:13.880 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_del": 0 2024-05-23T00:11:13.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.881 INFO:teuthology.orchestra.run.smithi105.stdout: "oft": { 2024-05-23T00:11:13.881 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_objs": 0, 2024-05-23T00:11:13.881 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_kv_pairs": 0, 2024-05-23T00:11:13.881 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_updates": 0, 2024-05-23T00:11:13.881 INFO:teuthology.orchestra.run.smithi105.stdout: "omap_total_removes": 0 2024-05-23T00:11:13.881 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.881 INFO:teuthology.orchestra.run.smithi105.stdout: "osd-slow-ops": { 2024-05-23T00:11:13.882 INFO:teuthology.orchestra.run.smithi105.stdout: "slow_ops_count": 0 2024-05-23T00:11:13.882 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.882 INFO:teuthology.orchestra.run.smithi105.stdout: "purge_queue": { 2024-05-23T00:11:13.882 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops": 0, 2024-05-23T00:11:13.882 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_ops_high_water": 0, 2024-05-23T00:11:13.882 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing": 0, 2024-05-23T00:11:13.882 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executing_high_water": 0, 2024-05-23T00:11:13.882 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed_ops": 0, 2024-05-23T00:11:13.883 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_executed": 0, 2024-05-23T00:11:13.883 INFO:teuthology.orchestra.run.smithi105.stdout: "pq_item_in_journal": 0 2024-05-23T00:11:13.883 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.883 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:13.883 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:13.883 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-23T00:11:13.883 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:13.883 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 2689, 2024-05-23T00:11:13.884 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 54540428, 2024-05-23T00:11:13.884 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:13.884 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 2689, 2024-05-23T00:11:13.884 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-23T00:11:13.884 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-23T00:11:13.884 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 2689, 2024-05-23T00:11:13.884 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 54540428, 2024-05-23T00:11:13.884 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:13.885 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.885 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.885 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.885 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.885 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.885 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_bytes": { 2024-05-23T00:11:13.885 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:13.885 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-23T00:11:13.886 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:13.886 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-23T00:11:13.886 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-23T00:11:13.886 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:13.886 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:13.886 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 2416, 2024-05-23T00:11:13.887 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 52306954, 2024-05-23T00:11:13.887 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 2416, 2024-05-23T00:11:13.887 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 52306954, 2024-05-23T00:11:13.887 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:13.887 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.887 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.887 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.887 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-objecter_ops": { 2024-05-23T00:11:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 1024, 2024-05-23T00:11:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-23T00:11:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-23T00:11:13.888 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 2416, 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 2416, 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 2416, 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 2416, 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.889 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.890 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.890 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.890 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle": { 2024-05-23T00:11:13.890 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:13.890 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-23T00:11:13.890 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:13.890 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-23T00:11:13.890 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-23T00:11:13.891 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:13.891 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:13.891 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-23T00:11:13.891 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-23T00:11:13.891 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-05-23T00:11:13.891 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-05-23T00:11:13.891 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:13.891 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-write_buf_throttle-0x55a18c3b66e0": { 2024-05-23T00:11:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 3758096384, 2024-05-23T00:11:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 0, 2024-05-23T00:11:13.893 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:13.894 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:13.894 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:13.894 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:13.894 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.894 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:13.894 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-23T00:11:14.683 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-05-23T00:11:14.684 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-05-23T00:11:14.684 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-05-23T00:11:14.684 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-05-23T00:11:14.684 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-05-23T00:11:14.684 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-05-23T00:11:14.684 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-05-23T00:11:14.684 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-05-23T00:11:14.684 DEBUG:tasks.check_counter:Getting stats from k 2024-05-23T00:11:14.685 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-23T00:11:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:14 smithi105 ceph-mon[36426]: pgmap v334: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:14 smithi105 ceph-mon[36426]: osdmap e106: 12 total, 12 up, 12 in 2024-05-23T00:11:14.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:14 smithi103 ceph-mon[31370]: pgmap v334: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:14.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:14 smithi103 ceph-mon[31370]: osdmap e106: 12 total, 12 up, 12 in 2024-05-23T00:11:14.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:14 smithi174 ceph-mon[36179]: pgmap v334: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-05-23T00:11:14.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:14 smithi174 ceph-mon[36179]: osdmap e106: 12 total, 12 up, 12 in 2024-05-23T00:11:14.921 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.b/config 2024-05-23T00:11:16.775 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:16 smithi105 ceph-mon[36426]: pgmap v336: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:11:16.798 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-05-23T00:11:16.798 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:16.798 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 4, 2024-05-23T00:11:16.799 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 4, 2024-05-23T00:11:16.799 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 3024, 2024-05-23T00:11:16.799 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 448, 2024-05-23T00:11:16.799 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 4, 2024-05-23T00:11:16.799 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 0, 2024-05-23T00:11:16.799 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.001275430, 2024-05-23T00:11:16.799 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.000029633, 2024-05-23T00:11:16.799 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.000530091, 2024-05-23T00:11:16.800 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:16.800 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:16.800 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 3, 2024-05-23T00:11:16.800 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000671035, 2024-05-23T00:11:16.800 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000223678 2024-05-23T00:11:16.800 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.800 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:16.800 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:16.801 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:16.801 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:16.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.801 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-23T00:11:16.801 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 448 2024-05-23T00:11:16.801 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.801 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:16.801 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 3, 2024-05-23T00:11:16.802 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 178, 2024-05-23T00:11:16.802 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 480, 2024-05-23T00:11:16.802 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 264768, 2024-05-23T00:11:16.802 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 6, 2024-05-23T00:11:16.802 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 3, 2024-05-23T00:11:16.802 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.012434449, 2024-05-23T00:11:16.802 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.008159306, 2024-05-23T00:11:16.802 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.001705758, 2024-05-23T00:11:16.803 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:16.803 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:16.803 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 174, 2024-05-23T00:11:16.803 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.020285010, 2024-05-23T00:11:16.803 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000116580 2024-05-23T00:11:16.803 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.803 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:16.803 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:16.804 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:16.804 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:16.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.804 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-23T00:11:16.804 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 264768 2024-05-23T00:11:16.804 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.804 INFO:teuthology.orchestra.run.smithi105.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:16.805 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_messages": 242, 2024-05-23T00:11:16.805 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages": 230, 2024-05-23T00:11:16.805 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_bytes": 1723936, 2024-05-23T00:11:16.805 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_bytes": 141984, 2024-05-23T00:11:16.805 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_created_connections": 2, 2024-05-23T00:11:16.805 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_active_connections": 1, 2024-05-23T00:11:16.805 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_total_time": 0.048120060, 2024-05-23T00:11:16.805 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_send_time": 0.014331466, 2024-05-23T00:11:16.806 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_recv_time": 0.036886951, 2024-05-23T00:11:16.806 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_running_fast_dispatch_time": 0.006075399, 2024-05-23T00:11:16.806 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:16.806 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 229, 2024-05-23T00:11:16.806 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.012498829, 2024-05-23T00:11:16.806 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000054580 2024-05-23T00:11:16.806 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.806 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:16.807 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:16.807 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:16.807 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:16.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.807 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_recv_encrypted_bytes": 1723936, 2024-05-23T00:11:16.807 INFO:teuthology.orchestra.run.smithi105.stdout: "msgr_send_encrypted_bytes": 141984 2024-05-23T00:11:16.807 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.807 INFO:teuthology.orchestra.run.smithi105.stdout: "cct": { 2024-05-23T00:11:16.808 INFO:teuthology.orchestra.run.smithi105.stdout: "total_workers": 0, 2024-05-23T00:11:16.808 INFO:teuthology.orchestra.run.smithi105.stdout: "unhealthy_workers": 0 2024-05-23T00:11:16.808 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.808 INFO:teuthology.orchestra.run.smithi105.stdout: "mempool": { 2024-05-23T00:11:16.808 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:11:16.808 INFO:teuthology.orchestra.run.smithi105.stdout: "bloom_filter_items": 0, 2024-05-23T00:11:16.808 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:16.808 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:16.809 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:16.809 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:16.809 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:16.809 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:16.809 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:16.809 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:16.809 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:16.809 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:16.810 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:16.810 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:16.810 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:16.810 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:16.810 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:16.810 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:16.810 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:16.810 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:16.811 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:16.812 INFO:teuthology.orchestra.run.smithi105.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:16.812 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:16.812 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_items": 0, 2024-05-23T00:11:16.812 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:16.812 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:16.812 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:16.812 INFO:teuthology.orchestra.run.smithi105.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:16.812 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_bytes": 148709, 2024-05-23T00:11:16.813 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_anon_items": 31, 2024-05-23T00:11:16.813 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:11:16.813 INFO:teuthology.orchestra.run.smithi105.stdout: "buffer_meta_items": 0, 2024-05-23T00:11:16.813 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_bytes": 0, 2024-05-23T00:11:16.813 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_items": 0, 2024-05-23T00:11:16.813 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:16.813 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:16.813 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_bytes": 0, 2024-05-23T00:11:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_items": 0, 2024-05-23T00:11:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: "pgmap_items": 0, 2024-05-23T00:11:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_bytes": 928, 2024-05-23T00:11:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: "mds_co_items": 2, 2024-05-23T00:11:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_1_items": 0, 2024-05-23T00:11:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: "unittest_2_items": 0 2024-05-23T00:11:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: }, 2024-05-23T00:11:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: "val": 0, 2024-05-23T00:11:16.816 INFO:teuthology.orchestra.run.smithi105.stdout: "max": 104857600, 2024-05-23T00:11:16.816 INFO:teuthology.orchestra.run.smithi105.stdout: "get_started": 0, 2024-05-23T00:11:16.816 INFO:teuthology.orchestra.run.smithi105.stdout: "get": 245, 2024-05-23T00:11:16.816 INFO:teuthology.orchestra.run.smithi105.stdout: "get_sum": 1689856, 2024-05-23T00:11:16.816 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:16.816 INFO:teuthology.orchestra.run.smithi105.stdout: "get_or_fail_success": 245, 2024-05-23T00:11:16.816 INFO:teuthology.orchestra.run.smithi105.stdout: "take": 0, 2024-05-23T00:11:16.816 INFO:teuthology.orchestra.run.smithi105.stdout: "take_sum": 0, 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout: "put": 245, 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout: "put_sum": 1689856, 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout: "wait": { 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout: "avgcount": 0, 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout: "sum": 0.000000000, 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout: "avgtime": 0.000000000 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout: } 2024-05-23T00:11:16.817 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-05-23T00:11:16.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:16 smithi103 ceph-mon[31370]: pgmap v336: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:11:16.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:16 smithi174 ceph-mon[36179]: pgmap v336: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-23T00:11:17.516 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-05-23T00:11:17.517 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-05-23T00:11:17.517 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-05-23T00:11:17.517 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-05-23T00:11:17.517 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-05-23T00:11:17.517 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-05-23T00:11:17.517 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-05-23T00:11:17.517 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-05-23T00:11:17.517 DEBUG:tasks.check_counter:Getting stats from c 2024-05-23T00:11:17.518 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-23T00:11:17.767 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-23T00:11:18.744 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:18 smithi105 ceph-mon[36426]: pgmap v337: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2024-05-23T00:11:18.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:18 smithi103 ceph-mon[31370]: pgmap v337: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2024-05-23T00:11:18.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:18 smithi174 ceph-mon[36179]: pgmap v337: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 B/s wr, 6 op/s 2024-05-23T00:11:19.630 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-05-23T00:11:19.630 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:19.631 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1145, 2024-05-23T00:11:19.631 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1147, 2024-05-23T00:11:19.631 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 4552739, 2024-05-23T00:11:19.631 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 364120, 2024-05-23T00:11:19.631 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-05-23T00:11:19.631 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-05-23T00:11:19.631 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.124024720, 2024-05-23T00:11:19.631 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.036986578, 2024-05-23T00:11:19.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.035025674, 2024-05-23T00:11:19.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.039662576, 2024-05-23T00:11:19.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:19.632 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1145, 2024-05-23T00:11:19.632 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.053116301, 2024-05-23T00:11:19.632 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000046389 2024-05-23T00:11:19.632 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.632 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:19.633 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.633 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.633 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.633 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-23T00:11:19.633 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-05-23T00:11:19.633 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.633 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:19.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 495, 2024-05-23T00:11:19.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 668, 2024-05-23T00:11:19.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 122184, 2024-05-23T00:11:19.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 694826, 2024-05-23T00:11:19.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 8, 2024-05-23T00:11:19.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-05-23T00:11:19.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.061933795, 2024-05-23T00:11:19.634 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.025290028, 2024-05-23T00:11:19.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.013553346, 2024-05-23T00:11:19.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.016699553, 2024-05-23T00:11:19.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:19.635 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 664, 2024-05-23T00:11:19.635 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.041605636, 2024-05-23T00:11:19.635 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000062659 2024-05-23T00:11:19.635 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.635 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 480, 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 538304 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:19.636 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 1133, 2024-05-23T00:11:19.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 1104, 2024-05-23T00:11:19.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 5114989, 2024-05-23T00:11:19.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 415717, 2024-05-23T00:11:19.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 3, 2024-05-23T00:11:19.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-05-23T00:11:19.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.129536014, 2024-05-23T00:11:19.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.046113534, 2024-05-23T00:11:19.637 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.058960167, 2024-05-23T00:11:19.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.027821389, 2024-05-23T00:11:19.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:19.638 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1104, 2024-05-23T00:11:19.638 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.043503427, 2024-05-23T00:11:19.638 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000039405 2024-05-23T00:11:19.638 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.638 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:19.639 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.639 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.639 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.639 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1813232, 2024-05-23T00:11:19.639 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 144144 2024-05-23T00:11:19.639 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.639 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-05-23T00:11:19.640 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-05-23T00:11:19.640 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-05-23T00:11:19.640 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.640 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-05-23T00:11:19.640 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-05-23T00:11:19.640 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-05-23T00:11:19.640 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4104, 2024-05-23T00:11:19.641 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.141871227, 2024-05-23T00:11:19.641 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000034569 2024-05-23T00:11:19.641 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.641 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.641 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-05-23T00:11:19.641 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-05-23T00:11:19.641 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-05-23T00:11:19.641 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.642 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.642 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.642 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.642 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.642 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-05-23T00:11:19.642 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-05-23T00:11:19.642 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-05-23T00:11:19.642 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-05-23T00:11:19.643 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.643 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.643 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.643 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.643 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-05-23T00:11:19.643 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-05-23T00:11:19.643 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-05-23T00:11:19.643 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-05-23T00:11:19.644 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-05-23T00:11:19.644 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-05-23T00:11:19.644 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-05-23T00:11:19.644 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 766, 2024-05-23T00:11:19.644 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 465, 2024-05-23T00:11:19.644 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 301, 2024-05-23T00:11:19.644 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-05-23T00:11:19.644 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 102, 2024-05-23T00:11:19.645 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-05-23T00:11:19.645 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-05-23T00:11:19.645 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-05-23T00:11:19.645 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 13, 2024-05-23T00:11:19.645 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-05-23T00:11:19.645 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-05-23T00:11:19.645 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-05-23T00:11:19.645 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-05-23T00:11:19.646 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-05-23T00:11:19.646 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-05-23T00:11:19.646 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-05-23T00:11:19.646 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-05-23T00:11:19.646 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-05-23T00:11:19.646 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-05-23T00:11:19.646 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 3454, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 26744693, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 6, 2024-05-23T00:11:19.647 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-05-23T00:11:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-05-23T00:11:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-05-23T00:11:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-05-23T00:11:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-05-23T00:11:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-05-23T00:11:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-05-23T00:11:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-05-23T00:11:19.648 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-05-23T00:11:19.649 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-05-23T00:11:19.649 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-05-23T00:11:19.649 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-05-23T00:11:19.649 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-23T00:11:19.649 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-23T00:11:19.649 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-05-23T00:11:19.649 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-05-23T00:11:19.649 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-05-23T00:11:19.650 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-05-23T00:11:19.650 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.650 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-05-23T00:11:19.650 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-05-23T00:11:19.650 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-05-23T00:11:19.650 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-05-23T00:11:19.650 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-05-23T00:11:19.650 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-05-23T00:11:19.651 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-05-23T00:11:19.651 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-05-23T00:11:19.651 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-05-23T00:11:19.651 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-05-23T00:11:19.651 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-05-23T00:11:19.651 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-05-23T00:11:19.651 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-05-23T00:11:19.651 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-05-23T00:11:19.652 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-05-23T00:11:19.653 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-05-23T00:11:19.653 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-05-23T00:11:19.653 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-05-23T00:11:19.653 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.653 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-05-23T00:11:19.653 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-05-23T00:11:19.653 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-05-23T00:11:19.653 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 2779, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 2, 2024-05-23T00:11:19.654 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 14, 2024-05-23T00:11:19.655 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-05-23T00:11:19.655 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-05-23T00:11:19.655 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 9956587, 2024-05-23T00:11:19.655 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 11471904, 2024-05-23T00:11:19.655 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 11471768, 2024-05-23T00:11:19.655 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-05-23T00:11:19.655 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.655 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.656 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.656 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 2779 2024-05-23T00:11:19.656 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.656 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-05-23T00:11:19.656 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 569, 2024-05-23T00:11:19.656 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 3009, 2024-05-23T00:11:19.656 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 2440, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 41, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 134, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 93, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 766, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 3725, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 2959, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-05-23T00:11:19.657 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-05-23T00:11:19.658 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 59684, 2024-05-23T00:11:19.658 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215420 2024-05-23T00:11:19.658 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.658 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-05-23T00:11:19.658 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-05-23T00:11:19.658 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-05-23T00:11:19.658 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-05-23T00:11:19.658 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-05-23T00:11:19.659 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-05-23T00:11:19.659 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-05-23T00:11:19.659 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.659 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.659 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.659 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.659 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-05-23T00:11:19.659 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.660 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.660 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.660 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-05-23T00:11:19.660 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.660 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.660 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.660 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.661 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-05-23T00:11:19.661 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.661 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.661 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.661 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.661 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-05-23T00:11:19.661 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.661 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.662 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.662 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-05-23T00:11:19.662 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.662 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.662 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.662 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.662 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.663 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.664 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-05-23T00:11:19.664 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.664 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.664 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.664 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.664 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-05-23T00:11:19.664 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.664 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.665 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.665 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-05-23T00:11:19.665 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.665 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.665 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.665 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.666 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-05-23T00:11:19.666 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-05-23T00:11:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.667 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.668 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.669 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-05-23T00:11:19.669 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.669 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.669 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.669 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.669 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-05-23T00:11:19.669 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.670 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.670 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.670 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-05-23T00:11:19.670 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.670 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.670 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.670 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.671 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-05-23T00:11:19.671 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.671 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.671 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.671 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.671 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-05-23T00:11:19.671 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.671 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.672 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.672 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-05-23T00:11:19.672 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.672 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.672 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.672 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.672 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-05-23T00:11:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-05-23T00:11:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.673 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-05-23T00:11:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-05-23T00:11:19.674 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.675 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-05-23T00:11:19.676 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.676 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.676 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.676 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.676 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-05-23T00:11:19.676 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.676 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.676 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.677 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-05-23T00:11:19.677 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.677 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.677 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.677 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.677 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-05-23T00:11:19.677 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-05-23T00:11:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-05-23T00:11:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-05-23T00:11:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 2, 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 6, 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 4, 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 4, 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 3920, 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-05-23T00:11:19.679 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.680 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-05-23T00:11:19.680 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:11:19.680 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-05-23T00:11:19.680 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:19.680 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:19.680 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:19.680 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:19.680 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:19.681 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:19.681 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:19.681 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:19.681 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:19.681 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:19.681 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:19.681 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:19.681 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:19.682 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:19.682 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:19.682 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:19.682 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:19.682 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:19.682 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:19.682 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:19.682 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:19.683 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-05-23T00:11:19.684 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:19.684 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:19.684 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:19.684 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:19.684 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 211380, 2024-05-23T00:11:19.684 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 626, 2024-05-23T00:11:19.684 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:11:19.684 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-05-23T00:11:19.685 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-05-23T00:11:19.685 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-05-23T00:11:19.685 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:19.685 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:19.685 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:19.685 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:19.685 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9896, 2024-05-23T00:11:19.686 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 190, 2024-05-23T00:11:19.686 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:19.686 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:19.686 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:19.686 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-05-23T00:11:19.686 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 1938555, 2024-05-23T00:11:19.686 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 18995, 2024-05-23T00:11:19.686 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:19.687 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-05-23T00:11:19.687 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:19.687 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-05-23T00:11:19.687 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.687 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-05-23T00:11:19.687 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-05-23T00:11:19.687 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-05-23T00:11:19.687 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 2491, 2024-05-23T00:11:19.688 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 26, 2024-05-23T00:11:19.688 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 4, 2024-05-23T00:11:19.688 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 2491, 2024-05-23T00:11:19.688 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-05-23T00:11:19.688 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2491, 2024-05-23T00:11:19.688 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2.499967496, 2024-05-23T00:11:19.688 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001003599 2024-05-23T00:11:19.688 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.689 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-05-23T00:11:19.689 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-05-23T00:11:19.689 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2491, 2024-05-23T00:11:19.689 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2492 2024-05-23T00:11:19.689 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.689 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 2491, 2024-05-23T00:11:19.689 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 2491, 2024-05-23T00:11:19.689 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-05-23T00:11:19.690 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-05-23T00:11:19.690 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-05-23T00:11:19.690 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 1584, 2024-05-23T00:11:19.690 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-05-23T00:11:19.690 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 904, 2024-05-23T00:11:19.690 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-05-23T00:11:19.690 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-05-23T00:11:19.690 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-05-23T00:11:19.691 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-05-23T00:11:19.691 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-05-23T00:11:19.691 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-05-23T00:11:19.691 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-05-23T00:11:19.691 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-05-23T00:11:19.691 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-05-23T00:11:19.691 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-05-23T00:11:19.691 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 2, 2024-05-23T00:11:19.692 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-05-23T00:11:19.692 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-05-23T00:11:19.692 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-05-23T00:11:19.692 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-05-23T00:11:19.692 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-05-23T00:11:19.692 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-05-23T00:11:19.692 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-05-23T00:11:19.692 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-05-23T00:11:19.693 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-05-23T00:11:19.693 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-05-23T00:11:19.693 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-05-23T00:11:19.693 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-05-23T00:11:19.693 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-05-23T00:11:19.693 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-05-23T00:11:19.693 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-05-23T00:11:19.694 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-05-23T00:11:19.694 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-05-23T00:11:19.694 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-05-23T00:11:19.694 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-05-23T00:11:19.694 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-05-23T00:11:19.694 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-05-23T00:11:19.694 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 106, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 15, 2024-05-23T00:11:19.695 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 5, 2024-05-23T00:11:19.696 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 5, 2024-05-23T00:11:19.696 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-05-23T00:11:19.696 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-05-23T00:11:19.696 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-05-23T00:11:19.696 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-05-23T00:11:19.696 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-05-23T00:11:19.696 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.696 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-05-23T00:11:19.697 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-05-23T00:11:19.697 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-05-23T00:11:19.697 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-05-23T00:11:19.697 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-05-23T00:11:19.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.697 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-05-23T00:11:19.697 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-05-23T00:11:19.697 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.698 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-05-23T00:11:19.698 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-05-23T00:11:19.698 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-05-23T00:11:19.698 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-05-23T00:11:19.698 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-05-23T00:11:19.698 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-05-23T00:11:19.698 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2769, 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 9523657, 2024-05-23T00:11:19.699 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:19.700 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2769, 2024-05-23T00:11:19.700 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:19.700 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:19.700 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2769, 2024-05-23T00:11:19.700 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 9523657, 2024-05-23T00:11:19.700 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:19.700 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.701 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.701 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.701 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.701 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.701 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-05-23T00:11:19.701 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:19.701 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-05-23T00:11:19.701 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2491, 2024-05-23T00:11:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 7277610, 2024-05-23T00:11:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2491, 2024-05-23T00:11:19.702 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 7277610, 2024-05-23T00:11:19.703 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:19.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.703 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.703 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.703 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.703 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.703 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-05-23T00:11:19.703 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2491, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2491, 2024-05-23T00:11:19.704 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2491, 2024-05-23T00:11:19.705 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2491, 2024-05-23T00:11:19.705 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:19.705 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.705 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.705 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.705 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.705 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.705 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-05-23T00:11:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-05-23T00:11:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:19.706 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:19.707 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:19.707 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-05-23T00:11:19.707 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-05-23T00:11:19.707 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:19.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.707 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.707 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.707 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.708 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:19.708 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x55db2a4446e0": { 2024-05-23T00:11:19.708 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:19.708 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-05-23T00:11:19.708 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:19.708 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:19.708 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:19.708 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:19.709 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:19.709 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:19.709 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:19.709 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-05-23T00:11:19.709 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-05-23T00:11:19.709 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:19.709 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:19.709 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:19.710 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:19.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.710 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:19.710 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-05-23T00:11:20.410 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-05-23T00:11:20.411 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-05-23T00:11:20.411 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-05-23T00:11:20.411 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-05-23T00:11:20.411 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-05-23T00:11:20.411 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-05-23T00:11:20.411 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-05-23T00:11:20.411 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-05-23T00:11:20.411 DEBUG:tasks.check_counter:Getting stats from f 2024-05-23T00:11:20.412 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-23T00:11:20.653 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-23T00:11:20.697 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:20 smithi174 ceph-mon[36179]: pgmap v338: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:20.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:20 smithi105 ceph-mon[36426]: pgmap v338: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:20.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:20 smithi103 ceph-mon[31370]: pgmap v338: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:22.366 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-05-23T00:11:22.366 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:22.366 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2324, 2024-05-23T00:11:22.367 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 2325, 2024-05-23T00:11:22.367 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 4769374, 2024-05-23T00:11:22.367 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 738721, 2024-05-23T00:11:22.367 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-05-23T00:11:22.367 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-05-23T00:11:22.368 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.204147432, 2024-05-23T00:11:22.368 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.066516096, 2024-05-23T00:11:22.368 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.052176168, 2024-05-23T00:11:22.368 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.069986569, 2024-05-23T00:11:22.368 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:22.369 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2323, 2024-05-23T00:11:22.369 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.085068759, 2024-05-23T00:11:22.369 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000036620 2024-05-23T00:11:22.369 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.370 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:22.370 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.370 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.370 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.370 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-23T00:11:22.371 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-05-23T00:11:22.371 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.371 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:22.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 276, 2024-05-23T00:11:22.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 250, 2024-05-23T00:11:22.372 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1810876, 2024-05-23T00:11:22.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 144874, 2024-05-23T00:11:22.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 3, 2024-05-23T00:11:22.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 2, 2024-05-23T00:11:22.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.051231020, 2024-05-23T00:11:22.373 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.014997388, 2024-05-23T00:11:22.374 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.040838484, 2024-05-23T00:11:22.374 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.006006547, 2024-05-23T00:11:22.374 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:22.374 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 249, 2024-05-23T00:11:22.375 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.012600779, 2024-05-23T00:11:22.375 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000050605 2024-05-23T00:11:22.375 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.375 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:22.376 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.376 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.376 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.376 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.376 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1810320, 2024-05-23T00:11:22.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 144176 2024-05-23T00:11:22.377 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.377 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:22.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 7, 2024-05-23T00:11:22.377 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 178, 2024-05-23T00:11:22.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 3612, 2024-05-23T00:11:22.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 461022, 2024-05-23T00:11:22.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-05-23T00:11:22.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 4, 2024-05-23T00:11:22.378 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.010718451, 2024-05-23T00:11:22.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.006756718, 2024-05-23T00:11:22.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.002114429, 2024-05-23T00:11:22.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000031062, 2024-05-23T00:11:22.379 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:22.379 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 176, 2024-05-23T00:11:22.380 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.015526727, 2024-05-23T00:11:22.380 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000088220 2024-05-23T00:11:22.380 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.380 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:22.381 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.381 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.381 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.381 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.381 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-23T00:11:22.382 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 460704 2024-05-23T00:11:22.382 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.382 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-05-23T00:11:22.383 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-05-23T00:11:22.383 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-05-23T00:11:22.383 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.383 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-05-23T00:11:22.383 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-05-23T00:11:22.383 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-05-23T00:11:22.383 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3933, 2024-05-23T00:11:22.384 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.108607534, 2024-05-23T00:11:22.384 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000027614 2024-05-23T00:11:22.384 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.384 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.384 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-05-23T00:11:22.384 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-05-23T00:11:22.384 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-05-23T00:11:22.384 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.385 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.385 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.385 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.385 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.385 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-05-23T00:11:22.385 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 0, 2024-05-23T00:11:22.385 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 0, 2024-05-23T00:11:22.385 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-05-23T00:11:22.386 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.386 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.386 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.386 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.386 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-05-23T00:11:22.386 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 0, 2024-05-23T00:11:22.386 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-05-23T00:11:22.387 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-05-23T00:11:22.387 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 0, 2024-05-23T00:11:22.387 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 0, 2024-05-23T00:11:22.387 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 0, 2024-05-23T00:11:22.387 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 5461, 2024-05-23T00:11:22.387 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 1561, 2024-05-23T00:11:22.387 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 3900, 2024-05-23T00:11:22.388 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 0, 2024-05-23T00:11:22.388 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 3231, 2024-05-23T00:11:22.388 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 0, 2024-05-23T00:11:22.388 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 0, 2024-05-23T00:11:22.388 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 0, 2024-05-23T00:11:22.388 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 3, 2024-05-23T00:11:22.388 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 0, 2024-05-23T00:11:22.388 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 0, 2024-05-23T00:11:22.389 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 0, 2024-05-23T00:11:22.389 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 0, 2024-05-23T00:11:22.389 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-05-23T00:11:22.389 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-05-23T00:11:22.389 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 0, 2024-05-23T00:11:22.389 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-05-23T00:11:22.389 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-05-23T00:11:22.390 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 0, 2024-05-23T00:11:22.390 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 0, 2024-05-23T00:11:22.390 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 0, 2024-05-23T00:11:22.390 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 0, 2024-05-23T00:11:22.390 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-05-23T00:11:22.390 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 0, 2024-05-23T00:11:22.390 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-05-23T00:11:22.391 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 0, 2024-05-23T00:11:22.391 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 0, 2024-05-23T00:11:22.391 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 0, 2024-05-23T00:11:22.391 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 0, 2024-05-23T00:11:22.391 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-05-23T00:11:22.391 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-05-23T00:11:22.391 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-05-23T00:11:22.391 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 0, 2024-05-23T00:11:22.392 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-05-23T00:11:22.392 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 0, 2024-05-23T00:11:22.392 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 0, 2024-05-23T00:11:22.392 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 0, 2024-05-23T00:11:22.392 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 0, 2024-05-23T00:11:22.393 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 0, 2024-05-23T00:11:22.393 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 0, 2024-05-23T00:11:22.393 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-23T00:11:22.393 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-23T00:11:22.393 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 0, 2024-05-23T00:11:22.394 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 0, 2024-05-23T00:11:22.394 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 0, 2024-05-23T00:11:22.394 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 0 2024-05-23T00:11:22.394 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.394 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-05-23T00:11:22.395 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 0, 2024-05-23T00:11:22.395 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 0, 2024-05-23T00:11:22.395 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 0, 2024-05-23T00:11:22.395 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 0, 2024-05-23T00:11:22.395 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 0, 2024-05-23T00:11:22.396 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 0, 2024-05-23T00:11:22.396 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-05-23T00:11:22.396 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-05-23T00:11:22.396 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 0, 2024-05-23T00:11:22.396 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 0, 2024-05-23T00:11:22.396 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 0, 2024-05-23T00:11:22.396 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-05-23T00:11:22.397 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-05-23T00:11:22.397 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-05-23T00:11:22.397 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-05-23T00:11:22.397 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-05-23T00:11:22.397 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-05-23T00:11:22.397 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-05-23T00:11:22.397 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-05-23T00:11:22.397 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-05-23T00:11:22.398 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 0, 2024-05-23T00:11:22.398 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-05-23T00:11:22.398 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 0, 2024-05-23T00:11:22.398 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-05-23T00:11:22.398 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-05-23T00:11:22.398 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.399 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-05-23T00:11:22.399 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-05-23T00:11:22.399 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 0, 2024-05-23T00:11:22.399 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-05-23T00:11:22.399 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-05-23T00:11:22.399 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 1108, 2024-05-23T00:11:22.399 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-05-23T00:11:22.399 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-05-23T00:11:22.400 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 0, 2024-05-23T00:11:22.400 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-05-23T00:11:22.400 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-05-23T00:11:22.400 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 3, 2024-05-23T00:11:22.400 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 2, 2024-05-23T00:11:22.400 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-05-23T00:11:22.400 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-05-23T00:11:22.401 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-05-23T00:11:22.401 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 8306893, 2024-05-23T00:11:22.401 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 8306832, 2024-05-23T00:11:22.401 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-05-23T00:11:22.401 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.401 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.401 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.401 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.402 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 1108 2024-05-23T00:11:22.402 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.402 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-05-23T00:11:22.402 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 4362, 2024-05-23T00:11:22.402 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 4452, 2024-05-23T00:11:22.402 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 90, 2024-05-23T00:11:22.403 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 170, 2024-05-23T00:11:22.403 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 170, 2024-05-23T00:11:22.403 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 0, 2024-05-23T00:11:22.403 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 5461, 2024-05-23T00:11:22.403 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 5461, 2024-05-23T00:11:22.403 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 0, 2024-05-23T00:11:22.404 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 0, 2024-05-23T00:11:22.404 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 0, 2024-05-23T00:11:22.404 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 0, 2024-05-23T00:11:22.404 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 67156, 2024-05-23T00:11:22.404 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 215420 2024-05-23T00:11:22.404 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.404 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-05-23T00:11:22.404 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 0, 2024-05-23T00:11:22.405 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 0, 2024-05-23T00:11:22.405 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 0, 2024-05-23T00:11:22.405 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 0, 2024-05-23T00:11:22.405 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 0, 2024-05-23T00:11:22.405 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-05-23T00:11:22.405 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.405 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.405 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.406 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-05-23T00:11:22.406 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.406 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.406 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.406 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.406 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-05-23T00:11:22.406 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.407 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.407 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.407 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.407 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-05-23T00:11:22.407 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.408 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.408 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.408 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.408 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-05-23T00:11:22.408 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.408 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.408 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.409 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-05-23T00:11:22.409 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.409 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.409 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.409 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.409 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-05-23T00:11:22.409 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.410 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.410 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.410 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.410 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-05-23T00:11:22.410 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.410 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.410 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-05-23T00:11:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-05-23T00:11:22.411 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.412 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.412 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.412 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-05-23T00:11:22.412 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.412 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.412 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.412 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.413 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-05-23T00:11:22.413 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.414 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.414 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.414 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.414 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-05-23T00:11:22.414 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.415 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.415 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.415 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.415 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-05-23T00:11:22.415 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.415 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.416 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.416 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-05-23T00:11:22.416 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.416 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.416 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.416 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.416 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-05-23T00:11:22.417 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.417 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.417 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.417 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.417 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-05-23T00:11:22.417 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.417 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.418 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.418 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-05-23T00:11:22.418 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.418 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.418 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.418 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.418 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-05-23T00:11:22.419 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.419 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.419 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.419 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.419 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-05-23T00:11:22.419 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.419 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.419 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.420 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-05-23T00:11:22.420 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.420 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.420 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.420 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.420 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-05-23T00:11:22.420 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.421 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-05-23T00:11:22.422 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.422 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.422 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.422 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.422 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-05-23T00:11:22.422 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.422 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.423 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.423 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.423 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-05-23T00:11:22.423 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.423 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.423 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.424 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-05-23T00:11:22.424 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.424 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.424 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.424 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.425 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-05-23T00:11:22.425 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.425 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.425 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.425 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.425 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-05-23T00:11:22.425 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.426 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.426 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.426 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.426 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-05-23T00:11:22.426 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-05-23T00:11:22.426 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-05-23T00:11:22.426 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.426 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.427 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.427 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.427 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.427 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-05-23T00:11:22.427 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 1, 2024-05-23T00:11:22.427 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 3, 2024-05-23T00:11:22.427 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 2, 2024-05-23T00:11:22.428 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 2, 2024-05-23T00:11:22.428 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-05-23T00:11:22.428 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 0, 2024-05-23T00:11:22.428 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 0, 2024-05-23T00:11:22.428 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 3927, 2024-05-23T00:11:22.428 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-05-23T00:11:22.428 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.428 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-05-23T00:11:22.429 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:11:22.429 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-05-23T00:11:22.429 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:22.429 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:22.429 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:22.429 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:22.429 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:22.430 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:22.430 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:22.430 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:22.430 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:22.430 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:22.430 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:22.431 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:22.431 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:22.431 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:22.431 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:22.431 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:22.431 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:22.431 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:22.431 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:22.432 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:22.432 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:22.432 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:22.432 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:22.432 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:22.432 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:22.432 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:22.432 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:22.433 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:22.433 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:22.433 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-05-23T00:11:22.433 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:22.433 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:22.433 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:22.433 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:22.434 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 343692, 2024-05-23T00:11:22.434 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 4417, 2024-05-23T00:11:22.434 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:11:22.434 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-05-23T00:11:22.434 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-05-23T00:11:22.434 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-05-23T00:11:22.435 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:22.435 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:22.435 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:22.435 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:22.435 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9896, 2024-05-23T00:11:22.435 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 190, 2024-05-23T00:11:22.436 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:22.436 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:22.436 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:22.436 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-05-23T00:11:22.436 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 14077195, 2024-05-23T00:11:22.436 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 118095, 2024-05-23T00:11:22.436 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:22.436 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-05-23T00:11:22.437 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:22.437 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-05-23T00:11:22.437 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.437 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-05-23T00:11:22.437 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-05-23T00:11:22.437 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-05-23T00:11:22.437 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 2326, 2024-05-23T00:11:22.437 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 21, 2024-05-23T00:11:22.438 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 3, 2024-05-23T00:11:22.438 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 2326, 2024-05-23T00:11:22.438 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-05-23T00:11:22.438 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2326, 2024-05-23T00:11:22.438 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 3.045960334, 2024-05-23T00:11:22.438 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001309527 2024-05-23T00:11:22.438 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.438 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-05-23T00:11:22.439 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-05-23T00:11:22.439 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2326, 2024-05-23T00:11:22.439 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 2327 2024-05-23T00:11:22.439 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.439 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 2326, 2024-05-23T00:11:22.439 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 2326, 2024-05-23T00:11:22.439 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 0, 2024-05-23T00:11:22.439 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-05-23T00:11:22.440 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-05-23T00:11:22.440 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 1506, 2024-05-23T00:11:22.440 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 0, 2024-05-23T00:11:22.440 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 818, 2024-05-23T00:11:22.440 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 0, 2024-05-23T00:11:22.440 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 0, 2024-05-23T00:11:22.440 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-05-23T00:11:22.440 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-05-23T00:11:22.441 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-05-23T00:11:22.441 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 0, 2024-05-23T00:11:22.441 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 0, 2024-05-23T00:11:22.441 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-05-23T00:11:22.441 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-05-23T00:11:22.441 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-05-23T00:11:22.441 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 1, 2024-05-23T00:11:22.441 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 0, 2024-05-23T00:11:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-05-23T00:11:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-05-23T00:11:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-05-23T00:11:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-05-23T00:11:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-05-23T00:11:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-05-23T00:11:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-05-23T00:11:22.442 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-05-23T00:11:22.443 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-05-23T00:11:22.443 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 0, 2024-05-23T00:11:22.443 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-05-23T00:11:22.443 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-05-23T00:11:22.443 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-05-23T00:11:22.443 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-05-23T00:11:22.443 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-05-23T00:11:22.444 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-05-23T00:11:22.444 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-05-23T00:11:22.444 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-05-23T00:11:22.444 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-05-23T00:11:22.444 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-05-23T00:11:22.445 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-05-23T00:11:22.445 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-05-23T00:11:22.445 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-05-23T00:11:22.445 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-05-23T00:11:22.445 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-05-23T00:11:22.445 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-05-23T00:11:22.446 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 106, 2024-05-23T00:11:22.446 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-05-23T00:11:22.446 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 15, 2024-05-23T00:11:22.446 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 4, 2024-05-23T00:11:22.446 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 4, 2024-05-23T00:11:22.446 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-05-23T00:11:22.446 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-05-23T00:11:22.446 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 0, 2024-05-23T00:11:22.447 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 2, 2024-05-23T00:11:22.447 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 0 2024-05-23T00:11:22.447 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.447 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-05-23T00:11:22.447 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 0, 2024-05-23T00:11:22.447 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-05-23T00:11:22.447 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-05-23T00:11:22.448 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-05-23T00:11:22.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.448 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-05-23T00:11:22.448 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-05-23T00:11:22.448 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.448 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-05-23T00:11:22.448 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-05-23T00:11:22.448 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 0, 2024-05-23T00:11:22.449 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-05-23T00:11:22.449 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 0, 2024-05-23T00:11:22.449 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 0, 2024-05-23T00:11:22.449 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 0, 2024-05-23T00:11:22.449 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-05-23T00:11:22.449 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.449 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:22.450 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:22.450 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-05-23T00:11:22.450 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:22.450 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 2603, 2024-05-23T00:11:22.450 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 6332584, 2024-05-23T00:11:22.450 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:22.450 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 2603, 2024-05-23T00:11:22.450 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:22.451 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:22.451 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2603, 2024-05-23T00:11:22.451 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 6332584, 2024-05-23T00:11:22.451 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:22.451 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.451 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.451 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.452 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.452 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.452 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-05-23T00:11:22.452 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:22.452 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-05-23T00:11:22.452 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:22.452 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:22.452 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:22.453 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:22.453 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:22.453 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2326, 2024-05-23T00:11:22.453 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 4112594, 2024-05-23T00:11:22.453 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2326, 2024-05-23T00:11:22.453 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 4112594, 2024-05-23T00:11:22.453 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:22.453 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.454 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.454 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.454 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.454 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.454 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-05-23T00:11:22.454 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:22.455 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-05-23T00:11:22.455 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:22.455 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:22.455 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:22.455 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:22.455 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:22.455 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 2326, 2024-05-23T00:11:22.456 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 2326, 2024-05-23T00:11:22.456 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 2326, 2024-05-23T00:11:22.456 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 2326, 2024-05-23T00:11:22.456 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:22.456 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.457 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.457 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.457 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.457 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.458 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-05-23T00:11:22.458 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:22.458 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-05-23T00:11:22.458 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:22.459 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:22.459 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:22.459 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:22.459 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:22.460 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:22.460 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:22.460 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-05-23T00:11:22.460 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-05-23T00:11:22.461 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:22.461 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.461 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.461 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.461 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.462 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:22.462 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x55aca85f66e0": { 2024-05-23T00:11:22.462 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:22.463 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-05-23T00:11:22.463 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:22.463 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:22.463 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:22.464 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:22.464 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:22.464 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:22.464 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:22.465 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 0, 2024-05-23T00:11:22.465 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 0, 2024-05-23T00:11:22.465 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:22.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:22.466 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:22.466 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:22.466 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.467 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:22.467 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-05-23T00:11:22.616 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:22 smithi174 ceph-mon[36179]: pgmap v339: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:22.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:22 smithi105 ceph-mon[36426]: pgmap v339: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:22.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:22 smithi103 ceph-mon[31370]: pgmap v339: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:23.187 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-05-23T00:11:23.188 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-05-23T00:11:23.188 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-05-23T00:11:23.188 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-05-23T00:11:23.188 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-05-23T00:11:23.188 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-05-23T00:11:23.188 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-05-23T00:11:23.188 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-05-23T00:11:23.188 DEBUG:tasks.check_counter:Getting stats from i 2024-05-23T00:11:23.189 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-23T00:11:23.425 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-23T00:11:24.805 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:24 smithi174 ceph-mon[36179]: pgmap v340: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:24.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:24 smithi105 ceph-mon[36426]: pgmap v340: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:24.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:24 smithi103 ceph-mon[31370]: pgmap v340: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-05-23T00:11:25.057 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-05-23T00:11:25.057 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:25.058 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 5120, 2024-05-23T00:11:25.058 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 3921, 2024-05-23T00:11:25.058 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 3132764, 2024-05-23T00:11:25.058 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 11398161, 2024-05-23T00:11:25.058 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 14, 2024-05-23T00:11:25.058 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-05-23T00:11:25.059 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.347122700, 2024-05-23T00:11:25.059 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.126316589, 2024-05-23T00:11:25.059 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.168704574, 2024-05-23T00:11:25.059 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.066536778, 2024-05-23T00:11:25.059 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:25.059 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3920, 2024-05-23T00:11:25.059 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.267716639, 2024-05-23T00:11:25.060 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000068295 2024-05-23T00:11:25.060 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.060 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:25.060 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 4339, 2024-05-23T00:11:25.060 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000545409, 2024-05-23T00:11:25.060 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000125 2024-05-23T00:11:25.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.061 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1825888, 2024-05-23T00:11:25.061 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 171968 2024-05-23T00:11:25.061 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.061 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:25.061 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 3059, 2024-05-23T00:11:25.061 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 3232, 2024-05-23T00:11:25.062 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 783110, 2024-05-23T00:11:25.062 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 30025600, 2024-05-23T00:11:25.062 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 17, 2024-05-23T00:11:25.062 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 5, 2024-05-23T00:11:25.062 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.388671910, 2024-05-23T00:11:25.062 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.205338255, 2024-05-23T00:11:25.063 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.062646221, 2024-05-23T00:11:25.063 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.100005224, 2024-05-23T00:11:25.063 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:25.063 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3229, 2024-05-23T00:11:25.063 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.340774683, 2024-05-23T00:11:25.063 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000105535 2024-05-23T00:11:25.063 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.064 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:25.064 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 13, 2024-05-23T00:11:25.064 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000001853, 2024-05-23T00:11:25.064 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000142 2024-05-23T00:11:25.064 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.064 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 4432, 2024-05-23T00:11:25.065 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 541648 2024-05-23T00:11:25.065 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.065 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:25.065 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 27538, 2024-05-23T00:11:25.065 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 47003, 2024-05-23T00:11:25.065 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 9617248, 2024-05-23T00:11:25.065 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 34435704, 2024-05-23T00:11:25.066 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 10, 2024-05-23T00:11:25.066 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 7, 2024-05-23T00:11:25.066 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 1.679481240, 2024-05-23T00:11:25.066 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.752322635, 2024-05-23T00:11:25.066 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 1.175700181, 2024-05-23T00:11:25.066 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.087065131, 2024-05-23T00:11:25.067 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:25.067 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 47001, 2024-05-23T00:11:25.067 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.517099580, 2024-05-23T00:11:25.067 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000032278 2024-05-23T00:11:25.067 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.067 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:25.067 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 58748, 2024-05-23T00:11:25.068 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.009549920, 2024-05-23T00:11:25.068 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000162 2024-05-23T00:11:25.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.068 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-23T00:11:25.068 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-05-23T00:11:25.068 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.068 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-05-23T00:11:25.069 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 1, 2024-05-23T00:11:25.069 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-05-23T00:11:25.069 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.069 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-MDSRank": { 2024-05-23T00:11:25.069 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-05-23T00:11:25.069 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-05-23T00:11:25.070 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1250, 2024-05-23T00:11:25.070 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 12.326040996, 2024-05-23T00:11:25.070 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.009860832 2024-05-23T00:11:25.070 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.070 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.070 INFO:teuthology.orchestra.run.smithi174.stdout: "finisher-PurgeQueue": { 2024-05-23T00:11:25.070 INFO:teuthology.orchestra.run.smithi174.stdout: "queue_len": 0, 2024-05-23T00:11:25.071 INFO:teuthology.orchestra.run.smithi174.stdout: "complete_latency": { 2024-05-23T00:11:25.071 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5175, 2024-05-23T00:11:25.071 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.302503197, 2024-05-23T00:11:25.071 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000251691 2024-05-23T00:11:25.071 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.071 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.071 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": { 2024-05-23T00:11:25.072 INFO:teuthology.orchestra.run.smithi174.stdout: "request": 11840, 2024-05-23T00:11:25.072 INFO:teuthology.orchestra.run.smithi174.stdout: "reply": 11624, 2024-05-23T00:11:25.072 INFO:teuthology.orchestra.run.smithi174.stdout: "reply_latency": { 2024-05-23T00:11:25.072 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 11624, 2024-05-23T00:11:25.072 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 31.612198351, 2024-05-23T00:11:25.072 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002719562 2024-05-23T00:11:25.072 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.073 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_reply": 0, 2024-05-23T00:11:25.073 INFO:teuthology.orchestra.run.smithi174.stdout: "forward": 216, 2024-05-23T00:11:25.073 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_complete": 0, 2024-05-23T00:11:25.073 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_fetch_keys": 0, 2024-05-23T00:11:25.073 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_commit": 218, 2024-05-23T00:11:25.073 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_split": 25, 2024-05-23T00:11:25.073 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_merge": 1, 2024-05-23T00:11:25.074 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes": 1446, 2024-05-23T00:11:25.074 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_top": 381, 2024-05-23T00:11:25.074 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_bottom": 407, 2024-05-23T00:11:25.074 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pin_tail": 658, 2024-05-23T00:11:25.074 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_pinned": 836, 2024-05-23T00:11:25.074 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_expired": 5672, 2024-05-23T00:11:25.074 INFO:teuthology.orchestra.run.smithi174.stdout: "inodes_with_caps": 123, 2024-05-23T00:11:25.075 INFO:teuthology.orchestra.run.smithi174.stdout: "caps": 124, 2024-05-23T00:11:25.075 INFO:teuthology.orchestra.run.smithi174.stdout: "subtrees": 14, 2024-05-23T00:11:25.075 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse": 13606, 2024-05-23T00:11:25.075 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_hit": 12888, 2024-05-23T00:11:25.075 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_forward": 202, 2024-05-23T00:11:25.075 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_discover": 8, 2024-05-23T00:11:25.076 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_dir_fetch": 0, 2024-05-23T00:11:25.076 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_remote_ino": 0, 2024-05-23T00:11:25.076 INFO:teuthology.orchestra.run.smithi174.stdout: "traverse_lock": 3, 2024-05-23T00:11:25.076 INFO:teuthology.orchestra.run.smithi174.stdout: "load_cent": 0, 2024-05-23T00:11:25.076 INFO:teuthology.orchestra.run.smithi174.stdout: "q": 0, 2024-05-23T00:11:25.076 INFO:teuthology.orchestra.run.smithi174.stdout: "exported": 101, 2024-05-23T00:11:25.076 INFO:teuthology.orchestra.run.smithi174.stdout: "exported_inodes": 5717, 2024-05-23T00:11:25.077 INFO:teuthology.orchestra.run.smithi174.stdout: "imported": 12, 2024-05-23T00:11:25.077 INFO:teuthology.orchestra.run.smithi174.stdout: "imported_inodes": 423, 2024-05-23T00:11:25.077 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_dir_fetch": 0, 2024-05-23T00:11:25.077 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_backtrace_fetch": 3, 2024-05-23T00:11:25.077 INFO:teuthology.orchestra.run.smithi174.stdout: "openino_peer_discover": 0, 2024-05-23T00:11:25.077 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rfiles": 4, 2024-05-23T00:11:25.077 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rbytes": 234, 2024-05-23T00:11:25.078 INFO:teuthology.orchestra.run.smithi174.stdout: "root_rsnaps": 1, 2024-05-23T00:11:25.078 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_fetch": 3, 2024-05-23T00:11:25.078 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_set_tag": 0, 2024-05-23T00:11:25.078 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_backtrace_repaired": 0, 2024-05-23T00:11:25.078 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_inotable_repaired": 0, 2024-05-23T00:11:25.078 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_inodes": 2, 2024-05-23T00:11:25.078 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dir_base_inodes": 0, 2024-05-23T00:11:25.079 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_dirfrag_rstats": 2, 2024-05-23T00:11:25.079 INFO:teuthology.orchestra.run.smithi174.stdout: "scrub_file_inodes": 1, 2024-05-23T00:11:25.079 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_inode_file_caps": 7, 2024-05-23T00:11:25.079 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_revoke": 144, 2024-05-23T00:11:25.079 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_grant": 10469, 2024-05-23T00:11:25.079 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_trunc": 6, 2024-05-23T00:11:25.079 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-23T00:11:25.080 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-23T00:11:25.080 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps": 8742, 2024-05-23T00:11:25.080 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_caps_dirty": 6483, 2024-05-23T00:11:25.080 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_cap_release": 884, 2024-05-23T00:11:25.080 INFO:teuthology.orchestra.run.smithi174.stdout: "process_request_cap_release": 4282 2024-05-23T00:11:25.080 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.080 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_cache": { 2024-05-23T00:11:25.081 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update": 246, 2024-05-23T00:11:25.081 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_update_receipt": 18, 2024-05-23T00:11:25.081 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_try_discover": 12, 2024-05-23T00:11:25.081 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_send_discover": 15, 2024-05-23T00:11:25.081 INFO:teuthology.orchestra.run.smithi174.stdout: "dir_handle_discover": 351, 2024-05-23T00:11:25.081 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays": 1, 2024-05-23T00:11:25.081 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_delayed": 0, 2024-05-23T00:11:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "num_strays_enqueuing": 0, 2024-05-23T00:11:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_created": 3040, 2024-05-23T00:11:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_enqueued": 3036, 2024-05-23T00:11:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_reintegrated": 3, 2024-05-23T00:11:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "strays_migrated": 0, 2024-05-23T00:11:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_processing": 0, 2024-05-23T00:11:25.082 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_enqueued": 0, 2024-05-23T00:11:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "num_recovering_prioritized": 0, 2024-05-23T00:11:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_started": 0, 2024-05-23T00:11:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "recovery_completed": 0, 2024-05-23T00:11:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_path": 0, 2024-05-23T00:11:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_quiesce_inode": 0, 2024-05-23T00:11:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_enqueue_scrub": 0, 2024-05-23T00:11:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_exportdir": 101, 2024-05-23T00:11:25.083 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_flush": 0, 2024-05-23T00:11:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragmentdir": 26, 2024-05-23T00:11:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_fragstats": 0, 2024-05-23T00:11:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "ireq_inodestats": 0 2024-05-23T00:11:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_log": { 2024-05-23T00:11:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "evlrg": 0, 2024-05-23T00:11:25.084 INFO:teuthology.orchestra.run.smithi174.stdout: "evadd": 22485, 2024-05-23T00:11:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "evex": 0, 2024-05-23T00:11:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "evtrm": 0, 2024-05-23T00:11:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "ev": 22485, 2024-05-23T00:11:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "evexg": 0, 2024-05-23T00:11:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "evexd": 0, 2024-05-23T00:11:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "segadd": 28, 2024-05-23T00:11:25.085 INFO:teuthology.orchestra.run.smithi174.stdout: "segex": 0, 2024-05-23T00:11:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "segtrm": 0, 2024-05-23T00:11:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "seg": 28, 2024-05-23T00:11:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "segmjr": 3, 2024-05-23T00:11:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "segexg": 0, 2024-05-23T00:11:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "segexd": 0, 2024-05-23T00:11:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "expos": 4194304, 2024-05-23T00:11:25.086 INFO:teuthology.orchestra.run.smithi174.stdout: "wrpos": 56490353, 2024-05-23T00:11:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "rdpos": 0, 2024-05-23T00:11:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "jlat": { 2024-05-23T00:11:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 584, 2024-05-23T00:11:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 17.123856911, 2024-05-23T00:11:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.029321672 2024-05-23T00:11:25.087 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "replayed": 0 2024-05-23T00:11:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_mem": { 2024-05-23T00:11:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "ino": 441, 2024-05-23T00:11:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "ino+": 7781, 2024-05-23T00:11:25.088 INFO:teuthology.orchestra.run.smithi174.stdout: "ino-": 7340, 2024-05-23T00:11:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "dir": 176, 2024-05-23T00:11:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "dir+": 316, 2024-05-23T00:11:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "dir-": 140, 2024-05-23T00:11:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "dn": 1446, 2024-05-23T00:11:25.089 INFO:teuthology.orchestra.run.smithi174.stdout: "dn+": 9458, 2024-05-23T00:11:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "dn-": 8012, 2024-05-23T00:11:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "cap": 124, 2024-05-23T00:11:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "cap+": 6727, 2024-05-23T00:11:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "cap-": 6603, 2024-05-23T00:11:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "rss": 94892, 2024-05-23T00:11:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: "heap": 223612 2024-05-23T00:11:25.090 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_server": { 2024-05-23T00:11:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_client_request": 12167, 2024-05-23T00:11:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "dispatch_server_request": 503, 2024-05-23T00:11:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_request": 11840, 2024-05-23T00:11:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_client_session": 131, 2024-05-23T00:11:25.091 INFO:teuthology.orchestra.run.smithi174.stdout: "handle_peer_request": 966, 2024-05-23T00:11:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "req_create_latency": { 2024-05-23T00:11:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 6460, 2024-05-23T00:11:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 19.300674102, 2024-05-23T00:11:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002987720 2024-05-23T00:11:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getattr_latency": { 2024-05-23T00:11:25.092 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 108, 2024-05-23T00:11:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 1.735566940, 2024-05-23T00:11:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.016070064 2024-05-23T00:11:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getfilelock_latency": { 2024-05-23T00:11:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.093 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "req_link_latency": { 2024-05-23T00:11:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 3, 2024-05-23T00:11:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.003233995, 2024-05-23T00:11:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.001077998 2024-05-23T00:11:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.094 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookup_latency": { 2024-05-23T00:11:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 1476, 2024-05-23T00:11:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.854670779, 2024-05-23T00:11:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000579045 2024-05-23T00:11:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookuphash_latency": { 2024-05-23T00:11:25.095 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupino_latency": { 2024-05-23T00:11:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2, 2024-05-23T00:11:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000213726, 2024-05-23T00:11:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000106863 2024-05-23T00:11:25.096 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupname_latency": { 2024-05-23T00:11:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupparent_latency": { 2024-05-23T00:11:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.097 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lookupsnap_latency": { 2024-05-23T00:11:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.098 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "req_lssnap_latency": { 2024-05-23T00:11:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.099 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mkdir_latency": { 2024-05-23T00:11:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 75, 2024-05-23T00:11:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.059835704, 2024-05-23T00:11:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000797809 2024-05-23T00:11:25.100 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mknod_latency": { 2024-05-23T00:11:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.101 INFO:teuthology.orchestra.run.smithi174.stdout: "req_mksnap_latency": { 2024-05-23T00:11:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "req_open_latency": { 2024-05-23T00:11:25.102 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 337, 2024-05-23T00:11:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.092576038, 2024-05-23T00:11:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000274706 2024-05-23T00:11:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "req_readdir_latency": { 2024-05-23T00:11:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 76, 2024-05-23T00:11:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.947913716, 2024-05-23T00:11:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.012472548 2024-05-23T00:11:25.103 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rename_latency": { 2024-05-23T00:11:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 63, 2024-05-23T00:11:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.154281406, 2024-05-23T00:11:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002448911 2024-05-23T00:11:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "req_renamesnap_latency": { 2024-05-23T00:11:25.104 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "req_snapdiff_latency": { 2024-05-23T00:11:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.105 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmdir_latency": { 2024-05-23T00:11:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 22, 2024-05-23T00:11:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.506054782, 2024-05-23T00:11:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.023002490 2024-05-23T00:11:25.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmsnap_latency": { 2024-05-23T00:11:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "req_rmxattr_latency": { 2024-05-23T00:11:25.107 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setattr_latency": { 2024-05-23T00:11:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 6, 2024-05-23T00:11:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.044021588, 2024-05-23T00:11:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.007336931 2024-05-23T00:11:25.108 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setdirlayout_latency": { 2024-05-23T00:11:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.109 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setfilelock_latency": { 2024-05-23T00:11:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 8, 2024-05-23T00:11:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.002815725, 2024-05-23T00:11:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000351965 2024-05-23T00:11:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.110 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setlayout_latency": { 2024-05-23T00:11:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "req_setxattr_latency": { 2024-05-23T00:11:25.111 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "req_symlink_latency": { 2024-05-23T00:11:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 5, 2024-05-23T00:11:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.003711961, 2024-05-23T00:11:25.112 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000742392 2024-05-23T00:11:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: "req_unlink_latency": { 2024-05-23T00:11:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 2983, 2024-05-23T00:11:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 7.906627889, 2024-05-23T00:11:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.002650562 2024-05-23T00:11:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.113 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_revoke_eviction": 0, 2024-05-23T00:11:25.114 INFO:teuthology.orchestra.run.smithi174.stdout: "cap_acquisition_throttle": 0, 2024-05-23T00:11:25.114 INFO:teuthology.orchestra.run.smithi174.stdout: "req_getvxattr_latency": { 2024-05-23T00:11:25.114 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.114 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.114 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.114 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.115 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.115 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_sessions": { 2024-05-23T00:11:25.115 INFO:teuthology.orchestra.run.smithi174.stdout: "session_count": 2, 2024-05-23T00:11:25.115 INFO:teuthology.orchestra.run.smithi174.stdout: "session_add": 4, 2024-05-23T00:11:25.115 INFO:teuthology.orchestra.run.smithi174.stdout: "session_remove": 2, 2024-05-23T00:11:25.115 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_open": 2, 2024-05-23T00:11:25.116 INFO:teuthology.orchestra.run.smithi174.stdout: "sessions_stale": 0, 2024-05-23T00:11:25.116 INFO:teuthology.orchestra.run.smithi174.stdout: "total_load": 6462, 2024-05-23T00:11:25.116 INFO:teuthology.orchestra.run.smithi174.stdout: "average_load": 3231, 2024-05-23T00:11:25.116 INFO:teuthology.orchestra.run.smithi174.stdout: "avg_session_uptime": 628, 2024-05-23T00:11:25.117 INFO:teuthology.orchestra.run.smithi174.stdout: "mdthresh_evicted": 0 2024-05-23T00:11:25.117 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.117 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-05-23T00:11:25.117 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 120, 2024-05-23T00:11:25.117 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 120, 2024-05-23T00:11:25.117 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:25.117 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:25.118 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:25.118 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:25.118 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:25.118 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:25.118 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:25.119 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:25.119 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:25.119 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:25.119 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:25.120 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:25.120 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:25.120 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:25.120 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:25.120 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:25.120 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:25.120 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:25.121 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:25.121 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:25.121 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:25.121 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:25.121 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:25.121 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:25.121 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:25.121 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:25.122 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:25.122 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:25.122 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:25.122 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-05-23T00:11:25.122 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:25.122 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:25.123 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:25.123 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:25.123 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 312999, 2024-05-23T00:11:25.123 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 523, 2024-05-23T00:11:25.123 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 880, 2024-05-23T00:11:25.124 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 10, 2024-05-23T00:11:25.124 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-05-23T00:11:25.124 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-05-23T00:11:25.124 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:25.124 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:25.124 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:25.125 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:25.125 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 9896, 2024-05-23T00:11:25.125 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 190, 2024-05-23T00:11:25.125 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:25.125 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:25.125 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:25.126 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-05-23T00:11:25.126 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 2458018, 2024-05-23T00:11:25.126 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 29058, 2024-05-23T00:11:25.126 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:25.126 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-05-23T00:11:25.127 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:25.127 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-05-23T00:11:25.127 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.127 INFO:teuthology.orchestra.run.smithi174.stdout: "objecter": { 2024-05-23T00:11:25.127 INFO:teuthology.orchestra.run.smithi174.stdout: "op_active": 0, 2024-05-23T00:11:25.128 INFO:teuthology.orchestra.run.smithi174.stdout: "op_laggy": 0, 2024-05-23T00:11:25.128 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send": 7202, 2024-05-23T00:11:25.128 INFO:teuthology.orchestra.run.smithi174.stdout: "op_send_bytes": 56241461, 2024-05-23T00:11:25.128 INFO:teuthology.orchestra.run.smithi174.stdout: "op_resend": 33, 2024-05-23T00:11:25.128 INFO:teuthology.orchestra.run.smithi174.stdout: "op_reply": 7202, 2024-05-23T00:11:25.128 INFO:teuthology.orchestra.run.smithi174.stdout: "op_latency": { 2024-05-23T00:11:25.128 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 7202, 2024-05-23T00:11:25.129 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 49.243355736, 2024-05-23T00:11:25.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.006837455 2024-05-23T00:11:25.129 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.129 INFO:teuthology.orchestra.run.smithi174.stdout: "op_inflight": 0, 2024-05-23T00:11:25.129 INFO:teuthology.orchestra.run.smithi174.stdout: "oplen_avg": { 2024-05-23T00:11:25.129 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 7202, 2024-05-23T00:11:25.129 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 7531 2024-05-23T00:11:25.130 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.130 INFO:teuthology.orchestra.run.smithi174.stdout: "op": 7202, 2024-05-23T00:11:25.130 INFO:teuthology.orchestra.run.smithi174.stdout: "op_r": 74, 2024-05-23T00:11:25.130 INFO:teuthology.orchestra.run.smithi174.stdout: "op_w": 7128, 2024-05-23T00:11:25.130 INFO:teuthology.orchestra.run.smithi174.stdout: "op_rmw": 0, 2024-05-23T00:11:25.130 INFO:teuthology.orchestra.run.smithi174.stdout: "op_pg": 0, 2024-05-23T00:11:25.131 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_stat": 0, 2024-05-23T00:11:25.131 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_create": 10, 2024-05-23T00:11:25.131 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_read": 71, 2024-05-23T00:11:25.131 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_write": 655, 2024-05-23T00:11:25.132 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writefull": 49, 2024-05-23T00:11:25.132 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_writesame": 0, 2024-05-23T00:11:25.132 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_append": 0, 2024-05-23T00:11:25.132 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_zero": 0, 2024-05-23T00:11:25.132 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_truncate": 9, 2024-05-23T00:11:25.133 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_delete": 6106, 2024-05-23T00:11:25.133 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_mapext": 0, 2024-05-23T00:11:25.133 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_sparse_read": 0, 2024-05-23T00:11:25.133 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_clonerange": 0, 2024-05-23T00:11:25.133 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_getxattr": 3, 2024-05-23T00:11:25.134 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_setxattr": 20, 2024-05-23T00:11:25.134 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_cmpxattr": 0, 2024-05-23T00:11:25.134 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_rmxattr": 0, 2024-05-23T00:11:25.134 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_resetxattrs": 0, 2024-05-23T00:11:25.134 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_call": 0, 2024-05-23T00:11:25.135 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_watch": 0, 2024-05-23T00:11:25.135 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_notify": 0, 2024-05-23T00:11:25.135 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_src_cmpxattr": 0, 2024-05-23T00:11:25.135 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls": 0, 2024-05-23T00:11:25.135 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_pgls_filter": 0, 2024-05-23T00:11:25.136 INFO:teuthology.orchestra.run.smithi174.stdout: "osdop_other": 6, 2024-05-23T00:11:25.136 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_active": 0, 2024-05-23T00:11:25.136 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_send": 0, 2024-05-23T00:11:25.136 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_resend": 0, 2024-05-23T00:11:25.137 INFO:teuthology.orchestra.run.smithi174.stdout: "linger_ping": 0, 2024-05-23T00:11:25.137 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_active": 0, 2024-05-23T00:11:25.137 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_send": 0, 2024-05-23T00:11:25.137 INFO:teuthology.orchestra.run.smithi174.stdout: "poolop_resend": 0, 2024-05-23T00:11:25.137 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_active": 0, 2024-05-23T00:11:25.137 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_send": 0, 2024-05-23T00:11:25.138 INFO:teuthology.orchestra.run.smithi174.stdout: "poolstat_resend": 0, 2024-05-23T00:11:25.138 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_active": 0, 2024-05-23T00:11:25.138 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_send": 0, 2024-05-23T00:11:25.138 INFO:teuthology.orchestra.run.smithi174.stdout: "statfs_resend": 0, 2024-05-23T00:11:25.138 INFO:teuthology.orchestra.run.smithi174.stdout: "command_active": 0, 2024-05-23T00:11:25.138 INFO:teuthology.orchestra.run.smithi174.stdout: "command_send": 0, 2024-05-23T00:11:25.138 INFO:teuthology.orchestra.run.smithi174.stdout: "command_resend": 0, 2024-05-23T00:11:25.139 INFO:teuthology.orchestra.run.smithi174.stdout: "map_epoch": 106, 2024-05-23T00:11:25.139 INFO:teuthology.orchestra.run.smithi174.stdout: "map_full": 0, 2024-05-23T00:11:25.139 INFO:teuthology.orchestra.run.smithi174.stdout: "map_inc": 15, 2024-05-23T00:11:25.140 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_sessions": 12, 2024-05-23T00:11:25.140 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_open": 12, 2024-05-23T00:11:25.140 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_session_close": 0, 2024-05-23T00:11:25.140 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_laggy": 0, 2024-05-23T00:11:25.140 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_wr": 561, 2024-05-23T00:11:25.140 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_rd": 0, 2024-05-23T00:11:25.140 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_del": 41 2024-05-23T00:11:25.140 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.141 INFO:teuthology.orchestra.run.smithi174.stdout: "oft": { 2024-05-23T00:11:25.141 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_objs": 1, 2024-05-23T00:11:25.141 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_kv_pairs": 0, 2024-05-23T00:11:25.141 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_updates": 0, 2024-05-23T00:11:25.141 INFO:teuthology.orchestra.run.smithi174.stdout: "omap_total_removes": 0 2024-05-23T00:11:25.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.142 INFO:teuthology.orchestra.run.smithi174.stdout: "osd-slow-ops": { 2024-05-23T00:11:25.142 INFO:teuthology.orchestra.run.smithi174.stdout: "slow_ops_count": 0 2024-05-23T00:11:25.142 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.142 INFO:teuthology.orchestra.run.smithi174.stdout: "purge_queue": { 2024-05-23T00:11:25.142 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops": 0, 2024-05-23T00:11:25.142 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_ops_high_water": 17, 2024-05-23T00:11:25.143 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing": 0, 2024-05-23T00:11:25.143 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executing_high_water": 10, 2024-05-23T00:11:25.143 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed_ops": 6058, 2024-05-23T00:11:25.143 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_executed": 3036, 2024-05-23T00:11:25.143 INFO:teuthology.orchestra.run.smithi174.stdout: "pq_item_in_journal": 0 2024-05-23T00:11:25.143 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.143 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:25.143 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:25.144 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-05-23T00:11:25.144 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:25.144 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 35713, 2024-05-23T00:11:25.144 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 10299239, 2024-05-23T00:11:25.144 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:25.144 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 35713, 2024-05-23T00:11:25.144 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:25.145 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:25.145 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 35713, 2024-05-23T00:11:25.145 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 10299239, 2024-05-23T00:11:25.145 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:25.145 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.145 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.145 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.145 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.146 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.146 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_bytes": { 2024-05-23T00:11:25.146 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:25.146 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-05-23T00:11:25.146 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:25.146 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:25.147 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:25.147 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:25.147 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:25.147 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 7202, 2024-05-23T00:11:25.147 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 56572297, 2024-05-23T00:11:25.147 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 7202, 2024-05-23T00:11:25.147 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 56572297, 2024-05-23T00:11:25.147 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-objecter_ops": { 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 1024, 2024-05-23T00:11:25.148 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:25.149 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 0, 2024-05-23T00:11:25.149 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 0, 2024-05-23T00:11:25.149 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:25.149 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 0, 2024-05-23T00:11:25.149 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 7202, 2024-05-23T00:11:25.149 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 7202, 2024-05-23T00:11:25.149 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 7202, 2024-05-23T00:11:25.149 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 7202, 2024-05-23T00:11:25.150 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:25.150 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.150 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.150 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.150 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.150 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.150 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle": { 2024-05-23T00:11:25.150 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:25.151 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-05-23T00:11:25.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:25.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 3036, 2024-05-23T00:11:25.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 330836, 2024-05-23T00:11:25.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:25.151 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 3036, 2024-05-23T00:11:25.151 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:25.151 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:25.152 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 71, 2024-05-23T00:11:25.152 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 330836, 2024-05-23T00:11:25.152 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:25.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.152 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.152 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.153 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.153 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:25.153 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-write_buf_throttle-0x559b32334500": { 2024-05-23T00:11:25.153 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:25.153 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 3758096384, 2024-05-23T00:11:25.153 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:25.153 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 22485, 2024-05-23T00:11:25.153 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 52296049, 2024-05-23T00:11:25.154 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:25.154 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 22485, 2024-05-23T00:11:25.154 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:25.154 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:25.154 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 584, 2024-05-23T00:11:25.154 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 52296049, 2024-05-23T00:11:25.154 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:25.155 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:25.155 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:25.155 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:25.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.155 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:25.155 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-05-23T00:11:25.878 INFO:tasks.check_counter:Daemon mds.i mds.exported=101 2024-05-23T00:11:25.879 INFO:tasks.check_counter:Daemon mds.i mds.imported=12 2024-05-23T00:11:25.879 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=246 2024-05-23T00:11:25.879 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=18 2024-05-23T00:11:25.879 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=1 2024-05-23T00:11:25.879 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-05-23T00:11:25.879 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-05-23T00:11:25.880 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=25 2024-05-23T00:11:25.880 DEBUG:tasks.check_counter:Getting stats from l 2024-05-23T00:11:25.880 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794 shell --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-23T00:11:26.117 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/mon.c/config 2024-05-23T00:11:26.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:26 smithi105 ceph-mon[36426]: pgmap v341: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:11:26.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:26 smithi103 ceph-mon[31370]: pgmap v341: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:11:26.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:26 smithi174 ceph-mon[36179]: pgmap v341: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 90 B/s wr, 8 op/s 2024-05-23T00:11:27.891 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-05-23T00:11:27.891 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-0": { 2024-05-23T00:11:27.891 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 250, 2024-05-23T00:11:27.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 236, 2024-05-23T00:11:27.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 1728048, 2024-05-23T00:11:27.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 144992, 2024-05-23T00:11:27.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 4, 2024-05-23T00:11:27.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 0, 2024-05-23T00:11:27.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.052669636, 2024-05-23T00:11:27.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.015341882, 2024-05-23T00:11:27.892 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.040512476, 2024-05-23T00:11:27.893 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.006040388, 2024-05-23T00:11:27.893 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:27.893 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 236, 2024-05-23T00:11:27.893 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.013959590, 2024-05-23T00:11:27.893 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000059150 2024-05-23T00:11:27.893 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.893 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:27.894 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:27.894 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:27.894 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:27.894 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 1728048, 2024-05-23T00:11:27.894 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 144992 2024-05-23T00:11:27.895 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.895 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-1": { 2024-05-23T00:11:27.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 2, 2024-05-23T00:11:27.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 180, 2024-05-23T00:11:27.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 320, 2024-05-23T00:11:27.895 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 265296, 2024-05-23T00:11:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 6, 2024-05-23T00:11:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 3, 2024-05-23T00:11:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.012720442, 2024-05-23T00:11:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.008585736, 2024-05-23T00:11:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.001604130, 2024-05-23T00:11:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:27.896 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 177, 2024-05-23T00:11:27.897 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.019879809, 2024-05-23T00:11:27.897 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000112315 2024-05-23T00:11:27.897 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.897 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:27.897 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:27.897 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:27.898 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:27.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.898 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-23T00:11:27.898 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 265296 2024-05-23T00:11:27.898 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.898 INFO:teuthology.orchestra.run.smithi174.stdout: "AsyncMessenger::Worker-2": { 2024-05-23T00:11:27.898 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_messages": 0, 2024-05-23T00:11:27.899 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages": 2, 2024-05-23T00:11:27.899 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_bytes": 0, 2024-05-23T00:11:27.899 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_bytes": 0, 2024-05-23T00:11:27.899 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_created_connections": 2, 2024-05-23T00:11:27.899 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_active_connections": 1, 2024-05-23T00:11:27.899 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_total_time": 0.000185154, 2024-05-23T00:11:27.899 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_send_time": 0.000000000, 2024-05-23T00:11:27.899 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_recv_time": 0.000132452, 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_messages_queue_lat": { 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_handle_ack_lat": { 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:27.900 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:27.901 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:27.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.901 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-23T00:11:27.901 INFO:teuthology.orchestra.run.smithi174.stdout: "msgr_send_encrypted_bytes": 0 2024-05-23T00:11:27.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.901 INFO:teuthology.orchestra.run.smithi174.stdout: "cct": { 2024-05-23T00:11:27.901 INFO:teuthology.orchestra.run.smithi174.stdout: "total_workers": 0, 2024-05-23T00:11:27.901 INFO:teuthology.orchestra.run.smithi174.stdout: "unhealthy_workers": 0 2024-05-23T00:11:27.902 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.902 INFO:teuthology.orchestra.run.smithi174.stdout: "mempool": { 2024-05-23T00:11:27.902 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_bytes": 0, 2024-05-23T00:11:27.902 INFO:teuthology.orchestra.run.smithi174.stdout: "bloom_filter_items": 0, 2024-05-23T00:11:27.902 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_bytes": 0, 2024-05-23T00:11:27.902 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_alloc_items": 0, 2024-05-23T00:11:27.902 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_bytes": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_data_items": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_onode_items": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_meta_items": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_bytes": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_other_items": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-23T00:11:27.903 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_cache_buffer_items": 0, 2024-05-23T00:11:27.904 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_bytes": 0, 2024-05-23T00:11:27.904 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_extent_items": 0, 2024-05-23T00:11:27.904 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_bytes": 0, 2024-05-23T00:11:27.904 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_blob_items": 0, 2024-05-23T00:11:27.904 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-23T00:11:27.904 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_shared_blob_items": 0, 2024-05-23T00:11:27.905 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-23T00:11:27.905 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_inline_bl_items": 0, 2024-05-23T00:11:27.905 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_bytes": 0, 2024-05-23T00:11:27.905 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_fsck_items": 0, 2024-05-23T00:11:27.905 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_bytes": 0, 2024-05-23T00:11:27.905 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_txc_items": 0, 2024-05-23T00:11:27.906 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-23T00:11:27.906 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_deferred_items": 0, 2024-05-23T00:11:27.906 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_bytes": 0, 2024-05-23T00:11:27.906 INFO:teuthology.orchestra.run.smithi174.stdout: "bluestore_writing_items": 0, 2024-05-23T00:11:27.906 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_bytes": 0, 2024-05-23T00:11:27.906 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_items": 0, 2024-05-23T00:11:27.906 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_bytes": 0, 2024-05-23T00:11:27.906 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_reader_items": 0, 2024-05-23T00:11:27.907 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_bytes": 0, 2024-05-23T00:11:27.907 INFO:teuthology.orchestra.run.smithi174.stdout: "bluefs_file_writer_items": 0, 2024-05-23T00:11:27.907 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_bytes": 148732, 2024-05-23T00:11:27.907 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_anon_items": 31, 2024-05-23T00:11:27.907 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_bytes": 0, 2024-05-23T00:11:27.907 INFO:teuthology.orchestra.run.smithi174.stdout: "buffer_meta_items": 0, 2024-05-23T00:11:27.907 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_bytes": 0, 2024-05-23T00:11:27.908 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_items": 0, 2024-05-23T00:11:27.908 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_bytes": 0, 2024-05-23T00:11:27.908 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_mapbl_items": 0, 2024-05-23T00:11:27.908 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_bytes": 0, 2024-05-23T00:11:27.908 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_pglog_items": 0, 2024-05-23T00:11:27.908 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_bytes": 0, 2024-05-23T00:11:27.908 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_items": 0, 2024-05-23T00:11:27.909 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_bytes": 0, 2024-05-23T00:11:27.909 INFO:teuthology.orchestra.run.smithi174.stdout: "osdmap_mapping_items": 0, 2024-05-23T00:11:27.909 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_bytes": 0, 2024-05-23T00:11:27.909 INFO:teuthology.orchestra.run.smithi174.stdout: "pgmap_items": 0, 2024-05-23T00:11:27.909 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_bytes": 928, 2024-05-23T00:11:27.910 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_co_items": 2, 2024-05-23T00:11:27.910 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_bytes": 0, 2024-05-23T00:11:27.910 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_1_items": 0, 2024-05-23T00:11:27.910 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_bytes": 0, 2024-05-23T00:11:27.910 INFO:teuthology.orchestra.run.smithi174.stdout: "unittest_2_items": 0 2024-05-23T00:11:27.910 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-05-23T00:11:27.910 INFO:teuthology.orchestra.run.smithi174.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-23T00:11:27.910 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 0, 2024-05-23T00:11:27.911 INFO:teuthology.orchestra.run.smithi174.stdout: "max": 104857600, 2024-05-23T00:11:27.911 INFO:teuthology.orchestra.run.smithi174.stdout: "get_started": 0, 2024-05-23T00:11:27.911 INFO:teuthology.orchestra.run.smithi174.stdout: "get": 248, 2024-05-23T00:11:27.911 INFO:teuthology.orchestra.run.smithi174.stdout: "get_sum": 1690349, 2024-05-23T00:11:27.911 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_fail": 0, 2024-05-23T00:11:27.911 INFO:teuthology.orchestra.run.smithi174.stdout: "get_or_fail_success": 248, 2024-05-23T00:11:27.912 INFO:teuthology.orchestra.run.smithi174.stdout: "take": 0, 2024-05-23T00:11:27.912 INFO:teuthology.orchestra.run.smithi174.stdout: "take_sum": 0, 2024-05-23T00:11:27.912 INFO:teuthology.orchestra.run.smithi174.stdout: "put": 248, 2024-05-23T00:11:27.912 INFO:teuthology.orchestra.run.smithi174.stdout: "put_sum": 1690349, 2024-05-23T00:11:27.912 INFO:teuthology.orchestra.run.smithi174.stdout: "wait": { 2024-05-23T00:11:27.912 INFO:teuthology.orchestra.run.smithi174.stdout: "avgcount": 0, 2024-05-23T00:11:27.912 INFO:teuthology.orchestra.run.smithi174.stdout: "sum": 0.000000000, 2024-05-23T00:11:27.912 INFO:teuthology.orchestra.run.smithi174.stdout: "avgtime": 0.000000000 2024-05-23T00:11:27.913 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:27.913 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-05-23T00:11:27.913 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-05-23T00:11:28.553 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:28 smithi174 ceph-mon[36179]: pgmap v342: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:28.580 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-05-23T00:11:28.580 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-05-23T00:11:28.581 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-05-23T00:11:28.581 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-05-23T00:11:28.581 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-05-23T00:11:28.581 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-05-23T00:11:28.581 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-05-23T00:11:28.581 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-05-23T00:11:28.582 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-23T00:11:28.593 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-23T00:11:28.594 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-23T00:11:28.594 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-23T00:11:28.649 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-23T00:11:28.649 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-23T00:11:28.696 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:11:28.800 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:28 smithi103 ceph-mon[31370]: pgmap v342: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:28.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:28 smithi105 ceph-mon[36426]: pgmap v342: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:28.932 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 -- 172.21.15.103:0/3225155239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20104400 msgr2=0x7f7a20110910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:28.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 --2- 172.21.15.103:0/3225155239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20104400 0x7f7a20110910 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009860 tx=0x7f7a1802a340 comp rx=0 tx=0).stop 2024-05-23T00:11:28.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 -- 172.21.15.103:0/3225155239 shutdown_connections 2024-05-23T00:11:28.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 --2- 172.21.15.103:0/3225155239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20104400 0x7f7a20110910 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:28.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 --2- 172.21.15.103:0/3225155239 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a20103a40 0x7f7a20103ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:28.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 --2- 172.21.15.103:0/3225155239 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20109a50 0x7f7a20109e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:28.933 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 -- 172.21.15.103:0/3225155239 >> 172.21.15.103:0/3225155239 conn(0x7f7a200ff570 msgr2=0x7f7a20101990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:28.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 -- 172.21.15.103:0/3225155239 shutdown_connections 2024-05-23T00:11:28.934 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.929+0000 7f7a27c00640 1 -- 172.21.15.103:0/3225155239 wait complete. 2024-05-23T00:11:28.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.930+0000 7f7a27c00640 1 Processor -- start 2024-05-23T00:11:28.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a27c00640 1 -- start start 2024-05-23T00:11:28.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a27c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20103a40 0x7f7a2019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:28.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a27c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a20104400 0x7f7a2019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:28.936 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a27c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20109a50 0x7f7a2019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:28.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a27c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a20113e60 con 0x7f7a20104400 2024-05-23T00:11:28.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a27c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7a20113ce0 con 0x7f7a20103a40 2024-05-23T00:11:28.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a27c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7a20113fe0 con 0x7f7a20109a50 2024-05-23T00:11:28.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a20104400 0x7f7a2019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:28.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a25c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20109a50 0x7f7a2019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:28.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a25200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20103a40 0x7f7a2019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:28.937 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a25c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20109a50 0x7f7a2019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59274/0 (socket says 172.21.15.103:59274) 2024-05-23T00:11:28.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.931+0000 7f7a25c00640 1 -- 172.21.15.103:0/1104252232 learned_addr learned my addr 172.21.15.103:0/1104252232 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:28.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.932+0000 7f7a25c00640 1 -- 172.21.15.103:0/1104252232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20103a40 msgr2=0x7f7a2019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:28.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.932+0000 7f7a25c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20103a40 0x7f7a2019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:28.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.932+0000 7f7a25c00640 1 -- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a20104400 msgr2=0x7f7a2019a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:28.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.932+0000 7f7a25c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a20104400 0x7f7a2019a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:28.938 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.932+0000 7f7a25c00640 1 -- 172.21.15.103:0/1104252232 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a2019f280 con 0x7f7a20109a50 2024-05-23T00:11:28.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.932+0000 7f7a25200640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20103a40 0x7f7a2019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:28.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.932+0000 7f7a25c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20109a50 0x7f7a2019eb50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7a18030500 tx=0x7f7a18033ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:28.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.933+0000 7f7a1d600640 1 -- 172.21.15.103:0/1104252232 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1803f020 con 0x7f7a20109a50 2024-05-23T00:11:28.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.933+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a2019f510 con 0x7f7a20109a50 2024-05-23T00:11:28.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.933+0000 7f7a1d600640 1 -- 172.21.15.103:0/1104252232 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a18004040 con 0x7f7a20109a50 2024-05-23T00:11:28.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.933+0000 7f7a1d600640 1 -- 172.21.15.103:0/1104252232 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a18037040 con 0x7f7a20109a50 2024-05-23T00:11:28.939 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.933+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a201a6e00 con 0x7f7a20109a50 2024-05-23T00:11:28.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.934+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a20068ae0 con 0x7f7a20109a50 2024-05-23T00:11:28.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.937+0000 7f7a1d600640 1 -- 172.21.15.103:0/1104252232 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f7a18042020 con 0x7f7a20109a50 2024-05-23T00:11:28.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.939+0000 7f7a1d600640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a000a2d00 0x7f7a000a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:28.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.939+0000 7f7a1d600640 1 -- 172.21.15.103:0/1104252232 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7f7a180ef550 con 0x7f7a20109a50 2024-05-23T00:11:28.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.939+0000 7f7a25200640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a000a2d00 0x7f7a000a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:28.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.940+0000 7f7a25200640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a000a2d00 0x7f7a000a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7a100095f0 tx=0x7f7a10005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:28.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:28.940+0000 7f7a1d600640 1 -- 172.21.15.103:0/1104252232 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7a18049030 con 0x7f7a20109a50 2024-05-23T00:11:29.148 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.145+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7a2010bfb0 con 0x7f7a20109a50 2024-05-23T00:11:29.150 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.147+0000 7f7a1d600640 1 -- 172.21.15.103:0/1104252232 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+25507 (secure 0 0 0) 0x7f7a180bbde0 con 0x7f7a20109a50 2024-05-23T00:11:29.151 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:11:29.151 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":106,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-23T00:11:13.636995+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/284924627":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3929139004":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3827978298":"2024-05-24T00:00:32.661117+0000","172.21.15.103:6801/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/2531882208":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:11:29.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a000a2d00 msgr2=0x7f7a000a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a000a2d00 0x7f7a000a51c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7a100095f0 tx=0x7f7a10005f90 comp rx=0 tx=0).stop 2024-05-23T00:11:29.154 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20109a50 msgr2=0x7f7a2019eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20109a50 0x7f7a2019eb50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7a18030500 tx=0x7f7a18033ae0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 shutdown_connections 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a000a2d00 0x7f7a000a51c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a20109a50 0x7f7a2019eb50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a20104400 0x7f7a2019a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 --2- 172.21.15.103:0/1104252232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7a20103a40 0x7f7a2019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 >> 172.21.15.103:0/1104252232 conn(0x7f7a200ff570 msgr2=0x7f7a200ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 shutdown_connections 2024-05-23T00:11:29.155 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.151+0000 7f7a27c00640 1 -- 172.21.15.103:0/1104252232 wait complete. 2024-05-23T00:11:29.175 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-22T23:52:16.669404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-22T23:57:30.907468+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-22T23:57:32.278867+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-22T23:57:36.296428+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-23T00:11:29.175 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-23T00:11:29.366 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 -- 172.21.15.103:0/4289438606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 msgr2=0x7f7aa4103da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 --2- 172.21.15.103:0/4289438606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 0x7f7aa4103da0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c009860 tx=0x7f7a8c02a340 comp rx=0 tx=0).stop 2024-05-23T00:11:29.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 -- 172.21.15.103:0/4289438606 shutdown_connections 2024-05-23T00:11:29.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 --2- 172.21.15.103:0/4289438606 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aa41042e0 0x7f7aa41107f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 --2- 172.21.15.103:0/4289438606 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 0x7f7aa4103da0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 --2- 172.21.15.103:0/4289438606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aa4109930 0x7f7aa4109d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 -- 172.21.15.103:0/4289438606 >> 172.21.15.103:0/4289438606 conn(0x7f7aa40ff3b0 msgr2=0x7f7aa41017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:29.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 -- 172.21.15.103:0/4289438606 shutdown_connections 2024-05-23T00:11:29.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.363+0000 7f7aaa400640 1 -- 172.21.15.103:0/4289438606 wait complete. 2024-05-23T00:11:29.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.364+0000 7f7aaa400640 1 Processor -- start 2024-05-23T00:11:29.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.364+0000 7f7aaa400640 1 -- start start 2024-05-23T00:11:29.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aaa400640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 0x7f7aa419a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:29.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aaa400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aa41042e0 0x7f7aa419a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:29.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aaa400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aa4109930 0x7f7aa419eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:29.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aaa400640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa4113d40 con 0x7f7aa4109930 2024-05-23T00:11:29.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aaa400640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa4113bc0 con 0x7f7aa4103920 2024-05-23T00:11:29.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aaa400640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7aa4113ec0 con 0x7f7aa41042e0 2024-05-23T00:11:29.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa9a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aa4109930 0x7f7aa419eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:29.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 0x7f7aa419a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:29.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa3e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aa41042e0 0x7f7aa419a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:29.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 0x7f7aa419a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48858/0 (socket says 172.21.15.103:48858) 2024-05-23T00:11:29.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa9a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aa4109930 0x7f7aa419eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36160/0 (socket says 172.21.15.103:36160) 2024-05-23T00:11:29.371 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa9000640 1 -- 172.21.15.103:0/1884635106 learned_addr learned my addr 172.21.15.103:0/1884635106 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:29.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa3e00640 1 -- 172.21.15.103:0/1884635106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 msgr2=0x7f7aa419a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa3e00640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 0x7f7aa419a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa3e00640 1 -- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aa4109930 msgr2=0x7f7aa419eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa3e00640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aa4109930 0x7f7aa419eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.365+0000 7f7aa3e00640 1 -- 172.21.15.103:0/1884635106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7aa419f1f0 con 0x7f7aa41042e0 2024-05-23T00:11:29.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.366+0000 7f7aa9000640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 0x7f7aa419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:29.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.366+0000 7f7aa3e00640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aa41042e0 0x7f7aa419a740 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c02a860 tx=0x7f7a8c033b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:29.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.366+0000 7f7aa1600640 1 -- 172.21.15.103:0/1884635106 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c037040 con 0x7f7aa41042e0 2024-05-23T00:11:29.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.366+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7aa419f480 con 0x7f7aa41042e0 2024-05-23T00:11:29.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.366+0000 7f7aa1600640 1 -- 172.21.15.103:0/1884635106 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a8c033d50 con 0x7f7aa41042e0 2024-05-23T00:11:29.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.366+0000 7f7aa1600640 1 -- 172.21.15.103:0/1884635106 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c0038c0 con 0x7f7aa41042e0 2024-05-23T00:11:29.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.366+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7aa41a6d70 con 0x7f7aa41042e0 2024-05-23T00:11:29.373 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.367+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7aa40685b0 con 0x7f7aa41042e0 2024-05-23T00:11:29.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.369+0000 7f7aa1600640 1 -- 172.21.15.103:0/1884635106 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f7a8c042020 con 0x7f7aa41042e0 2024-05-23T00:11:29.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.371+0000 7f7aa1600640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a880a2d00 0x7f7a880a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:29.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.371+0000 7f7aa1600640 1 -- 172.21.15.103:0/1884635106 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7f7a8c0ef2d0 con 0x7f7aa41042e0 2024-05-23T00:11:29.374 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.371+0000 7f7aa9000640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a880a2d00 0x7f7a880a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:29.375 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.372+0000 7f7aa9000640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a880a2d00 0x7f7a880a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a940049d0 tx=0x7f7a94009500 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:29.376 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.373+0000 7f7aa1600640 1 -- 172.21.15.103:0/1884635106 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f7a8c049020 con 0x7f7aa41042e0 2024-05-23T00:11:29.578 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.576+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7aa410be90 con 0x7f7aa41042e0 2024-05-23T00:11:29.579 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.577+0000 7f7aa1600640 1 -- 172.21.15.103:0/1884635106 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) ==== 93+0+10 (secure 0 0 0) 0x7f7a8c0bbb60 con 0x7f7aa41042e0 2024-05-23T00:11:29.579 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 1 2024-05-23T00:11:29.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a880a2d00 msgr2=0x7f7a880a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.583 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a880a2d00 0x7f7a880a51c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7a940049d0 tx=0x7f7a94009500 comp rx=0 tx=0).stop 2024-05-23T00:11:29.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aa41042e0 msgr2=0x7f7aa419a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aa41042e0 0x7f7aa419a740 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c02a860 tx=0x7f7a8c033b60 comp rx=0 tx=0).stop 2024-05-23T00:11:29.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 shutdown_connections 2024-05-23T00:11:29.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f7a880a2d00 0x7f7a880a51c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7aa4109930 0x7f7aa419eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7aa41042e0 0x7f7aa419a740 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.584 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 --2- 172.21.15.103:0/1884635106 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7aa4103920 0x7f7aa419a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 >> 172.21.15.103:0/1884635106 conn(0x7f7aa40ff3b0 msgr2=0x7f7aa40ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:29.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 shutdown_connections 2024-05-23T00:11:29.585 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.580+0000 7f7aaa400640 1 -- 172.21.15.103:0/1884635106 wait complete. 2024-05-23T00:11:29.601 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-23T00:11:29.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 -- 172.21.15.103:0/3448240592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 msgr2=0x7f4af4103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 --2- 172.21.15.103:0/3448240592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 0x7f4af4103ed0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4ae0009930 tx=0x7f4ae002a440 comp rx=0 tx=0).stop 2024-05-23T00:11:29.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 -- 172.21.15.103:0/3448240592 shutdown_connections 2024-05-23T00:11:29.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 --2- 172.21.15.103:0/3448240592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4af4104410 0x7f4af4110920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.803 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 --2- 172.21.15.103:0/3448240592 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 0x7f4af4103ed0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 --2- 172.21.15.103:0/3448240592 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4af4109a60 0x7f4af4109e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 -- 172.21.15.103:0/3448240592 >> 172.21.15.103:0/3448240592 conn(0x7f4af40ff520 msgr2=0x7f4af4101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:29.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 -- 172.21.15.103:0/3448240592 shutdown_connections 2024-05-23T00:11:29.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.800+0000 7f4afaa00640 1 -- 172.21.15.103:0/3448240592 wait complete. 2024-05-23T00:11:29.804 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.801+0000 7f4afaa00640 1 Processor -- start 2024-05-23T00:11:29.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.801+0000 7f4afaa00640 1 -- start start 2024-05-23T00:11:29.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.801+0000 7f4afaa00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 0x7f4af419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:29.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.801+0000 7f4afaa00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4af4104410 0x7f4af419a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:29.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.801+0000 7f4afaa00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4af4109a60 0x7f4af419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:29.806 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.801+0000 7f4afaa00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4113ec0 con 0x7f4af4103a50 2024-05-23T00:11:29.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.801+0000 7f4afaa00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4113d40 con 0x7f4af4109a60 2024-05-23T00:11:29.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.801+0000 7f4afaa00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4af4114040 con 0x7f4af4104410 2024-05-23T00:11:29.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af3e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 0x7f4af419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:29.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4af4109a60 0x7f4af419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:29.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4af4109a60 0x7f4af419ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48872/0 (socket says 172.21.15.103:48872) 2024-05-23T00:11:29.807 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 -- 172.21.15.103:0/552625171 learned_addr learned my addr 172.21.15.103:0/552625171 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af3400640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4af4104410 0x7f4af419a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 -- 172.21.15.103:0/552625171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4af4104410 msgr2=0x7f4af419a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4af4104410 0x7f4af419a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 -- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 msgr2=0x7f4af419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 0x7f4af419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 -- 172.21.15.103:0/552625171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4af419f370 con 0x7f4af4109a60 2024-05-23T00:11:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af3e00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 0x7f4af419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:29.808 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.802+0000 7f4af8a00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4af4109a60 0x7f4af419ec40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00eb10 tx=0x7f4aec00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:29.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.803+0000 7f4af0c00640 1 -- 172.21.15.103:0/552625171 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec004b20 con 0x7f4af4109a60 2024-05-23T00:11:29.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.803+0000 7f4af0c00640 1 -- 172.21.15.103:0/552625171 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4aec004cc0 con 0x7f4af4109a60 2024-05-23T00:11:29.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.803+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4af419f660 con 0x7f4af4109a60 2024-05-23T00:11:29.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.803+0000 7f4af0c00640 1 -- 172.21.15.103:0/552625171 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aec010760 con 0x7f4af4109a60 2024-05-23T00:11:29.809 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.803+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4af41a6ef0 con 0x7f4af4109a60 2024-05-23T00:11:29.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:29 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1104252232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:11:29.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.804+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af4068690 con 0x7f4af4109a60 2024-05-23T00:11:29.810 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.806+0000 7f4af0c00640 1 -- 172.21.15.103:0/552625171 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f4aec005020 con 0x7f4af4109a60 2024-05-23T00:11:29.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.807+0000 7f4af0c00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f4ad80a2d00 0x7f4ad80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:29.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.807+0000 7f4af0c00640 1 -- 172.21.15.103:0/552625171 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7f4aec0cff90 con 0x7f4af4109a60 2024-05-23T00:11:29.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.807+0000 7f4af3e00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f4ad80a2d00 0x7f4ad80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:29.811 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.808+0000 7f4af3e00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f4ad80a2d00 0x7f4ad80a51c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40095f0 tx=0x7f4ae4005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:29.814 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:29.812+0000 7f4af0c00640 1 -- 172.21.15.103:0/552625171 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f4aec09c710 con 0x7f4af4109a60 2024-05-23T00:11:29.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:29 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1104252232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:11:29.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:29 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1104252232' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:11:30.019 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.016+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4af402d200 con 0x7f4af4109a60 2024-05-23T00:11:30.020 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.017+0000 7f4af0c00640 1 -- 172.21.15.103:0/552625171 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) ==== 104+0+11 (secure 0 0 0) 0x7f4aec09c010 con 0x7f4af4109a60 2024-05-23T00:11:30.020 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:11:30.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f4ad80a2d00 msgr2=0x7f4ad80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f4ad80a2d00 0x7f4ad80a51c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40095f0 tx=0x7f4ae4005f90 comp rx=0 tx=0).stop 2024-05-23T00:11:30.023 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4af4109a60 msgr2=0x7f4af419ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4af4109a60 0x7f4af419ec40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00eb10 tx=0x7f4aec00c2d0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 shutdown_connections 2024-05-23T00:11:30.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f4ad80a2d00 0x7f4ad80a51c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4af4109a60 0x7f4af419ec40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4af4104410 0x7f4af419a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 --2- 172.21.15.103:0/552625171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4af4103a50 0x7f4af419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 >> 172.21.15.103:0/552625171 conn(0x7f4af40ff520 msgr2=0x7f4af40ffb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:30.024 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 shutdown_connections 2024-05-23T00:11:30.025 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.020+0000 7f4afaa00640 1 -- 172.21.15.103:0/552625171 wait complete. 2024-05-23T00:11:30.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-23T00:11:30.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 -- 172.21.15.103:0/2456850894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c073ca0 msgr2=0x7f8d6c074080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 --2- 172.21.15.103:0/2456850894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c073ca0 0x7f8d6c074080 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c009900 tx=0x7f8d5c02a480 comp rx=0 tx=0).stop 2024-05-23T00:11:30.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 -- 172.21.15.103:0/2456850894 shutdown_connections 2024-05-23T00:11:30.250 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 --2- 172.21.15.103:0/2456850894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d6c10e060 0x7f8d6c1104b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 --2- 172.21.15.103:0/2456850894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d6c0745c0 0x7f8d6c10db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 --2- 172.21.15.103:0/2456850894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c073ca0 0x7f8d6c074080 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 -- 172.21.15.103:0/2456850894 >> 172.21.15.103:0/2456850894 conn(0x7f8d6c0fd2f0 msgr2=0x7f8d6c0ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:30.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 -- 172.21.15.103:0/2456850894 shutdown_connections 2024-05-23T00:11:30.251 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.247+0000 7f8d71c00640 1 -- 172.21.15.103:0/2456850894 wait complete. 2024-05-23T00:11:30.253 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.248+0000 7f8d71c00640 1 Processor -- start 2024-05-23T00:11:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.248+0000 7f8d71c00640 1 -- start start 2024-05-23T00:11:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d71c00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d6c073ca0 0x7f8d6c19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d71c00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c0745c0 0x7f8d6c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d71c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d6c10e060 0x7f8d6c19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d71c00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c113dc0 con 0x7f8d6c10e060 2024-05-23T00:11:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d71c00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c113c40 con 0x7f8d6c0745c0 2024-05-23T00:11:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d71c00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d6c113f40 con 0x7f8d6c073ca0 2024-05-23T00:11:30.254 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d6aa00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d6c073ca0 0x7f8d6c19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:30.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d6b400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d6c10e060 0x7f8d6c19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:30.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d6aa00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d6c073ca0 0x7f8d6c19a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59318/0 (socket says 172.21.15.103:59318) 2024-05-23T00:11:30.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d6aa00640 1 -- 172.21.15.103:0/25652981 learned_addr learned my addr 172.21.15.103:0/25652981 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:30.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.249+0000 7f8d6a000640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c0745c0 0x7f8d6c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:30.255 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d6aa00640 1 -- 172.21.15.103:0/25652981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c0745c0 msgr2=0x7f8d6c19a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d6aa00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c0745c0 0x7f8d6c19a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d6aa00640 1 -- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d6c10e060 msgr2=0x7f8d6c19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d6aa00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d6c10e060 0x7f8d6c19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d6aa00640 1 -- 172.21.15.103:0/25652981 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d6c19f380 con 0x7f8d6c073ca0 2024-05-23T00:11:30.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d6a000640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c0745c0 0x7f8d6c19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:30.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d6aa00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d6c073ca0 0x7f8d6c19a390 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c006010 tx=0x7f8d5c004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:30.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d63400640 1 -- 172.21.15.103:0/25652981 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c03f030 con 0x7f8d6c073ca0 2024-05-23T00:11:30.256 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d6c19f610 con 0x7f8d6c073ca0 2024-05-23T00:11:30.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d63400640 1 -- 172.21.15.103:0/25652981 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d5c004440 con 0x7f8d6c073ca0 2024-05-23T00:11:30.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.250+0000 7f8d63400640 1 -- 172.21.15.103:0/25652981 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d5c035070 con 0x7f8d6c073ca0 2024-05-23T00:11:30.257 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.251+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d6c1a6f00 con 0x7f8d6c073ca0 2024-05-23T00:11:30.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.252+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d6c067e00 con 0x7f8d6c073ca0 2024-05-23T00:11:30.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.254+0000 7f8d63400640 1 -- 172.21.15.103:0/25652981 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f8d5c042030 con 0x7f8d6c073ca0 2024-05-23T00:11:30.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.255+0000 7f8d63400640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8d4c0a2d80 0x7f8d4c0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:30.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.255+0000 7f8d63400640 1 -- 172.21.15.103:0/25652981 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7f8d5c0efab0 con 0x7f8d6c073ca0 2024-05-23T00:11:30.258 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.255+0000 7f8d6a000640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8d4c0a2d80 0x7f8d4c0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:30.259 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.256+0000 7f8d6a000640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8d4c0a2d80 0x7f8d4c0a5240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c10f470 tx=0x7f8d54005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:30.261 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.259+0000 7f8d63400640 1 -- 172.21.15.103:0/25652981 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8d5c0bc230 con 0x7f8d6c073ca0 2024-05-23T00:11:30.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.463+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8d6c1091e0 con 0x7f8d6c073ca0 2024-05-23T00:11:30.467 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.464+0000 7f8d63400640 1 -- 172.21.15.103:0/25652981 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) ==== 100+0+11 (secure 0 0 0) 0x7f8d5c0bbb30 con 0x7f8d6c073ca0 2024-05-23T00:11:30.467 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 64 2024-05-23T00:11:30.470 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.467+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8d4c0a2d80 msgr2=0x7f8d4c0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.467+0000 7f8d71c00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8d4c0a2d80 0x7f8d4c0a5240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8d6c10f470 tx=0x7f8d54005f90 comp rx=0 tx=0).stop 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.467+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d6c073ca0 msgr2=0x7f8d6c19a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.467+0000 7f8d71c00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d6c073ca0 0x7f8d6c19a390 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c006010 tx=0x7f8d5c004040 comp rx=0 tx=0).stop 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.468+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 shutdown_connections 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.468+0000 7f8d71c00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f8d4c0a2d80 0x7f8d4c0a5240 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.468+0000 7f8d71c00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d6c10e060 0x7f8d6c19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.468+0000 7f8d71c00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d6c0745c0 0x7f8d6c19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.468+0000 7f8d71c00640 1 --2- 172.21.15.103:0/25652981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d6c073ca0 0x7f8d6c19a390 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.471 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.468+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 >> 172.21.15.103:0/25652981 conn(0x7f8d6c0fd2f0 msgr2=0x7f8d6c0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:30.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.468+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 shutdown_connections 2024-05-23T00:11:30.472 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.468+0000 7f8d71c00640 1 -- 172.21.15.103:0/25652981 wait complete. 2024-05-23T00:11:30.489 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-23T00:11:30.698 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.695+0000 7feff3200640 1 -- 172.21.15.103:0/3282266333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff4107ee0 msgr2=0x7feff41082c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.695+0000 7feff3200640 1 --2- 172.21.15.103:0/3282266333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff4107ee0 0x7feff41082c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009860 tx=0x7fefe402a340 comp rx=0 tx=0).stop 2024-05-23T00:11:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.696+0000 7feff3200640 1 -- 172.21.15.103:0/3282266333 shutdown_connections 2024-05-23T00:11:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.696+0000 7feff3200640 1 --2- 172.21.15.103:0/3282266333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feff410cfa0 0x7feff4114b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.696+0000 7feff3200640 1 --2- 172.21.15.103:0/3282266333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feff4108890 0x7feff410c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.699 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.696+0000 7feff3200640 1 --2- 172.21.15.103:0/3282266333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff4107ee0 0x7feff41082c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.696+0000 7feff3200640 1 -- 172.21.15.103:0/3282266333 >> 172.21.15.103:0/3282266333 conn(0x7feff4077480 msgr2=0x7feff40798a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.696+0000 7feff3200640 1 -- 172.21.15.103:0/3282266333 shutdown_connections 2024-05-23T00:11:30.700 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.696+0000 7feff3200640 1 -- 172.21.15.103:0/3282266333 wait complete. 2024-05-23T00:11:30.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.697+0000 7feff3200640 1 Processor -- start 2024-05-23T00:11:30.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.697+0000 7feff3200640 1 -- start start 2024-05-23T00:11:30.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.697+0000 7feff3200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff4107ee0 0x7feff419e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:30.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff3200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feff410cfa0 0x7feff419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:30.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff3200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feff41a2df0 0x7feff41a32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:30.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff3200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7feff4118050 con 0x7feff41a2df0 2024-05-23T00:11:30.702 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff3200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7feff4117ed0 con 0x7feff4107ee0 2024-05-23T00:11:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff3200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feff41181d0 con 0x7feff410cfa0 2024-05-23T00:11:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feff41a2df0 0x7feff41a32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7fefe9400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feff410cfa0 0x7feff419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff2800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feff41a2df0 0x7feff41a32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36200/0 (socket says 172.21.15.103:36200) 2024-05-23T00:11:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff2800640 1 -- 172.21.15.103:0/1213681689 learned_addr learned my addr 172.21.15.103:0/1213681689 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff1e00640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff4107ee0 0x7feff419e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff2800640 1 -- 172.21.15.103:0/1213681689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feff410cfa0 msgr2=0x7feff419ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.703 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff2800640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feff410cfa0 0x7feff419ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff2800640 1 -- 172.21.15.103:0/1213681689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff4107ee0 msgr2=0x7feff419e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff2800640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff4107ee0 0x7feff419e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.698+0000 7feff2800640 1 -- 172.21.15.103:0/1213681689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff41a38e0 con 0x7feff41a2df0 2024-05-23T00:11:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.699+0000 7feff2800640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feff41a2df0 0x7feff41a32a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ef00 tx=0x7fefec00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.699+0000 7fefeb400640 1 -- 172.21.15.103:0/1213681689 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec003970 con 0x7feff41a2df0 2024-05-23T00:11:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.699+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff41a3bd0 con 0x7feff41a2df0 2024-05-23T00:11:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.699+0000 7fefeb400640 1 -- 172.21.15.103:0/1213681689 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefec004020 con 0x7feff41a2df0 2024-05-23T00:11:30.704 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.699+0000 7fefeb400640 1 -- 172.21.15.103:0/1213681689 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec00b910 con 0x7feff41a2df0 2024-05-23T00:11:30.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.699+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff41ab3c0 con 0x7feff41a2df0 2024-05-23T00:11:30.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.700+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff4068a00 con 0x7feff41a2df0 2024-05-23T00:11:30.705 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.702+0000 7fefeb400640 1 -- 172.21.15.103:0/1213681689 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fefec01d020 con 0x7feff41a2df0 2024-05-23T00:11:30.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.704+0000 7fefeb400640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fefd80a2d80 0x7fefd80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:30.707 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.705+0000 7fefeb400640 1 -- 172.21.15.103:0/1213681689 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7fefec0cf780 con 0x7feff41a2df0 2024-05-23T00:11:30.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.705+0000 7feff1e00640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fefd80a2d80 0x7fefd80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:30.708 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.705+0000 7feff1e00640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fefd80a2d80 0x7fefd80a5240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe402a930 tx=0x7fefe40058d0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:30.712 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.709+0000 7fefeb400640 1 -- 172.21.15.103:0/1213681689 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fefec09be60 con 0x7feff41a2df0 2024-05-23T00:11:30.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:30 smithi105 ceph-mon[36426]: pgmap v343: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:11:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:30 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1884635106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:30 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/552625171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:30 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/25652981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:30 smithi103 ceph-mon[31370]: pgmap v343: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:11:30.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:30 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1884635106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:30 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/552625171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:30 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/25652981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:30 smithi174 ceph-mon[36179]: pgmap v343: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-23T00:11:30.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:30 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1884635106' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:30 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/552625171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:30 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/25652981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-23T00:11:30.910 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.908+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7feff4108890 con 0x7feff41a2df0 2024-05-23T00:11:30.912 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.909+0000 7fefeb400640 1 -- 172.21.15.103:0/1213681689 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v106) ==== 103+0+11 (secure 0 0 0) 0x7fefec09b760 con 0x7feff41a2df0 2024-05-23T00:11:30.912 INFO:teuthology.orchestra.run.smithi103.stdout:pg_num: 32 2024-05-23T00:11:30.914 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.911+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fefd80a2d80 msgr2=0x7fefd80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.911+0000 7feff3200640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fefd80a2d80 0x7fefd80a5240 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fefe402a930 tx=0x7fefe40058d0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feff41a2df0 msgr2=0x7feff41a32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:30.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feff41a2df0 0x7feff41a32a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ef00 tx=0x7fefec00c740 comp rx=0 tx=0).stop 2024-05-23T00:11:30.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 shutdown_connections 2024-05-23T00:11:30.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fefd80a2d80 0x7fefd80a5240 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7feff41a2df0 0x7feff41a32a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.915 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feff410cfa0 0x7feff419ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 --2- 172.21.15.103:0/1213681689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff4107ee0 0x7feff419e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:30.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 >> 172.21.15.103:0/1213681689 conn(0x7feff4077480 msgr2=0x7feff40798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:30.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 shutdown_connections 2024-05-23T00:11:30.916 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:30.912+0000 7feff3200640 1 -- 172.21.15.103:0/1213681689 wait complete. 2024-05-23T00:11:30.933 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-23T00:11:31.136 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.133+0000 7ff60ea00640 1 -- 172.21.15.103:0/3197187787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6081126f0 msgr2=0x7ff608114ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:31.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.133+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3197187787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6081126f0 0x7ff608114ae0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff600009c10 tx=0x7ff60002a6f0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.134+0000 7ff60ea00640 1 -- 172.21.15.103:0/3197187787 shutdown_connections 2024-05-23T00:11:31.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.134+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3197187787 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6081126f0 0x7ff608114ae0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.134+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3197187787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6080766c0 0x7ff6081121b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.134+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3197187787 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff608075da0 0x7ff608076180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.137 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.134+0000 7ff60ea00640 1 -- 172.21.15.103:0/3197187787 >> 172.21.15.103:0/3197187787 conn(0x7ff6080ff4e0 msgr2=0x7ff608101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:31.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.134+0000 7ff60ea00640 1 -- 172.21.15.103:0/3197187787 shutdown_connections 2024-05-23T00:11:31.138 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.134+0000 7ff60ea00640 1 -- 172.21.15.103:0/3197187787 wait complete. 2024-05-23T00:11:31.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ea00640 1 Processor -- start 2024-05-23T00:11:31.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ea00640 1 -- start start 2024-05-23T00:11:31.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff608075da0 0x7ff6081a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:31.140 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6080766c0 0x7ff6081a4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:31.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ea00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6081126f0 0x7ff60819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:31.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ea00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6081181c0 con 0x7ff6080766c0 2024-05-23T00:11:31.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ea00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff608118040 con 0x7ff608075da0 2024-05-23T00:11:31.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ea00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff608118340 con 0x7ff6081126f0 2024-05-23T00:11:31.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff607400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6080766c0 0x7ff6081a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:31.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff607e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff608075da0 0x7ff6081a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:31.141 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff60ca00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6081126f0 0x7ff60819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:31.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff607e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff608075da0 0x7ff6081a4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48934/0 (socket says 172.21.15.103:48934) 2024-05-23T00:11:31.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.135+0000 7ff607e00640 1 -- 172.21.15.103:0/3918030472 learned_addr learned my addr 172.21.15.103:0/3918030472 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:31.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.136+0000 7ff607e00640 1 -- 172.21.15.103:0/3918030472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6081126f0 msgr2=0x7ff60819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:31.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.136+0000 7ff607e00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6081126f0 0x7ff60819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.142 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.136+0000 7ff607e00640 1 -- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6080766c0 msgr2=0x7ff6081a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:31.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.136+0000 7ff607e00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6080766c0 0x7ff6081a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.136+0000 7ff607e00640 1 -- 172.21.15.103:0/3918030472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60819edd0 con 0x7ff608075da0 2024-05-23T00:11:31.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.136+0000 7ff607e00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff608075da0 0x7ff6081a4450 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800e9f0 tx=0x7ff5f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:31.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.136+0000 7ff604c00640 1 -- 172.21.15.103:0/3918030472 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f800cd80 con 0x7ff608075da0 2024-05-23T00:11:31.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.137+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60819f0c0 con 0x7ff608075da0 2024-05-23T00:11:31.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.137+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6081ab310 con 0x7ff608075da0 2024-05-23T00:11:31.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.137+0000 7ff604c00640 1 -- 172.21.15.103:0/3918030472 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5f8010910 con 0x7ff608075da0 2024-05-23T00:11:31.143 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.137+0000 7ff604c00640 1 -- 172.21.15.103:0/3918030472 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8018980 con 0x7ff608075da0 2024-05-23T00:11:31.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.138+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e8005290 con 0x7ff608075da0 2024-05-23T00:11:31.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.140+0000 7ff604c00640 1 -- 172.21.15.103:0/3918030472 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7ff5f8010ac0 con 0x7ff608075da0 2024-05-23T00:11:31.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.141+0000 7ff604c00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff5ec0a2d00 0x7ff5ec0a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:31.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.141+0000 7ff604c00640 1 -- 172.21.15.103:0/3918030472 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7ff5f80cfd20 con 0x7ff608075da0 2024-05-23T00:11:31.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.142+0000 7ff607400640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff5ec0a2d00 0x7ff5ec0a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:31.145 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.142+0000 7ff607400640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff5ec0a2d00 0x7ff5ec0a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40096c0 tx=0x7ff5f4009260 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:31.146 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.143+0000 7ff604c00640 1 -- 172.21.15.103:0/3918030472 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff5f809beb0 con 0x7ff608075da0 2024-05-23T00:11:31.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.361+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff5e8005580 con 0x7ff608075da0 2024-05-23T00:11:31.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.362+0000 7ff604c00640 1 -- 172.21.15.103:0/3918030472 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v106) ==== 81+0+960 (secure 0 0 0) 0x7ff5f80a01b0 con 0x7ff608075da0 2024-05-23T00:11:31.365 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/284924627 2024-05-24T00:00:32.661117+0000 2024-05-23T00:11:31.365 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3929139004 2024-05-24T00:00:32.661117+0000 2024-05-23T00:11:31.365 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3827978298 2024-05-24T00:00:32.661117+0000 2024-05-23T00:11:31.365 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/1744216601 2024-05-24T00:00:32.661117+0000 2024-05-23T00:11:31.365 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/3190009536 2024-05-23T23:49:29.953234+0000 2024-05-23T00:11:31.366 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2531882208 2024-05-24T00:00:32.661117+0000 2024-05-23T00:11:31.366 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1448560264 2024-05-23T23:49:29.953234+0000 2024-05-23T00:11:31.366 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/274656845 2024-05-23T23:48:54.926142+0000 2024-05-23T00:11:31.366 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/1744216601 2024-05-24T00:00:32.661117+0000 2024-05-23T00:11:31.366 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2141778982 2024-05-23T23:48:54.926142+0000 2024-05-23T00:11:31.366 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/2592202413 2024-05-23T23:48:54.926142+0000 2024-05-23T00:11:31.366 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/3762323194 2024-05-23T23:49:29.953234+0000 2024-05-23T00:11:31.367 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/3495336128 2024-05-23T23:48:54.926142+0000 2024-05-23T00:11:31.367 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6800/3762323194 2024-05-23T23:49:29.953234+0000 2024-05-23T00:11:31.367 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:6801/3495336128 2024-05-23T23:48:54.926142+0000 2024-05-23T00:11:31.367 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103:0/1425794392 2024-05-23T23:49:29.953234+0000 2024-05-23T00:11:31.368 INFO:teuthology.orchestra.run.smithi103.stderr:listed 16 entries 2024-05-23T00:11:31.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff5ec0a2d00 msgr2=0x7ff5ec0a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:31.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff5ec0a2d00 0x7ff5ec0a51c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40096c0 tx=0x7ff5f4009260 comp rx=0 tx=0).stop 2024-05-23T00:11:31.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff608075da0 msgr2=0x7ff6081a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:31.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff608075da0 0x7ff6081a4450 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800e9f0 tx=0x7ff5f800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 shutdown_connections 2024-05-23T00:11:31.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff5ec0a2d00 0x7ff5ec0a51c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6081126f0 0x7ff60819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6080766c0 0x7ff6081a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 --2- 172.21.15.103:0/3918030472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff608075da0 0x7ff6081a4450 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:31.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 >> 172.21.15.103:0/3918030472 conn(0x7ff6080ff4e0 msgr2=0x7ff6081018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:31.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 shutdown_connections 2024-05-23T00:11:31.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:31.365+0000 7ff60ea00640 1 -- 172.21.15.103:0/3918030472 wait complete. 2024-05-23T00:11:31.387 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-23T00:11:31.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:11:31.387 DEBUG:teuthology.orchestra.run.smithi103:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-23T00:11:31.526 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:11:31.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:11:31.526 DEBUG:teuthology.orchestra.run.smithi103:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-23T00:11:31.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:11:31.594 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-05-23T00:11:31.663 INFO:teuthology.orchestra.run.smithi103.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-23T00:11:31.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:11:31.664 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-23T00:11:31.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:31 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/1213681689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:11:31.718 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:31 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/3918030472' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-23T00:11:31.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:11:31.729 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-05-23T00:11:31.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:31 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/1213681689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:11:31.824 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:31 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/3918030472' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-23T00:11:31.836 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-23T00:11:31.837 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-23T00:11:31.837 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-23T00:11:31.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:31 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/1213681689' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-23T00:11:31.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:31 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/3918030472' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-23T00:11:31.880 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052734,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/df49e73f547dcdc4b3bf4ffb3d49798897278b98e66f874faba555488bbbbef9/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/df49e73f547dcdc4b3bf4ffb3d49798897278b98e66f874faba555488bbbbef9/diff,workdir=/var/lib/containers/storage/overlay/df49e73f547dcdc4b3bf4ffb3d49798897278b98e66f874faba555488bbbbef9/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c16298afa3412001ed74ae97c92f34e455e60843994da2b3bb055af913f0acbe/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/c16298afa3412001ed74ae97c92f34e455e60843994da2b3bb055af913f0acbe/diff,workdir=/var/lib/containers/storage/overlay/c16298afa3412001ed74ae97c92f34e455e60843994da2b3bb055af913f0acbe/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249000k,nr_inodes=812250,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/eb0b14552f4162df842230224c9bb075d516aec593dfe48a20a9a8fcc5675f3f/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/eb0b14552f4162df842230224c9bb075d516aec593dfe48a20a9a8fcc5675f3f/diff,workdir=/var/lib/containers/storage/overlay/eb0b14552f4162df842230224c9bb075d516aec593dfe48a20a9a8fcc5675f3f/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/57498e415745938ffdc754bae2dd4a6d7199bbf80a5aada7fd586adf7b006141/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/57498e415745938ffdc754bae2dd4a6d7199bbf80a5aada7fd586adf7b006141/diff,workdir=/var/lib/containers/storage/overlay/57498e415745938ffdc754bae2dd4a6d7199bbf80a5aada7fd586adf7b006141/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a97f95810007a039a5e2e1e8c1a0d0cd306316a8748ceb697d2ac83def7d34e8/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/a97f95810007a039a5e2e1e8c1a0d0cd306316a8748ceb697d2ac83def7d34e8/diff,workdir=/var/lib/containers/storage/overlay/a97f95810007a039a5e2e1e8c1a0d0cd306316a8748ceb697d2ac83def7d34e8/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/94dd9aa425d40df3f500aaa4fe40647c71699fcf33abe1ad2df376f1433fd97c/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/94dd9aa425d40df3f500aaa4fe40647c71699fcf33abe1ad2df376f1433fd97c/diff,workdir=/var/lib/containers/storage/overlay/94dd9aa425d40df3f500aaa4fe40647c71699fcf33abe1ad2df376f1433fd97c/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/fa63bce37f59009fed33a1c9d9ab131a7716ae70d0dae61aa63d07e6cc4d7541/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/fa63bce37f59009fed33a1c9d9ab131a7716ae70d0dae61aa63d07e6cc4d7541/diff,workdir=/var/lib/containers/storage/overlay/fa63bce37f59009fed33a1c9d9ab131a7716ae70d0dae61aa63d07e6cc4d7541/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/de85e5bff8327540b841416f25e095ca0412b39abcd98306eeddaff9fe70e612/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/de85e5bff8327540b841416f25e095ca0412b39abcd98306eeddaff9fe70e612/diff,workdir=/var/lib/containers/storage/overlay/de85e5bff8327540b841416f25e095ca0412b39abcd98306eeddaff9fe70e612/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/da0d052c753aa2fa6d3e2d35ba4b6eb02eee5d23ebb3e5dfca4e883f20fe215d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/da0d052c753aa2fa6d3e2d35ba4b6eb02eee5d23ebb3e5dfca4e883f20fe215d/diff,workdir=/var/lib/containers/storage/overlay/da0d052c753aa2fa6d3e2d35ba4b6eb02eee5d23ebb3e5dfca4e883f20fe215d/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/af31f68906530943afa0697fe494e127a5d18abe9b7f4d1dbaa7936606a0dc16/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/Y5R6FGOZOYDPKCT6XVHNFON7OG:/var/lib/containers/storage/overlay/l/FA5VRCTARPO6JSZP4TAXJS65YG,upperdir=/var/lib/containers/storage/overlay/af31f68906530943afa0697fe494e127a5d18abe9b7f4d1dbaa7936606a0dc16/diff,workdir=/var/lib/containers/storage/overlay/af31f68906530943afa0697fe494e127a5d18abe9b7f4d1dbaa7936606a0dc16/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498004k,nr_inodes=819200,mode=755 0 0 2024-05-23T00:11:31.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:11:31.880 DEBUG:teuthology.orchestra.run.smithi103:> ip netns list 2024-05-23T00:11:31.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-23T00:11:31.973 DEBUG:teuthology.orchestra.run.smithi103:> sudo ip link delete ceph-brx 2024-05-23T00:11:32.054 INFO:teuthology.orchestra.run.smithi103.stderr:Cannot find device "ceph-brx" 2024-05-23T00:11:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-23T00:11:32.058 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-23T00:11:32.068 INFO:tasks.cephadm:Teardown begin 2024-05-23T00:11:32.069 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-23T00:11:32.069 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:11:32.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 -- 172.21.15.103:0/3832307136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3601086b0 msgr2=0x7fa360114b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:32.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 --2- 172.21.15.103:0/3832307136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3601086b0 0x7fa360114b20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa358009ca0 tx=0x7fa35802a950 comp rx=0 tx=0).stop 2024-05-23T00:11:32.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 -- 172.21.15.103:0/3832307136 shutdown_connections 2024-05-23T00:11:32.273 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 --2- 172.21.15.103:0/3832307136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa3601086b0 0x7fa360114b20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 --2- 172.21.15.103:0/3832307136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa360107d10 0x7fa360108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 --2- 172.21.15.103:0/3832307136 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa36010dd80 0x7fa36010e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 -- 172.21.15.103:0/3832307136 >> 172.21.15.103:0/3832307136 conn(0x7fa3600773f0 msgr2=0x7fa360079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:32.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 -- 172.21.15.103:0/3832307136 shutdown_connections 2024-05-23T00:11:32.274 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.270+0000 7fa366000640 1 -- 172.21.15.103:0/3832307136 wait complete. 2024-05-23T00:11:32.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.271+0000 7fa366000640 1 Processor -- start 2024-05-23T00:11:32.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa366000640 1 -- start start 2024-05-23T00:11:32.276 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa366000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa360107d10 0x7fa36019e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:32.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa366000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3601086b0 0x7fa36019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:32.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa366000640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa36010dd80 0x7fa3601a2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:32.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa366000640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa360118040 con 0x7fa360107d10 2024-05-23T00:11:32.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa366000640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7fa360117ec0 con 0x7fa3601086b0 2024-05-23T00:11:32.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa366000640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa3601181c0 con 0x7fa36010dd80 2024-05-23T00:11:32.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa35fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3601086b0 0x7fa36019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:32.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa365600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa36010dd80 0x7fa3601a2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:32.277 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa364c00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa360107d10 0x7fa36019e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:32.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa35fe00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3601086b0 0x7fa36019e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:48948/0 (socket says 172.21.15.103:48948) 2024-05-23T00:11:32.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa365600640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa36010dd80 0x7fa3601a2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.103:59388/0 (socket says 172.21.15.103:59388) 2024-05-23T00:11:32.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.272+0000 7fa35fe00640 1 -- 172.21.15.103:0/387279688 learned_addr learned my addr 172.21.15.103:0/387279688 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:32.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35fe00640 1 -- 172.21.15.103:0/387279688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa36010dd80 msgr2=0x7fa3601a2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:32.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35fe00640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa36010dd80 0x7fa3601a2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35fe00640 1 -- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa360107d10 msgr2=0x7fa36019e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:32.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35fe00640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa360107d10 0x7fa36019e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.278 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35fe00640 1 -- 172.21.15.103:0/387279688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3601a3470 con 0x7fa3601086b0 2024-05-23T00:11:32.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35fe00640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3601086b0 0x7fa36019e9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00ea60 tx=0x7fa34c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:32.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35d600640 1 -- 172.21.15.103:0/387279688 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c00cde0 con 0x7fa3601086b0 2024-05-23T00:11:32.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35d600640 1 -- 172.21.15.103:0/387279688 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa34c0035f0 con 0x7fa3601086b0 2024-05-23T00:11:32.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3601a3760 con 0x7fa3601086b0 2024-05-23T00:11:32.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa35d600640 1 -- 172.21.15.103:0/387279688 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa34c010670 con 0x7fa3601086b0 2024-05-23T00:11:32.279 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.273+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3601aaff0 con 0x7fa3601086b0 2024-05-23T00:11:32.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.274+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa360068a00 con 0x7fa3601086b0 2024-05-23T00:11:32.280 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.277+0000 7fa35d600640 1 -- 172.21.15.103:0/387279688 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7fa34c0037a0 con 0x7fa3601086b0 2024-05-23T00:11:32.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.278+0000 7fa35d600640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa3480a2d80 0x7fa3480a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:32.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.279+0000 7fa35d600640 1 -- 172.21.15.103:0/387279688 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7fa34c0cfb00 con 0x7fa3601086b0 2024-05-23T00:11:32.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.279+0000 7fa364c00640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa3480a2d80 0x7fa3480a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:32.281 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.279+0000 7fa364c00640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa3480a2d80 0x7fa3480a5240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa354005d00 tx=0x7fa35400b410 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:32.285 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.282+0000 7fa35d600640 1 -- 172.21.15.103:0/387279688 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7fa34c09c2e0 con 0x7fa3601086b0 2024-05-23T00:11:32.532 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.530+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa36019f7d0 con 0x7fa3601086b0 2024-05-23T00:11:32.533 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.531+0000 7fa35d600640 1 -- 172.21.15.103:0/387279688 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11896 (secure 0 0 0) 0x7fa34c09bbe0 con 0x7fa3601086b0 2024-05-23T00:11:32.534 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:11:32.534 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-23T00:06:51.959078+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":71,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24563,24575,24559]},"id":1}]} 2024-05-23T00:11:32.534 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 38 2024-05-23T00:11:32.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.533+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa3480a2d80 msgr2=0x7fa3480a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:32.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.533+0000 7fa366000640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa3480a2d80 0x7fa3480a5240 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa354005d00 tx=0x7fa35400b410 comp rx=0 tx=0).stop 2024-05-23T00:11:32.536 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.533+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3601086b0 msgr2=0x7fa36019e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:32.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.533+0000 7fa366000640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3601086b0 0x7fa36019e9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa34c00ea60 tx=0x7fa34c00ef20 comp rx=0 tx=0).stop 2024-05-23T00:11:32.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.534+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 shutdown_connections 2024-05-23T00:11:32.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.534+0000 7fa366000640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7fa3480a2d80 0x7fa3480a5240 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.534+0000 7fa366000640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa36010dd80 0x7fa3601a2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.534+0000 7fa366000640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3601086b0 0x7fa36019e9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.534+0000 7fa366000640 1 --2- 172.21.15.103:0/387279688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa360107d10 0x7fa36019e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.534+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 >> 172.21.15.103:0/387279688 conn(0x7fa3600773f0 msgr2=0x7fa360078b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:32.537 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.534+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 shutdown_connections 2024-05-23T00:11:32.538 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.534+0000 7fa366000640 1 -- 172.21.15.103:0/387279688 wait complete. 2024-05-23T00:11:32.556 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-23T00:11:32.755 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 -- 172.21.15.103:0/3009612623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fd4107d10 msgr2=0x7f3fd4108170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/3009612623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fd4107d10 0x7f3fd4108170 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009900 tx=0x7f3fbc02a490 comp rx=0 tx=0).stop 2024-05-23T00:11:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 -- 172.21.15.103:0/3009612623 shutdown_connections 2024-05-23T00:11:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/3009612623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fd41086b0 0x7f3fd4114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/3009612623 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fd4107d10 0x7f3fd4108170 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/3009612623 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd410dd80 0x7f3fd410e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 -- 172.21.15.103:0/3009612623 >> 172.21.15.103:0/3009612623 conn(0x7f3fd40773f0 msgr2=0x7f3fd4079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:32.756 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 -- 172.21.15.103:0/3009612623 shutdown_connections 2024-05-23T00:11:32.757 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.753+0000 7f3fd8a00640 1 -- 172.21.15.103:0/3009612623 wait complete. 2024-05-23T00:11:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd8a00640 1 Processor -- start 2024-05-23T00:11:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd8a00640 1 -- start start 2024-05-23T00:11:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd8a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fd4107d10 0x7f3fd419e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd8a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd41086b0 0x7f3fd419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd8a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fd410dd80 0x7f3fd41a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd8a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd41180b0 con 0x7f3fd4107d10 2024-05-23T00:11:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd8a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd4117f30 con 0x7f3fd410dd80 2024-05-23T00:11:32.759 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd8a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd4118230 con 0x7f3fd41086b0 2024-05-23T00:11:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.754+0000 7f3fd3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fd4107d10 0x7f3fd419e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3400640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fd4107d10 0x7f3fd419e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36240/0 (socket says 172.21.15.103:36240) 2024-05-23T00:11:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3400640 1 -- 172.21.15.103:0/2422578123 learned_addr learned my addr 172.21.15.103:0/2422578123 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fcaa00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd41086b0 0x7f3fd419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3e00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fd410dd80 0x7f3fd41a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3400640 1 -- 172.21.15.103:0/2422578123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd41086b0 msgr2=0x7f3fd419ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3400640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd41086b0 0x7f3fd419ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.760 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3400640 1 -- 172.21.15.103:0/2422578123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fd410dd80 msgr2=0x7f3fd41a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:32.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3400640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fd410dd80 0x7f3fd41a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:32.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3400640 1 -- 172.21.15.103:0/2422578123 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd41a34c0 con 0x7f3fd4107d10 2024-05-23T00:11:32.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.755+0000 7f3fd3400640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fd4107d10 0x7f3fd419e4d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400e9f0 tx=0x7f3fc400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:32.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.756+0000 7f3fd0c00640 1 -- 172.21.15.103:0/2422578123 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc400cd80 con 0x7f3fd4107d10 2024-05-23T00:11:32.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.756+0000 7f3fd0c00640 1 -- 172.21.15.103:0/2422578123 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3fc4005500 con 0x7f3fd4107d10 2024-05-23T00:11:32.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.756+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd41a37b0 con 0x7f3fd4107d10 2024-05-23T00:11:32.761 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.756+0000 7f3fd0c00640 1 -- 172.21.15.103:0/2422578123 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4010690 con 0x7f3fd4107d10 2024-05-23T00:11:32.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.756+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd41ab040 con 0x7f3fd4107d10 2024-05-23T00:11:32.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.757+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd4068a00 con 0x7f3fd4107d10 2024-05-23T00:11:32.762 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.759+0000 7f3fd0c00640 1 -- 172.21.15.103:0/2422578123 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f3fc40046c0 con 0x7f3fd4107d10 2024-05-23T00:11:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.760+0000 7f3fd0c00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3fb80a2d00 0x7f3fb80a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.760+0000 7f3fd0c00640 1 -- 172.21.15.103:0/2422578123 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7f3fc40cff00 con 0x7f3fd4107d10 2024-05-23T00:11:32.763 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.761+0000 7f3fcaa00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3fb80a2d00 0x7f3fb80a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:32.764 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.761+0000 7f3fcaa00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3fb80a2d00 0x7f3fb80a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000c00 tx=0x7f3fbc005b50 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:32.765 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:32.763+0000 7f3fd0c00640 1 -- 172.21.15.103:0/2422578123 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f3fc4053850 con 0x7f3fd4107d10 2024-05-23T00:11:32.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:32 smithi105 ceph-mon[36426]: pgmap v344: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:32.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:32 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/387279688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:11:32.828 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:32 smithi103 ceph-mon[31370]: pgmap v344: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:32.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:32 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/387279688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:11:32.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:32 smithi174 ceph-mon[36179]: pgmap v344: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:32.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:32 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/387279688' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:11:33.007 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.005+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3fd419f820 con 0x7f3fd4107d10 2024-05-23T00:11:33.009 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.007+0000 7f3fd0c00640 1 -- 172.21.15.103:0/2422578123 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+11896 (secure 0 0 0) 0x7f3fc409c680 con 0x7f3fd4107d10 2024-05-23T00:11:33.010 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:11:33.010 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14406,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1586986651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1586986651},{"type":"v1","addr":"172.21.15.103:6837","nonce":1586986651}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14454,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/2610863022","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":2610863022},{"type":"v1","addr":"172.21.15.103:6839","nonce":2610863022}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/1828383607","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":1828383607},{"type":"v1","addr":"172.21.15.105:6833","nonce":1828383607}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6833/1973931443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1973931443},{"type":"v1","addr":"172.21.15.174:6833","nonce":1973931443}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/1622172738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":1622172738},{"type":"v1","addr":"172.21.15.103:6835","nonce":1622172738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/1500378287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":1500378287},{"type":"v1","addr":"172.21.15.105:6835","nonce":1500378287}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":38,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-22T23:57:32.804111+0000","modified":"2024-05-23T00:06:51.959078+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24575,"mds_1":24563,"mds_2":24559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6837/2433750399","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2433750399},{"type":"v1","addr":"172.21.15.105:6837","nonce":2433750399}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6835/3839469517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":3839469517},{"type":"v1","addr":"172.21.15.174:6835","nonce":3839469517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.174:6837/2641083628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2641083628},{"type":"v1","addr":"172.21.15.174:6837","nonce":2641083628}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24559":{"gid":24559,"name":"i","rank":2,"incarnation":22,"state":"up:active","state_seq":71,"addr":"172.21.15.174:6839/3021454884","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6838","nonce":3021454884},{"type":"v1","addr":"172.21.15.174:6839","nonce":3021454884}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"a","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6841/4258722678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6840","nonce":4258722678},{"type":"v1","addr":"172.21.15.103:6841","nonce":4258722678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/160741513","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":160741513},{"type":"v1","addr":"172.21.15.105:6839","nonce":160741513}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24563,24575,24559]},"id":1}]} 2024-05-23T00:11:33.010 INFO:teuthology.orchestra.run.smithi103.stderr:dumped fsmap epoch 38 2024-05-23T00:11:33.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.009+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3fb80a2d00 msgr2=0x7f3fb80a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.012 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3fb80a2d00 0x7f3fb80a51c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000c00 tx=0x7f3fbc005b50 comp rx=0 tx=0).stop 2024-05-23T00:11:33.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fd4107d10 msgr2=0x7f3fd419e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fd4107d10 0x7f3fd419e4d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3fc400e9f0 tx=0x7f3fc400edf0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 shutdown_connections 2024-05-23T00:11:33.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f3fb80a2d00 0x7f3fb80a51c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3fd410dd80 0x7f3fd41a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3fd41086b0 0x7f3fd419ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.013 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 --2- 172.21.15.103:0/2422578123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3fd4107d10 0x7f3fd419e4d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 >> 172.21.15.103:0/2422578123 conn(0x7f3fd40773f0 msgr2=0x7f3fd4078ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:33.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 shutdown_connections 2024-05-23T00:11:33.014 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.010+0000 7f3fd8a00640 1 -- 172.21.15.103:0/2422578123 wait complete. 2024-05-23T00:11:33.031 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-23T00:11:33.214 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.211+0000 7f252be00640 1 -- 172.21.15.103:0/2026981896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 msgr2=0x7f252c10e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.211+0000 7f252be00640 1 --2- 172.21.15.103:0/2026981896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 0x7f252c10e160 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f251c009860 tx=0x7f251c02a3d0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.211+0000 7f252be00640 1 -- 172.21.15.103:0/2026981896 shutdown_connections 2024-05-23T00:11:33.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.211+0000 7f252be00640 1 --2- 172.21.15.103:0/2026981896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f252c1086b0 0x7f252c114b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.212+0000 7f252be00640 1 --2- 172.21.15.103:0/2026981896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f252c107d10 0x7f252c108170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.212+0000 7f252be00640 1 --2- 172.21.15.103:0/2026981896 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 0x7f252c10e160 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.212+0000 7f252be00640 1 -- 172.21.15.103:0/2026981896 >> 172.21.15.103:0/2026981896 conn(0x7f252c0773f0 msgr2=0x7f252c079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:33.215 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.212+0000 7f252be00640 1 -- 172.21.15.103:0/2026981896 shutdown_connections 2024-05-23T00:11:33.216 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.212+0000 7f252be00640 1 -- 172.21.15.103:0/2026981896 wait complete. 2024-05-23T00:11:33.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.213+0000 7f252be00640 1 Processor -- start 2024-05-23T00:11:33.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.213+0000 7f252be00640 1 -- start start 2024-05-23T00:11:33.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.213+0000 7f252be00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f252c107d10 0x7f252c19e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:33.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.213+0000 7f252be00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f252c1086b0 0x7f252c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:33.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.213+0000 7f252be00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 0x7f252c1a2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:33.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.213+0000 7f252be00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f252c118090 con 0x7f252c1086b0 2024-05-23T00:11:33.218 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.213+0000 7f252be00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f252c117f10 con 0x7f252c10dd80 2024-05-23T00:11:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.213+0000 7f252be00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f252c118210 con 0x7f252c107d10 2024-05-23T00:11:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f252c1086b0 0x7f252c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252a000640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f252c1086b0 0x7f252c19e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36254/0 (socket says 172.21.15.103:36254) 2024-05-23T00:11:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252a000640 1 -- 172.21.15.103:0/690076870 learned_addr learned my addr 172.21.15.103:0/690076870 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252b400640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 0x7f252c1a2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252aa00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f252c107d10 0x7f252c19e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252a000640 1 -- 172.21.15.103:0/690076870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f252c107d10 msgr2=0x7f252c19e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.219 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252a000640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f252c107d10 0x7f252c19e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252a000640 1 -- 172.21.15.103:0/690076870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 msgr2=0x7f252c1a2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252a000640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 0x7f252c1a2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252a000640 1 -- 172.21.15.103:0/690076870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f252c1a34a0 con 0x7f252c1086b0 2024-05-23T00:11:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.214+0000 7f252b400640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 0x7f252c1a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.215+0000 7f252a000640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f252c1086b0 0x7f252c19e9f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f251800e9f0 tx=0x7f251800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.215+0000 7f2523400640 1 -- 172.21.15.103:0/690076870 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251800cd80 con 0x7f252c1086b0 2024-05-23T00:11:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.215+0000 7f2523400640 1 -- 172.21.15.103:0/690076870 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2518010910 con 0x7f252c1086b0 2024-05-23T00:11:33.220 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.215+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f252c1a3790 con 0x7f252c1086b0 2024-05-23T00:11:33.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.215+0000 7f2523400640 1 -- 172.21.15.103:0/690076870 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2518018990 con 0x7f252c1086b0 2024-05-23T00:11:33.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.215+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f252c1ab020 con 0x7f252c1086b0 2024-05-23T00:11:33.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.216+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f252c068a00 con 0x7f252c1086b0 2024-05-23T00:11:33.221 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.219+0000 7f2523400640 1 -- 172.21.15.103:0/690076870 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f2518010ac0 con 0x7f252c1086b0 2024-05-23T00:11:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.220+0000 7f2523400640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f25100a2d00 0x7f25100a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.221+0000 7f2523400640 1 -- 172.21.15.103:0/690076870 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7f25180cfbe0 con 0x7f252c1086b0 2024-05-23T00:11:33.223 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.221+0000 7f252aa00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f25100a2d00 0x7f25100a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:33.224 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.221+0000 7f252aa00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f25100a2d00 0x7f25100a51c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f251c02a9c0 tx=0x7f251c005c90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:33.228 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.225+0000 7f2523400640 1 -- 172.21.15.103:0/690076870 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f251809bd10 con 0x7f252c1086b0 2024-05-23T00:11:33.440 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.437+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f252c02d200 con 0x7f252c1086b0 2024-05-23T00:11:33.444 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.442+0000 7f2523400640 1 -- 172.21.15.103:0/690076870 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) ==== 74+0+25507 (secure 0 0 0) 0x7f25180a0010 con 0x7f252c1086b0 2024-05-23T00:11:33.445 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-23T00:11:33.445 INFO:teuthology.orchestra.run.smithi103.stdout:{"epoch":106,"fsid":"9c0f3dc8-1895-11ef-bc9a-c7b262605968","created":"2024-05-22T23:48:21.338539+0000","modified":"2024-05-23T00:11:13.636995+0000","last_up_change":"2024-05-22T23:56:34.725743+0000","last_in_change":"2024-05-22T23:56:14.488819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-22T23:52:16.669404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-22T23:57:30.907468+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-22T23:57:32.278867+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-22T23:57:36.296428+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2976753d-f629-42a4-ab40-e6c3bee4d852","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6803","nonce":2549120661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6805","nonce":2549120661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6809","nonce":2549120661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":2549120661},{"type":"v1","addr":"172.21.15.103:6807","nonce":2549120661}]},"public_addr":"172.21.15.103:6803/2549120661","cluster_addr":"172.21.15.103:6805/2549120661","heartbeat_back_addr":"172.21.15.103:6809/2549120661","heartbeat_front_addr":"172.21.15.103:6807/2549120661","state":["exists","up"]},{"osd":1,"uuid":"5c7cafdd-dbef-4f37-8d01-d9fa634a8d1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6801","nonce":732590607}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6803","nonce":732590607}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6807","nonce":732590607}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":732590607},{"type":"v1","addr":"172.21.15.105:6805","nonce":732590607}]},"public_addr":"172.21.15.105:6801/732590607","cluster_addr":"172.21.15.105:6803/732590607","heartbeat_back_addr":"172.21.15.105:6807/732590607","heartbeat_front_addr":"172.21.15.105:6805/732590607","state":["exists","up"]},{"osd":2,"uuid":"63567395-a0a0-48f9-9f29-d65443b5c4ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6801","nonce":3252167839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6803","nonce":3252167839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6807","nonce":3252167839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3252167839},{"type":"v1","addr":"172.21.15.174:6805","nonce":3252167839}]},"public_addr":"172.21.15.174:6801/3252167839","cluster_addr":"172.21.15.174:6803/3252167839","heartbeat_back_addr":"172.21.15.174:6807/3252167839","heartbeat_front_addr":"172.21.15.174:6805/3252167839","state":["exists","up"]},{"osd":3,"uuid":"eea36383-d6a9-49c6-a3cb-9d55f4275529","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6811","nonce":29866944}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6813","nonce":29866944}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6817","nonce":29866944}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":29866944},{"type":"v1","addr":"172.21.15.103:6815","nonce":29866944}]},"public_addr":"172.21.15.103:6811/29866944","cluster_addr":"172.21.15.103:6813/29866944","heartbeat_back_addr":"172.21.15.103:6817/29866944","heartbeat_front_addr":"172.21.15.103:6815/29866944","state":["exists","up"]},{"osd":4,"uuid":"71550539-dfa3-4158-b4b9-4357ae474da2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6809","nonce":4091773686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6811","nonce":4091773686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6815","nonce":4091773686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":4091773686},{"type":"v1","addr":"172.21.15.105:6813","nonce":4091773686}]},"public_addr":"172.21.15.105:6809/4091773686","cluster_addr":"172.21.15.105:6811/4091773686","heartbeat_back_addr":"172.21.15.105:6815/4091773686","heartbeat_front_addr":"172.21.15.105:6813/4091773686","state":["exists","up"]},{"osd":5,"uuid":"d76fc01a-a243-4277-bbdc-1c31965446db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6809","nonce":1001493350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6811","nonce":1001493350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6815","nonce":1001493350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":1001493350},{"type":"v1","addr":"172.21.15.174:6813","nonce":1001493350}]},"public_addr":"172.21.15.174:6809/1001493350","cluster_addr":"172.21.15.174:6811/1001493350","heartbeat_back_addr":"172.21.15.174:6815/1001493350","heartbeat_front_addr":"172.21.15.174:6813/1001493350","state":["exists","up"]},{"osd":6,"uuid":"9494aa47-b660-41dd-af98-e7c55899525c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6819","nonce":1312276577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6821","nonce":1312276577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6825","nonce":1312276577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":1312276577},{"type":"v1","addr":"172.21.15.103:6823","nonce":1312276577}]},"public_addr":"172.21.15.103:6819/1312276577","cluster_addr":"172.21.15.103:6821/1312276577","heartbeat_back_addr":"172.21.15.103:6825/1312276577","heartbeat_front_addr":"172.21.15.103:6823/1312276577","state":["exists","up"]},{"osd":7,"uuid":"8afd33b3-c3ff-41f4-8a04-a1bb4e2bca05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6817","nonce":4084427396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6819","nonce":4084427396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6823","nonce":4084427396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":4084427396},{"type":"v1","addr":"172.21.15.105:6821","nonce":4084427396}]},"public_addr":"172.21.15.105:6817/4084427396","cluster_addr":"172.21.15.105:6819/4084427396","heartbeat_back_addr":"172.21.15.105:6823/4084427396","heartbeat_front_addr":"172.21.15.105:6821/4084427396","state":["exists","up"]},{"osd":8,"uuid":"ad862c3c-61c3-46da-adc1-008a4d5f5fce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":61,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6817","nonce":1749666610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6819","nonce":1749666610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6823","nonce":1749666610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":1749666610},{"type":"v1","addr":"172.21.15.174:6821","nonce":1749666610}]},"public_addr":"172.21.15.174:6817/1749666610","cluster_addr":"172.21.15.174:6819/1749666610","heartbeat_back_addr":"172.21.15.174:6823/1749666610","heartbeat_front_addr":"172.21.15.174:6821/1749666610","state":["exists","up"]},{"osd":9,"uuid":"91405846-87e9-40b3-a158-ffcc2b762c2d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6827","nonce":3886855673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6829","nonce":3886855673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6833","nonce":3886855673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":3886855673},{"type":"v1","addr":"172.21.15.103:6831","nonce":3886855673}]},"public_addr":"172.21.15.103:6827/3886855673","cluster_addr":"172.21.15.103:6829/3886855673","heartbeat_back_addr":"172.21.15.103:6833/3886855673","heartbeat_front_addr":"172.21.15.103:6831/3886855673","state":["exists","up"]},{"osd":10,"uuid":"32c228c2-bec7-4f25-99a9-137e0514381d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":74,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6825","nonce":3074362984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6827","nonce":3074362984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6831","nonce":3074362984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":3074362984},{"type":"v1","addr":"172.21.15.105:6829","nonce":3074362984}]},"public_addr":"172.21.15.105:6825/3074362984","cluster_addr":"172.21.15.105:6827/3074362984","heartbeat_back_addr":"172.21.15.105:6831/3074362984","heartbeat_front_addr":"172.21.15.105:6829/3074362984","state":["exists","up"]},{"osd":11,"uuid":"6c93124e-d53a-4091-bc85-bc3dd328a43c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":82,"up_thru":95,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6825","nonce":1551576781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6827","nonce":1551576781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6831","nonce":1551576781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1551576781},{"type":"v1","addr":"172.21.15.174:6829","nonce":1551576781}]},"public_addr":"172.21.15.174:6825/1551576781","cluster_addr":"172.21.15.174:6827/1551576781","heartbeat_back_addr":"172.21.15.174:6831/1551576781","heartbeat_front_addr":"172.21.15.174:6829/1551576781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:23.287331+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:51:48.366706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:12.600647+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:52:45.134159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:11.273713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:53:36.825003+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:11.708249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:54:39.119616+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:04.484715+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:55:39.106252+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:06.511557+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-22T23:56:32.876392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":7}]},{"pgid":"4.2","mappings":[{"from":9,"to":1}]},{"pgid":"4.6","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.103:0/284924627":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3929139004":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3827978298":"2024-05-24T00:00:32.661117+0000","172.21.15.103:6801/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/3190009536":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/2531882208":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/1448560264":"2024-05-23T23:49:29.953234+0000","172.21.15.103:0/274656845":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/1744216601":"2024-05-24T00:00:32.661117+0000","172.21.15.103:0/2141778982":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/2592202413":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6801/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6800/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:6800/3762323194":"2024-05-23T23:49:29.953234+0000","172.21.15.103:6801/3495336128":"2024-05-23T23:48:54.926142+0000","172.21.15.103:0/1425794392":"2024-05-23T23:49:29.953234+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-23T00:11:33.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f25100a2d00 msgr2=0x7f25100a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f25100a2d00 0x7f25100a51c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f251c02a9c0 tx=0x7f251c005c90 comp rx=0 tx=0).stop 2024-05-23T00:11:33.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f252c1086b0 msgr2=0x7f252c19e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f252c1086b0 0x7f252c19e9f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f251800e9f0 tx=0x7f251800eeb0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 shutdown_connections 2024-05-23T00:11:33.448 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f25100a2d00 0x7f25100a51c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f252c10dd80 0x7f252c1a2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f252c1086b0 0x7f252c19e9f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 --2- 172.21.15.103:0/690076870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f252c107d10 0x7f252c19e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 >> 172.21.15.103:0/690076870 conn(0x7f252c0773f0 msgr2=0x7f252c078c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:33.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.445+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 shutdown_connections 2024-05-23T00:11:33.449 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.446+0000 7f252be00640 1 -- 172.21.15.103:0/690076870 wait complete. 2024-05-23T00:11:33.468 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-23T00:11:33.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 -- 172.21.15.103:0/3037322757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff738107e10 msgr2=0x7ff738108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 --2- 172.21.15.103:0/3037322757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff738107e10 0x7ff738108270 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff724009900 tx=0x7ff72402a480 comp rx=0 tx=0).stop 2024-05-23T00:11:33.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 -- 172.21.15.103:0/3037322757 shutdown_connections 2024-05-23T00:11:33.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 --2- 172.21.15.103:0/3037322757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7381087b0 0x7ff738114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 --2- 172.21.15.103:0/3037322757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff738107e10 0x7ff738108270 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.673 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 --2- 172.21.15.103:0/3037322757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff73810de30 0x7ff73810e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 -- 172.21.15.103:0/3037322757 >> 172.21.15.103:0/3037322757 conn(0x7ff738077690 msgr2=0x7ff738106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:33.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 -- 172.21.15.103:0/3037322757 shutdown_connections 2024-05-23T00:11:33.674 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.670+0000 7ff73e800640 1 -- 172.21.15.103:0/3037322757 wait complete. 2024-05-23T00:11:33.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.671+0000 7ff73e800640 1 Processor -- start 2024-05-23T00:11:33.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.671+0000 7ff73e800640 1 -- start start 2024-05-23T00:11:33.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.671+0000 7ff73e800640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff738107e10 0x7ff73819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:33.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.671+0000 7ff73e800640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7381087b0 0x7ff73819eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:33.676 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.671+0000 7ff73e800640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73810de30 0x7ff7381a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:33.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73e800640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff738118160 con 0x7ff7381087b0 2024-05-23T00:11:33.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73e800640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7ff738117fe0 con 0x7ff73810de30 2024-05-23T00:11:33.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73e800640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff7381182e0 con 0x7ff738107e10 2024-05-23T00:11:33.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7381087b0 0x7ff73819eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:33.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73d400640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff738107e10 0x7ff73819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:33.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73ca00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7381087b0 0x7ff73819eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:36268/0 (socket says 172.21.15.103:36268) 2024-05-23T00:11:33.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73ca00640 1 -- 172.21.15.103:0/2392007873 learned_addr learned my addr 172.21.15.103:0/2392007873 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:33.677 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73de00640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73810de30 0x7ff7381a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:33.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73d400640 1 -- 172.21.15.103:0/2392007873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73810de30 msgr2=0x7ff7381a2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73d400640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73810de30 0x7ff7381a2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73d400640 1 -- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7381087b0 msgr2=0x7ff73819eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73d400640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7381087b0 0x7ff73819eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73d400640 1 -- 172.21.15.103:0/2392007873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7381a3560 con 0x7ff738107e10 2024-05-23T00:11:33.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.672+0000 7ff73de00640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73810de30 0x7ff7381a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:33.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.673+0000 7ff73d400640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff738107e10 0x7ff73819e5c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00e9f0 tx=0x7ff72c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:33.678 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.673+0000 7ff736000640 1 -- 172.21.15.103:0/2392007873 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c00ccc0 con 0x7ff738107e10 2024-05-23T00:11:33.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.673+0000 7ff736000640 1 -- 172.21.15.103:0/2392007873 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff72c005020 con 0x7ff738107e10 2024-05-23T00:11:33.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.673+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7381a3850 con 0x7ff738107e10 2024-05-23T00:11:33.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.673+0000 7ff736000640 1 -- 172.21.15.103:0/2392007873 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff72c0049e0 con 0x7ff738107e10 2024-05-23T00:11:33.679 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.673+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7381ab0e0 con 0x7ff738107e10 2024-05-23T00:11:33.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.674+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff738068a00 con 0x7ff738107e10 2024-05-23T00:11:33.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.676+0000 7ff736000640 1 -- 172.21.15.103:0/2392007873 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7ff72c005560 con 0x7ff738107e10 2024-05-23T00:11:33.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.677+0000 7ff736000640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff7200a2d80 0x7ff7200a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:33.680 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.678+0000 7ff736000640 1 -- 172.21.15.103:0/2392007873 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 9612+0+0 (secure 0 0 0) 0x7ff72c0cf7e0 con 0x7ff738107e10 2024-05-23T00:11:33.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.678+0000 7ff73ca00640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff7200a2d80 0x7ff7200a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:33.681 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.678+0000 7ff73ca00640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff7200a2d80 0x7ff7200a5240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff7240096c0 tx=0x7ff7240046c0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:33.684 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.682+0000 7ff736000640 1 -- 172.21.15.103:0/2392007873 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7ff72c0051d0 con 0x7ff738107e10 2024-05-23T00:11:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:33 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2422578123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:11:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:33 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/690076870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:11:33.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:33 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2422578123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:11:33.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:33 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/690076870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:11:33.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:33 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2422578123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-23T00:11:33.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:33 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/690076870' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-23T00:11:33.928 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.925+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff738110220 con 0x7ff738107e10 2024-05-23T00:11:33.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.936+0000 7ff736000640 1 -- 172.21.15.103:0/2392007873 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v39) ==== 161+0+0 (secure 0 0 0) 0x7ff72c09bf60 con 0x7ff738107e10 2024-05-23T00:11:33.940 INFO:teuthology.orchestra.run.smithi103.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-23T00:11:33.941 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff7200a2d80 msgr2=0x7ff7200a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff7200a2d80 0x7ff7200a5240 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff7240096c0 tx=0x7ff7240046c0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff738107e10 msgr2=0x7ff73819e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:33.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff738107e10 0x7ff73819e5c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff72c00e9f0 tx=0x7ff72c00eeb0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 shutdown_connections 2024-05-23T00:11:33.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7ff7200a2d80 0x7ff7200a5240 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.942 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff73810de30 0x7ff7381a2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff7381087b0 0x7ff73819eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 --2- 172.21.15.103:0/2392007873 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff738107e10 0x7ff73819e5c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:33.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 >> 172.21.15.103:0/2392007873 conn(0x7ff738077690 msgr2=0x7ff738078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:33.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 shutdown_connections 2024-05-23T00:11:33.943 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:33.939+0000 7ff73e800640 1 -- 172.21.15.103:0/2392007873 wait complete. 2024-05-23T00:11:33.962 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-23T00:11:34.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 -- 172.21.15.103:0/1798327550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4102970 msgr2=0x7f6cb4102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:34.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 --2- 172.21.15.103:0/1798327550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4102970 0x7f6cb4102d50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4009ca0 tx=0x7f6ca402a940 comp rx=0 tx=0).stop 2024-05-23T00:11:34.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 -- 172.21.15.103:0/1798327550 shutdown_connections 2024-05-23T00:11:34.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 --2- 172.21.15.103:0/1798327550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cb410be90 0x7f6cb410e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 --2- 172.21.15.103:0/1798327550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb4103290 0x7f6cb410b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 --2- 172.21.15.103:0/1798327550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4102970 0x7f6cb4102d50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.157 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 -- 172.21.15.103:0/1798327550 >> 172.21.15.103:0/1798327550 conn(0x7f6cb40778d0 msgr2=0x7f6cb4077ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:34.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 -- 172.21.15.103:0/1798327550 shutdown_connections 2024-05-23T00:11:34.158 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.154+0000 7f6cbba00640 1 -- 172.21.15.103:0/1798327550 wait complete. 2024-05-23T00:11:34.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.155+0000 7f6cbba00640 1 Processor -- start 2024-05-23T00:11:34.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.155+0000 7f6cbba00640 1 -- start start 2024-05-23T00:11:34.161 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cbba00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cb4102970 0x7f6cb419a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:34.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cbba00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4103290 0x7f6cb419a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:34.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cbba00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb410be90 0x7f6cb419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:34.162 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cbba00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb4111c80 con 0x7f6cb410be90 2024-05-23T00:11:34.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cbba00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb4111b00 con 0x7f6cb4102970 2024-05-23T00:11:34.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cbba00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb4111e00 con 0x7f6cb4103290 2024-05-23T00:11:34.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cb4102970 0x7f6cb419a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:34.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cabe00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4103290 0x7f6cb419a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:34.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb410be90 0x7f6cb419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:34.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9000640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cb4102970 0x7f6cb419a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.103:51302/0 (socket says 172.21.15.103:51302) 2024-05-23T00:11:34.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9000640 1 -- 172.21.15.103:0/2831981351 learned_addr learned my addr 172.21.15.103:0/2831981351 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:34.163 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9000640 1 -- 172.21.15.103:0/2831981351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4103290 msgr2=0x7f6cb419a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:34.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9000640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4103290 0x7f6cb419a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9000640 1 -- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb410be90 msgr2=0x7f6cb419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:34.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9000640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb410be90 0x7f6cb419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cb9000640 1 -- 172.21.15.103:0/2831981351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb419f310 con 0x7f6cb4102970 2024-05-23T00:11:34.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.156+0000 7f6cabe00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4103290 0x7f6cb419a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:34.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.157+0000 7f6cb9000640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cb4102970 0x7f6cb419a320 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4004af0 tx=0x7f6ca402f6b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:34.164 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.157+0000 7f6cb2000640 1 -- 172.21.15.103:0/2831981351 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4034040 con 0x7f6cb4102970 2024-05-23T00:11:34.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.157+0000 7f6cb2000640 1 -- 172.21.15.103:0/2831981351 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ca402fe10 con 0x7f6cb4102970 2024-05-23T00:11:34.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.157+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb419f5a0 con 0x7f6cb4102970 2024-05-23T00:11:34.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.157+0000 7f6cb2000640 1 -- 172.21.15.103:0/2831981351 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca40046e0 con 0x7f6cb4102970 2024-05-23T00:11:34.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.157+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb41a6e90 con 0x7f6cb4102970 2024-05-23T00:11:34.165 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.158+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb4067e00 con 0x7f6cb4102970 2024-05-23T00:11:34.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.160+0000 7f6cb2000640 1 -- 172.21.15.103:0/2831981351 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f6ca4037020 con 0x7f6cb4102970 2024-05-23T00:11:34.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.162+0000 7f6cb2000640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6c980a2d00 0x7f6c980a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:34.166 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.162+0000 7f6cb2000640 1 -- 172.21.15.103:0/2831981351 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9782+0+0 (secure 0 0 0) 0x7f6ca40f0770 con 0x7f6cb4102970 2024-05-23T00:11:34.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.162+0000 7f6cabe00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6c980a2d00 0x7f6c980a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:34.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.163+0000 7f6cabe00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6c980a2d00 0x7f6c980a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6cb410d9f0 tx=0x7f6ca0005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:34.167 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.164+0000 7f6cb2000640 1 -- 172.21.15.103:0/2831981351 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6ca40bcf50 con 0x7f6cb4102970 2024-05-23T00:11:34.307 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:11:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b[66976]: ignoring --setuser ceph since I am not root 2024-05-23T00:11:34.307 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:11:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b[66976]: ignoring --setgroup ceph since I am not root 2024-05-23T00:11:34.308 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:11:33 smithi105 ceph-mds[66980]: -- 172.21.15.105:0/3180094361 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b1995e7a40 con 0x55b1995ba800 2024-05-23T00:11:34.308 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:11:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b[66976]: starting mds.b at 2024-05-23T00:11:34.308 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:11:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h[66663]: ignoring --setuser ceph since I am not root 2024-05-23T00:11:34.308 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:11:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h[66663]: ignoring --setgroup ceph since I am not root 2024-05-23T00:11:34.308 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:11:33 smithi105 ceph-mds[66667]: -- 172.21.15.105:0/3539354257 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55641bb97a40 con 0x55641bb6b000 2024-05-23T00:11:34.309 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:11:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h[66663]: starting mds.h at 2024-05-23T00:11:34.328 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[31366]: 2024-05-23T00:11:33.928+0000 7fe442a00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-23T00:11:34.329 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a[73394]: ignoring --setuser ceph since I am not root 2024-05-23T00:11:34.329 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a[73394]: ignoring --setgroup ceph since I am not root 2024-05-23T00:11:34.329 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mds[73398]: -- 172.21.15.103:0/2369165972 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555849d79a40 con 0x555849d4cc00 2024-05-23T00:11:34.330 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a[73394]: starting mds.a at 2024-05-23T00:11:34.367 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c[67104]: ignoring --setuser ceph since I am not root 2024-05-23T00:11:34.367 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c[67104]: ignoring --setgroup ceph since I am not root 2024-05-23T00:11:34.367 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:11:33 smithi174 ceph-mds[67108]: -- 172.21.15.174:0/451692987 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c87b629a40 con 0x55c87b5fcc00 2024-05-23T00:11:34.367 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c[67104]: starting mds.c at 2024-05-23T00:11:34.368 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: ignoring --setuser ceph since I am not root 2024-05-23T00:11:34.368 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: ignoring --setgroup ceph since I am not root 2024-05-23T00:11:34.368 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:11:33 smithi174 ceph-mds[67413]: -- 172.21.15.174:0/1796851171 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55751a90da40 con 0x55751a8e0800 2024-05-23T00:11:34.368 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: starting mds.i at 2024-05-23T00:11:34.369 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f[66798]: ignoring --setuser ceph since I am not root 2024-05-23T00:11:34.369 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f[66798]: ignoring --setgroup ceph since I am not root 2024-05-23T00:11:34.369 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:11:33 smithi174 ceph-mds[66802]: -- 172.21.15.174:0/3432901829 <== mon.1 v2:172.21.15.174:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563258699a40 con 0x56325866c800 2024-05-23T00:11:34.369 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:11:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f[66798]: starting mds.f at 2024-05-23T00:11:34.411 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.409+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6cb41a7060 con 0x7f6cb4102970 2024-05-23T00:11:34.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: pgmap v345: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:34.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2392007873' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-23T00:11:34.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-23T00:11:34.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-23T00:11:34.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-23T00:11:34.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-05-23T00:11:34.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-23T00:11:34.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: osdmap e107: 12 total, 12 up, 12 in 2024-05-23T00:11:34.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-23T00:11:34.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2831981351' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-23T00:11:34.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:34 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-23T00:11:34.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: pgmap v345: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:34.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2392007873' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-23T00:11:34.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-23T00:11:34.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-23T00:11:34.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-23T00:11:34.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-05-23T00:11:34.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-23T00:11:34.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: osdmap e107: 12 total, 12 up, 12 in 2024-05-23T00:11:34.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-23T00:11:34.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2831981351' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-23T00:11:34.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:34 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-23T00:11:34.940 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.937+0000 7f6cb2000640 1 -- 172.21.15.103:0/2831981351 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v40) ==== 104+0+0 (secure 0 0 0) 0x7f6ca40bc850 con 0x7f6cb4102970 2024-05-23T00:11:34.944 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6c980a2d00 msgr2=0x7f6c980a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:34.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6c980a2d00 0x7f6c980a51c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6cb410d9f0 tx=0x7f6ca0005f90 comp rx=0 tx=0).stop 2024-05-23T00:11:34.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cb4102970 msgr2=0x7f6cb419a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:34.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cb4102970 0x7f6cb419a320 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4004af0 tx=0x7f6ca402f6b0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.945 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 shutdown_connections 2024-05-23T00:11:34.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6c980a2d00 0x7f6c980a51c0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6cb410d9f0 tx=0x7f6ca0005f90 comp rx=0 tx=0).stop 2024-05-23T00:11:34.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6cb410be90 0x7f6cb419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cb4103290 0x7f6cb419a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 --2- 172.21.15.103:0/2831981351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6cb4102970 0x7f6cb419a320 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:34.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.942+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 >> 172.21.15.103:0/2831981351 conn(0x7f6cb40778d0 msgr2=0x7f6cb4107d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:34.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.943+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 shutdown_connections 2024-05-23T00:11:34.946 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:34.943+0000 7f6cbba00640 1 -- 172.21.15.103:0/2831981351 wait complete. 2024-05-23T00:11:34.967 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-23T00:11:35.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: pgmap v345: 129 pgs: 129 active+clean; 84 MiB data, 600 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-23T00:11:35.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2392007873' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-23T00:11:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-23T00:11:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-23T00:11:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-23T00:11:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: Health check cleared: MDS_DAMAGE (was: 1 MDSs report damaged metadata) 2024-05-23T00:11:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-23T00:11:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: osdmap e107: 12 total, 12 up, 12 in 2024-05-23T00:11:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-23T00:11:35.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2831981351' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-23T00:11:35.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:34 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-23T00:11:35.171 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.168+0000 7f30e7200640 1 -- 172.21.15.103:0/796240841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e81125c0 msgr2=0x7f30e81149b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:35.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.168+0000 7f30e7200640 1 --2- 172.21.15.103:0/796240841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e81125c0 0x7f30e81149b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f30e0009ca0 tx=0x7f30e002a8c0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.168+0000 7f30e7200640 1 -- 172.21.15.103:0/796240841 shutdown_connections 2024-05-23T00:11:35.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.168+0000 7f30e7200640 1 --2- 172.21.15.103:0/796240841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e81125c0 0x7f30e81149b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.168+0000 7f30e7200640 1 --2- 172.21.15.103:0/796240841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30e80766c0 0x7f30e8112080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.168+0000 7f30e7200640 1 --2- 172.21.15.103:0/796240841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e8075da0 0x7f30e8076180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.169+0000 7f30e7200640 1 -- 172.21.15.103:0/796240841 >> 172.21.15.103:0/796240841 conn(0x7f30e80ff3b0 msgr2=0x7f30e81017d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:35.172 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.169+0000 7f30e7200640 1 -- 172.21.15.103:0/796240841 shutdown_connections 2024-05-23T00:11:35.173 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.169+0000 7f30e7200640 1 -- 172.21.15.103:0/796240841 wait complete. 2024-05-23T00:11:35.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.170+0000 7f30e7200640 1 Processor -- start 2024-05-23T00:11:35.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.170+0000 7f30e7200640 1 -- start start 2024-05-23T00:11:35.174 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e7200640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e8075da0 0x7f30e819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:35.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e7200640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e80766c0 0x7f30e819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:35.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e7200640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30e81125c0 0x7f30e81a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:35.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e7200640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30e8118150 con 0x7f30e8075da0 2024-05-23T00:11:35.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e7200640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f30e8117fd0 con 0x7f30e81125c0 2024-05-23T00:11:35.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e7200640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30e81182d0 con 0x7f30e80766c0 2024-05-23T00:11:35.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e8075da0 0x7f30e819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:35.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e8075da0 0x7f30e819e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51746/0 (socket says 172.21.15.103:51746) 2024-05-23T00:11:35.175 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5e00640 1 -- 172.21.15.103:0/2911436198 learned_addr learned my addr 172.21.15.103:0/2911436198 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:35.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5400640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e80766c0 0x7f30e819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:35.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e6800640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30e81125c0 0x7f30e81a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:35.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5e00640 1 -- 172.21.15.103:0/2911436198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e80766c0 msgr2=0x7f30e819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:35.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5e00640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e80766c0 0x7f30e819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5e00640 1 -- 172.21.15.103:0/2911436198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30e81125c0 msgr2=0x7f30e81a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:35.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5e00640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30e81125c0 0x7f30e81a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.171+0000 7f30e5e00640 1 -- 172.21.15.103:0/2911436198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e81a3610 con 0x7f30e8075da0 2024-05-23T00:11:35.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.172+0000 7f30e5400640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e80766c0 0x7f30e819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:35.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.172+0000 7f30e5e00640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e8075da0 0x7f30e819e670 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f30d800e9f0 tx=0x7f30d800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:35.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.172+0000 7f30dea00640 1 -- 172.21.15.103:0/2911436198 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d800cd80 con 0x7f30e8075da0 2024-05-23T00:11:35.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.172+0000 7f30dea00640 1 -- 172.21.15.103:0/2911436198 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f30d8005500 con 0x7f30e8075da0 2024-05-23T00:11:35.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.172+0000 7f30dea00640 1 -- 172.21.15.103:0/2911436198 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d8010690 con 0x7f30e8075da0 2024-05-23T00:11:35.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.172+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e81a3900 con 0x7f30e8075da0 2024-05-23T00:11:35.178 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.172+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e81ab190 con 0x7f30e8075da0 2024-05-23T00:11:35.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.173+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e80685b0 con 0x7f30e8075da0 2024-05-23T00:11:35.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.176+0000 7f30dea00640 1 -- 172.21.15.103:0/2911436198 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f30d80046c0 con 0x7f30e8075da0 2024-05-23T00:11:35.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.177+0000 7f30dea00640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30cc0a2d80 0x7f30cc0a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:35.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.177+0000 7f30dea00640 1 -- 172.21.15.103:0/2911436198 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(107..107 src has 1..107) ==== 9782+0+0 (secure 0 0 0) 0x7f30d80cfd20 con 0x7f30e8075da0 2024-05-23T00:11:35.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.177+0000 7f30e5400640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30cc0a2d80 0x7f30cc0a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:35.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.178+0000 7f30e5400640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30cc0a2d80 0x7f30cc0a5240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f30e8102f50 tx=0x7f30d4005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:35.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.182+0000 7f30dea00640 1 -- 172.21.15.103:0/2911436198 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f30d809c3f0 con 0x7f30e8075da0 2024-05-23T00:11:35.387 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.384+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f30e819f970 con 0x7f30e8075da0 2024-05-23T00:11:35.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-23T00:11:35.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-23T00:11:35.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: Cluster is now healthy 2024-05-23T00:11:35.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-23T00:11:35.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.105:6836/3078654437,v1:172.21.15.105:6838/3078654437] up:boot 2024-05-23T00:11:35.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.105:6837/2071840803,v1:172.21.15.105:6839/2071840803] up:boot 2024-05-23T00:11:35.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6834/1069876351,v1:172.21.15.174:6835/1069876351] up:boot 2024-05-23T00:11:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6836/3625637884,v1:172.21.15.174:6838/3625637884] up:boot 2024-05-23T00:11:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.174:6837/1105456927,v1:172.21.15.174:6839/1105456927] up:boot 2024-05-23T00:11:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: mds.? [v2:172.21.15.103:6840/681464891,v1:172.21.15.103:6841/681464891] up:boot 2024-05-23T00:11:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: fsmap 12 up:standby 2024-05-23T00:11:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-23T00:11:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-23T00:11:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-23T00:11:35.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-23T00:11:35.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-23T00:11:35.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-23T00:11:35.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:35 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2911436198' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:35.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-23T00:11:35.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-23T00:11:35.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: Cluster is now healthy 2024-05-23T00:11:35.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-23T00:11:35.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.105:6836/3078654437,v1:172.21.15.105:6838/3078654437] up:boot 2024-05-23T00:11:35.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.105:6837/2071840803,v1:172.21.15.105:6839/2071840803] up:boot 2024-05-23T00:11:35.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6834/1069876351,v1:172.21.15.174:6835/1069876351] up:boot 2024-05-23T00:11:35.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6836/3625637884,v1:172.21.15.174:6838/3625637884] up:boot 2024-05-23T00:11:35.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.174:6837/1105456927,v1:172.21.15.174:6839/1105456927] up:boot 2024-05-23T00:11:35.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: mds.? [v2:172.21.15.103:6840/681464891,v1:172.21.15.103:6841/681464891] up:boot 2024-05-23T00:11:35.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: fsmap 12 up:standby 2024-05-23T00:11:35.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-23T00:11:35.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-23T00:11:35.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-23T00:11:35.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-23T00:11:35.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-23T00:11:35.831 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-23T00:11:35.832 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:35 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2911436198' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:35.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-23T00:11:35.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-23T00:11:35.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: Cluster is now healthy 2024-05-23T00:11:35.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-23T00:11:35.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.105:6836/3078654437,v1:172.21.15.105:6838/3078654437] up:boot 2024-05-23T00:11:35.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.105:6837/2071840803,v1:172.21.15.105:6839/2071840803] up:boot 2024-05-23T00:11:35.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6834/1069876351,v1:172.21.15.174:6835/1069876351] up:boot 2024-05-23T00:11:35.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6836/3625637884,v1:172.21.15.174:6838/3625637884] up:boot 2024-05-23T00:11:35.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.174:6837/1105456927,v1:172.21.15.174:6839/1105456927] up:boot 2024-05-23T00:11:35.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: mds.? [v2:172.21.15.103:6840/681464891,v1:172.21.15.103:6841/681464891] up:boot 2024-05-23T00:11:35.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: fsmap 12 up:standby 2024-05-23T00:11:35.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-23T00:11:35.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-23T00:11:35.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-23T00:11:35.868 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-23T00:11:35.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-23T00:11:35.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: from='mgr.25390 172.21.15.103:0/1706405880' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-23T00:11:35.869 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:35 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2911436198' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.946+0000 7f30dea00640 1 -- 172.21.15.103:0/2911436198 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v108) ==== 181+0+0 (secure 0 0 0) 0x7f30d809bcf0 con 0x7f30e8075da0 2024-05-23T00:11:35.949 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'cephfs_metadata' removed 2024-05-23T00:11:35.953 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.950+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30cc0a2d80 msgr2=0x7f30cc0a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:35.953 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.950+0000 7f30e7200640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30cc0a2d80 0x7f30cc0a5240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f30e8102f50 tx=0x7f30d4005f90 comp rx=0 tx=0).stop 2024-05-23T00:11:35.953 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.950+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e8075da0 msgr2=0x7f30e819e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:35.953 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.950+0000 7f30e7200640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e8075da0 0x7f30e819e670 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f30d800e9f0 tx=0x7f30d800edf0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.951+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 shutdown_connections 2024-05-23T00:11:35.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.951+0000 7f30e7200640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f30cc0a2d80 0x7f30cc0a5240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.951+0000 7f30e7200640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f30e81125c0 0x7f30e81a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.951+0000 7f30e7200640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30e80766c0 0x7f30e819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.951+0000 7f30e7200640 1 --2- 172.21.15.103:0/2911436198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f30e8075da0 0x7f30e819e670 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:35.954 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.951+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 >> 172.21.15.103:0/2911436198 conn(0x7f30e80ff3b0 msgr2=0x7f30e81017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:35.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.951+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 shutdown_connections 2024-05-23T00:11:35.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:35.952+0000 7f30e7200640 1 -- 172.21.15.103:0/2911436198 wait complete. 2024-05-23T00:11:35.983 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-23T00:11:36.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.172+0000 7f8172a00640 1 -- 172.21.15.103:0/2717748047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c107e10 msgr2=0x7f816c108270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:36.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.172+0000 7f8172a00640 1 --2- 172.21.15.103:0/2717748047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c107e10 0x7f816c108270 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8158009900 tx=0x7f815802a480 comp rx=0 tx=0).stop 2024-05-23T00:11:36.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.173+0000 7f8172a00640 1 -- 172.21.15.103:0/2717748047 shutdown_connections 2024-05-23T00:11:36.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.173+0000 7f8172a00640 1 --2- 172.21.15.103:0/2717748047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f816c1087b0 0x7f816c114bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.173+0000 7f8172a00640 1 --2- 172.21.15.103:0/2717748047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c107e10 0x7f816c108270 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.176 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.173+0000 7f8172a00640 1 --2- 172.21.15.103:0/2717748047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f816c10de30 0x7f816c10e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.173+0000 7f8172a00640 1 -- 172.21.15.103:0/2717748047 >> 172.21.15.103:0/2717748047 conn(0x7f816c077690 msgr2=0x7f816c106690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:36.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.173+0000 7f8172a00640 1 -- 172.21.15.103:0/2717748047 shutdown_connections 2024-05-23T00:11:36.177 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.173+0000 7f8172a00640 1 -- 172.21.15.103:0/2717748047 wait complete. 2024-05-23T00:11:36.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172a00640 1 Processor -- start 2024-05-23T00:11:36.179 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172a00640 1 -- start start 2024-05-23T00:11:36.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172a00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f816c107e10 0x7f816c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:36.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172a00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f816c1087b0 0x7f816c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:36.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172a00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c10de30 0x7f816c1a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:36.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172a00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f816c1180f0 con 0x7f816c107e10 2024-05-23T00:11:36.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172a00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f816c117f70 con 0x7f816c10de30 2024-05-23T00:11:36.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172a00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f816c118270 con 0x7f816c1087b0 2024-05-23T00:11:36.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8171600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f816c107e10 0x7f816c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:36.180 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8171600640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f816c107e10 0x7f816c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51764/0 (socket says 172.21.15.103:51764) 2024-05-23T00:11:36.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8171600640 1 -- 172.21.15.103:0/782279931 learned_addr learned my addr 172.21.15.103:0/782279931 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:36.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8172000640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c10de30 0x7f816c1a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:36.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.174+0000 7f8170c00640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f816c1087b0 0x7f816c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:36.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f8172000640 1 -- 172.21.15.103:0/782279931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f816c1087b0 msgr2=0x7f816c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:36.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f8172000640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f816c1087b0 0x7f816c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f8172000640 1 -- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f816c107e10 msgr2=0x7f816c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:36.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f8172000640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f816c107e10 0x7f816c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.181 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f8172000640 1 -- 172.21.15.103:0/782279931 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816c1a35f0 con 0x7f816c10de30 2024-05-23T00:11:36.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f8171600640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f816c107e10 0x7f816c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:36.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f8172000640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c10de30 0x7f816c1a2ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f816400e960 tx=0x7f816400ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:36.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f816a000640 1 -- 172.21.15.103:0/782279931 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8164004d20 con 0x7f816c10de30 2024-05-23T00:11:36.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f816a000640 1 -- 172.21.15.103:0/782279931 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8164005020 con 0x7f816c10de30 2024-05-23T00:11:36.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f816c1a38e0 con 0x7f816c10de30 2024-05-23T00:11:36.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.175+0000 7f816a000640 1 -- 172.21.15.103:0/782279931 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81640116d0 con 0x7f816c10de30 2024-05-23T00:11:36.182 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.176+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f816c1ab170 con 0x7f816c10de30 2024-05-23T00:11:36.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.177+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f816c068a00 con 0x7f816c10de30 2024-05-23T00:11:36.183 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.179+0000 7f816a000640 1 -- 172.21.15.103:0/782279931 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f8164005540 con 0x7f816c10de30 2024-05-23T00:11:36.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.180+0000 7f816a000640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f81540a2d00 0x7f81540a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:36.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.180+0000 7f816a000640 1 -- 172.21.15.103:0/782279931 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(108..108 src has 1..108) ==== 9199+0+0 (secure 0 0 0) 0x7f81640cfb50 con 0x7f816c10de30 2024-05-23T00:11:36.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.180+0000 7f8171600640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f81540a2d00 0x7f81540a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:36.184 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.181+0000 7f8171600640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f81540a2d00 0x7f81540a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8160011440 tx=0x7f8160005f90 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:36.187 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.184+0000 7f816a000640 1 -- 172.21.15.103:0/782279931 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f8164016030 con 0x7f816c10de30 2024-05-23T00:11:36.390 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.387+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f816c110280 con 0x7f816c10de30 2024-05-23T00:11:36.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:36 smithi103 ceph-mon[31370]: pgmap v347: 129 pgs: 129 active+clean; 84 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2024-05-23T00:11:36.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:36 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2911436198' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-23T00:11:36.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:36 smithi103 ceph-mon[31370]: osdmap e108: 12 total, 12 up, 12 in 2024-05-23T00:11:36.829 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:36 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/782279931' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:36.830 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:36 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:36.866 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:36 smithi174 ceph-mon[36179]: pgmap v347: 129 pgs: 129 active+clean; 84 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2024-05-23T00:11:36.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:36 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2911436198' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-23T00:11:36.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:36 smithi174 ceph-mon[36179]: osdmap e108: 12 total, 12 up, 12 in 2024-05-23T00:11:36.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:36 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/782279931' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:36.867 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:36 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:36.955 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.953+0000 7f816a000640 1 -- 172.21.15.103:0/782279931 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v109) ==== 169+0+0 (secure 0 0 0) 0x7f816409be20 con 0x7f816c10de30 2024-05-23T00:11:36.956 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'cephfs_data' removed 2024-05-23T00:11:36.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.961+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f81540a2d00 msgr2=0x7f81540a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:36.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.961+0000 7f8172a00640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f81540a2d00 0x7f81540a51c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8160011440 tx=0x7f8160005f90 comp rx=0 tx=0).stop 2024-05-23T00:11:36.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.961+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c10de30 msgr2=0x7f816c1a2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:36.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.961+0000 7f8172a00640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c10de30 0x7f816c1a2ec0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f816400e960 tx=0x7f816400ee20 comp rx=0 tx=0).stop 2024-05-23T00:11:36.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.962+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 shutdown_connections 2024-05-23T00:11:36.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.962+0000 7f8172a00640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f81540a2d00 0x7f81540a51c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.964 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.962+0000 7f8172a00640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f816c10de30 0x7f816c1a2ec0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.962+0000 7f8172a00640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f816c1087b0 0x7f816c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.962+0000 7f8172a00640 1 --2- 172.21.15.103:0/782279931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f816c107e10 0x7f816c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:36.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.962+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 >> 172.21.15.103:0/782279931 conn(0x7f816c077690 msgr2=0x7f816c078780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:36.965 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.962+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 shutdown_connections 2024-05-23T00:11:36.966 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:36.962+0000 7f8172a00640 1 -- 172.21.15.103:0/782279931 wait complete. 2024-05-23T00:11:36.995 DEBUG:teuthology.orchestra.run.smithi103:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-23T00:11:37.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:36 smithi105 ceph-mon[36426]: pgmap v347: 129 pgs: 129 active+clean; 84 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2024-05-23T00:11:37.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:36 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2911436198' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-23T00:11:37.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:36 smithi105 ceph-mon[36426]: osdmap e108: 12 total, 12 up, 12 in 2024-05-23T00:11:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:36 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/782279931' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:37.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:36 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:37.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 -- 172.21.15.103:0/2637914296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec0766c0 msgr2=0x7f6aec1121b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:37.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2637914296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec0766c0 0x7f6aec1121b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009900 tx=0x7f6ad402a3f0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 -- 172.21.15.103:0/2637914296 shutdown_connections 2024-05-23T00:11:37.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2637914296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec1126f0 0x7f6aec114ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2637914296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec0766c0 0x7f6aec1121b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.197 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2637914296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6aec075da0 0x7f6aec076180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 -- 172.21.15.103:0/2637914296 >> 172.21.15.103:0/2637914296 conn(0x7f6aec0ff4e0 msgr2=0x7f6aec101900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:37.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 -- 172.21.15.103:0/2637914296 shutdown_connections 2024-05-23T00:11:37.198 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.194+0000 7f6af0e00640 1 -- 172.21.15.103:0/2637914296 wait complete. 2024-05-23T00:11:37.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.195+0000 7f6af0e00640 1 Processor -- start 2024-05-23T00:11:37.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.195+0000 7f6af0e00640 1 -- start start 2024-05-23T00:11:37.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.195+0000 7f6af0e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec075da0 0x7f6aec1a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:37.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.195+0000 7f6af0e00640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6aec0766c0 0x7f6aec1a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:37.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.195+0000 7f6af0e00640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec1126f0 0x7f6aec19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:37.200 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6af0e00640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec118240 con 0x7f6aec075da0 2024-05-23T00:11:37.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6af0e00640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1180c0 con 0x7f6aec1126f0 2024-05-23T00:11:37.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6af0e00640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6aec1183c0 con 0x7f6aec0766c0 2024-05-23T00:11:37.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec075da0 0x7f6aec1a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:37.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae9e00640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec075da0 0x7f6aec1a4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:51788/0 (socket says 172.21.15.103:51788) 2024-05-23T00:11:37.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae9e00640 1 -- 172.21.15.103:0/2663536224 learned_addr learned my addr 172.21.15.103:0/2663536224 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-23T00:11:37.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6aea800640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec1126f0 0x7f6aec19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:37.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae3e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6aec0766c0 0x7f6aec1a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:37.201 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae9e00640 1 -- 172.21.15.103:0/2663536224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6aec0766c0 msgr2=0x7f6aec1a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:37.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae9e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6aec0766c0 0x7f6aec1a4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae9e00640 1 -- 172.21.15.103:0/2663536224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec1126f0 msgr2=0x7f6aec19e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:37.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae9e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec1126f0 0x7f6aec19e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6ae9e00640 1 -- 172.21.15.103:0/2663536224 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aec19ed40 con 0x7f6aec075da0 2024-05-23T00:11:37.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.196+0000 7f6aea800640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec1126f0 0x7f6aec19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-23T00:11:37.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.197+0000 7f6ae9e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec075da0 0x7f6aec1a4410 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00e9f0 tx=0x7f6adc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:37.202 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.197+0000 7f6ae2a00640 1 -- 172.21.15.103:0/2663536224 <== mon.0 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc00cd80 con 0x7f6aec075da0 2024-05-23T00:11:37.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.197+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aec19f030 con 0x7f6aec075da0 2024-05-23T00:11:37.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.197+0000 7f6ae2a00640 1 -- 172.21.15.103:0/2663536224 <== mon.0 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6adc010910 con 0x7f6aec075da0 2024-05-23T00:11:37.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.197+0000 7f6ae2a00640 1 -- 172.21.15.103:0/2663536224 <== mon.0 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0188f0 con 0x7f6aec075da0 2024-05-23T00:11:37.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.197+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aec1ab2d0 con 0x7f6aec075da0 2024-05-23T00:11:37.203 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.198+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aec068690 con 0x7f6aec075da0 2024-05-23T00:11:37.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.200+0000 7f6ae2a00640 1 -- 172.21.15.103:0/2663536224 <== mon.0 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 27) ==== 137965+0+0 (secure 0 0 0) 0x7f6adc010ac0 con 0x7f6aec075da0 2024-05-23T00:11:37.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.202+0000 7f6ae2a00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6ad00a2d00 0x7f6ad00a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-23T00:11:37.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.202+0000 7f6ae2a00640 1 -- 172.21.15.103:0/2663536224 <== mon.0 v2:172.21.15.103:3300/0 5 ==== osd_map(109..109 src has 1..109) ==== 8582+0+0 (secure 0 0 0) 0x7f6adc0cf880 con 0x7f6aec075da0 2024-05-23T00:11:37.204 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.202+0000 7f6ae3e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6ad00a2d00 0x7f6ad00a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-23T00:11:37.205 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.203+0000 7f6ae3e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6ad00a2d00 0x7f6ad00a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009230 tx=0x7f6ad40058d0 comp rx=0 tx=0).ready entity=mgr.25390 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-23T00:11:37.208 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.206+0000 7f6ae2a00640 1 -- 172.21.15.103:0/2663536224 <== mon.0 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186418 (secure 0 0 0) 0x7f6adc09bdb0 con 0x7f6aec075da0 2024-05-23T00:11:37.412 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.409+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6aec02d200 con 0x7f6aec075da0 2024-05-23T00:11:37.963 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.960+0000 7f6ae2a00640 1 -- 172.21.15.103:0/2663536224 <== mon.0 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v110) ==== 178+0+0 (secure 0 0 0) 0x7f6adc0a00b0 con 0x7f6aec075da0 2024-05-23T00:11:37.963 INFO:teuthology.orchestra.run.smithi103.stderr:pool 'cephfs_data_ec' removed 2024-05-23T00:11:37.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.965+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6ad00a2d00 msgr2=0x7f6ad00a51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:37.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.965+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6ad00a2d00 0x7f6ad00a51c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009230 tx=0x7f6ad40058d0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.968 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.965+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec075da0 msgr2=0x7f6aec1a4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-23T00:11:37.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.965+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec075da0 0x7f6aec1a4410 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00e9f0 tx=0x7f6adc00eeb0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.969 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.967+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 shutdown_connections 2024-05-23T00:11:37.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.967+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:6800/1519813025,v1:172.21.15.103:6801/1519813025] conn(0x7f6ad00a2d00 0x7f6ad00a51c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.967+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6aec1126f0 0x7f6aec19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.967+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6aec0766c0 0x7f6aec1a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.967+0000 7f6af0e00640 1 --2- 172.21.15.103:0/2663536224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6aec075da0 0x7f6aec1a4410 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-23T00:11:37.970 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.967+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 >> 172.21.15.103:0/2663536224 conn(0x7f6aec0ff4e0 msgr2=0x7f6aec1018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-23T00:11:37.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.968+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 shutdown_connections 2024-05-23T00:11:37.971 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-23T00:11:37.968+0000 7f6af0e00640 1 -- 172.21.15.103:0/2663536224 wait complete. 2024-05-23T00:11:37.997 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-23T00:11:38.068 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-23T00:11:38.100 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-23T00:11:38.127 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-23T00:11:38.127 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-23T00:11:38.184 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-23T00:11:38.200 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-23T00:11:38.216 INFO:tasks.cephadm:Stopping all daemons... 2024-05-23T00:11:38.216 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-23T00:11:38.216 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a 2024-05-23T00:11:38.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:37 smithi103 ceph-mon[31370]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-23T00:11:38.289 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:37 smithi103 ceph-mon[31370]: osdmap e109: 12 total, 12 up, 12 in 2024-05-23T00:11:38.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:37 smithi103 ceph-mon[31370]: pgmap v350: 33 pgs: 33 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:11:38.290 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:37 smithi103 ceph-mon[31370]: from='client.? 172.21.15.103:0/2663536224' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:38.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:37 smithi105 ceph-mon[36426]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-23T00:11:38.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:37 smithi105 ceph-mon[36426]: osdmap e109: 12 total, 12 up, 12 in 2024-05-23T00:11:38.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:37 smithi105 ceph-mon[36426]: pgmap v350: 33 pgs: 33 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:11:38.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:37 smithi105 ceph-mon[36426]: from='client.? 172.21.15.103:0/2663536224' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:38.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:37 smithi174 ceph-mon[36179]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-23T00:11:38.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:37 smithi174 ceph-mon[36179]: osdmap e109: 12 total, 12 up, 12 in 2024-05-23T00:11:38.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:37 smithi174 ceph-mon[36179]: pgmap v350: 33 pgs: 33 active+clean; 26 MiB data, 604 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-23T00:11:38.367 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:37 smithi174 ceph-mon[36179]: from='client.? 172.21.15.103:0/2663536224' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-23T00:11:38.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:38 smithi103 systemd[1]: Stopping Ceph mon.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:11:39.251 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[31366]: 2024-05-23T00:11:38.829+0000 7fe449800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:11:39.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a[31366]: 2024-05-23T00:11:38.829+0000 7fe449800700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-23T00:11:39.252 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:38 smithi103 podman[100358]: 2024-05-23 00:11:38.911857891 +0000 UTC m=+0.253079439 container died 5991f9961eca9c90b37667cedc4d48ea60dc400738805add2efcf99fc3cc9477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-05-23T00:11:39.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:39 smithi103 podman[100358]: 2024-05-23 00:11:39.383678909 +0000 UTC m=+0.724900510 container cleanup 5991f9961eca9c90b37667cedc4d48ea60dc400738805add2efcf99fc3cc9477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-23T00:11:39.579 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:39 smithi103 bash[100358]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a 2024-05-23T00:11:40.078 INFO:journalctl@ceph.mon.a.smithi103.stdout:May 23 00:11:39 smithi103 podman[100371]: 2024-05-23 00:11:39.685333745 +0000 UTC m=+0.763862292 container remove 5991f9961eca9c90b37667cedc4d48ea60dc400738805add2efcf99fc3cc9477 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-23T00:11:40.279 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.a.service' 2024-05-23T00:11:40.408 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:11:40.409 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-23T00:11:40.409 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-23T00:11:40.409 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.b 2024-05-23T00:11:40.734 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:40 smithi105 systemd[1]: Stopping Ceph mon.b for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:11:41.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-b[36422]: 2024-05-23T00:11:40.732+0000 7f4010b7b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:11:41.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-b[36422]: 2024-05-23T00:11:40.732+0000 7f4010b7b700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-23T00:11:41.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:40 smithi105 podman[72581]: 2024-05-23 00:11:40.977734594 +0000 UTC m=+0.365674638 container died 7d4d62ba0f8dcf60f88637d97b8e328af8f80abd043633fcc1e48c8e8214887e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True) 2024-05-23T00:11:41.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:41 smithi105 podman[72581]: 2024-05-23 00:11:41.491430461 +0000 UTC m=+0.879370505 container cleanup 7d4d62ba0f8dcf60f88637d97b8e328af8f80abd043633fcc1e48c8e8214887e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-23T00:11:41.736 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:41 smithi105 bash[72581]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-b 2024-05-23T00:11:41.972 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.b.service' 2024-05-23T00:11:41.994 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:41 smithi105 podman[72592]: 2024-05-23 00:11:41.735038506 +0000 UTC m=+0.759412261 container remove 7d4d62ba0f8dcf60f88637d97b8e328af8f80abd043633fcc1e48c8e8214887e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-23T00:11:41.995 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:41 smithi105 systemd[1]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-23T00:11:41.995 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:41 smithi105 systemd[1]: Stopped Ceph mon.b for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-23T00:11:41.995 INFO:journalctl@ceph.mon.b.smithi105.stdout:May 23 00:11:41 smithi105 systemd[1]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.b.service: Consumed 14.625s CPU time. 2024-05-23T00:11:42.058 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:11:42.058 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-23T00:11:42.058 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-23T00:11:42.059 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.c 2024-05-23T00:11:42.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:42 smithi174 systemd[1]: Stopping Ceph mon.c for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:11:42.759 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:42 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-c[36162]: 2024-05-23T00:11:42.387+0000 7fa3ddeff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:11:42.759 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:42 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-c[36162]: 2024-05-23T00:11:42.387+0000 7fa3ddeff700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-23T00:11:42.759 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:42 smithi174 podman[73044]: 2024-05-23 00:11:42.459758272 +0000 UTC m=+0.199112717 container died bf01ac9e5176755fce6e0611148b74411c56235d93fa5855dc6a865623eaa459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-23T00:11:43.099 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:42 smithi174 podman[73044]: 2024-05-23 00:11:42.855482043 +0000 UTC m=+0.594836488 container cleanup bf01ac9e5176755fce6e0611148b74411c56235d93fa5855dc6a865623eaa459 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-c, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762) 2024-05-23T00:11:43.100 INFO:journalctl@ceph.mon.c.smithi174.stdout:May 23 00:11:42 smithi174 bash[73044]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mon-c 2024-05-23T00:11:43.239 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mon.c.service' 2024-05-23T00:11:43.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:11:43.289 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-23T00:11:43.289 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-23T00:11:43.289 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.x 2024-05-23T00:11:43.829 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:11:43 smithi103 systemd[1]: Stopping Ceph mgr.x for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:11:44.306 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:11:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:11:43.869+0000 7fd8d3200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:11:44.306 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:11:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x[31593]: 2024-05-23T00:11:43.869+0000 7fd8d3200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-05-23T00:11:44.306 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:11:43 smithi103 podman[100480]: 2024-05-23 00:11:43.967226199 +0000 UTC m=+0.239410449 container died d82c41646a04b592d6af0d70616f081d66eecc37e39a94f26853d0ba47446467 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:11:44.579 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:11:44 smithi103 podman[100480]: 2024-05-23 00:11:44.489878212 +0000 UTC m=+0.762062454 container cleanup d82c41646a04b592d6af0d70616f081d66eecc37e39a94f26853d0ba47446467 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-23T00:11:44.579 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:11:44 smithi103 bash[100480]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x 2024-05-23T00:11:45.079 INFO:journalctl@ceph.mgr.x.smithi103.stdout:May 23 00:11:44 smithi103 podman[100492]: 2024-05-23 00:11:44.725407785 +0000 UTC m=+0.747295582 container remove d82c41646a04b592d6af0d70616f081d66eecc37e39a94f26853d0ba47446467 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e1f76222e88afe0f1cda37775b2d45d88289794, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, ceph=True) 2024-05-23T00:11:45.214 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.x.service' 2024-05-23T00:11:45.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:11:45.328 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-23T00:11:45.328 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-23T00:11:45.328 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.y 2024-05-23T00:11:45.676 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:11:45 smithi105 systemd[1]: Stopping Ceph mgr.y for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:11:45.982 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:11:45 smithi105 podman[72702]: 2024-05-23 00:11:45.675315532 +0000 UTC m=+0.150218956 container died 4c2a1c3611d0c422576116421d8454c60a8d988d4938776202ed15849742c9c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-23T00:11:46.307 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:11:46 smithi105 podman[72702]: 2024-05-23 00:11:46.080433107 +0000 UTC m=+0.555336525 container cleanup 4c2a1c3611d0c422576116421d8454c60a8d988d4938776202ed15849742c9c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-23T00:11:46.307 INFO:journalctl@ceph.mgr.y.smithi105.stdout:May 23 00:11:46 smithi105 bash[72702]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-y 2024-05-23T00:11:46.504 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.y.service' 2024-05-23T00:11:46.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:11:46.538 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-23T00:11:46.538 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-23T00:11:46.538 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.z 2024-05-23T00:11:46.866 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:11:46 smithi174 systemd[1]: Stopping Ceph mgr.z for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:11:47.187 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:11:46 smithi174 podman[73163]: 2024-05-23 00:11:46.881030217 +0000 UTC m=+0.145054559 container died 9e6c9e4eb7105b88b06289536178d9a9fe45d1f6d54730763bd641745702bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-23T00:11:47.530 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:11:47 smithi174 podman[73163]: 2024-05-23 00:11:47.28584541 +0000 UTC m=+0.549869762 container cleanup 9e6c9e4eb7105b88b06289536178d9a9fe45d1f6d54730763bd641745702bd66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:11:47.530 INFO:journalctl@ceph.mgr.z.smithi174.stdout:May 23 00:11:47 smithi174 bash[73163]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mgr-z 2024-05-23T00:11:47.671 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mgr.z.service' 2024-05-23T00:11:47.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:11:47.704 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-23T00:11:47.704 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-23T00:11:47.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.0 2024-05-23T00:11:48.078 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:47 smithi103 systemd[1]: Stopping Ceph osd.0 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:11:48.579 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0[46288]: 2024-05-23T00:11:48.241+0000 7f6557800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:11:48.579 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0[46288]: 2024-05-23T00:11:48.241+0000 7f6557800700 -1 osd.0 110 *** Got signal Terminated *** 2024-05-23T00:11:48.579 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0[46288]: 2024-05-23T00:11:48.241+0000 7f6557800700 -1 osd.0 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:11:53.919 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:53 smithi103 podman[100600]: 2024-05-23 00:11:53.615739721 +0000 UTC m=+5.504225829 container died 577f5ffd4f07cafbd772a40ab6ffab09937d37e78189fd4acce5c033f4501315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True) 2024-05-23T00:11:54.329 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:54 smithi103 podman[100600]: 2024-05-23 00:11:54.054791833 +0000 UTC m=+5.943277993 container cleanup 577f5ffd4f07cafbd772a40ab6ffab09937d37e78189fd4acce5c033f4501315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, ceph=True, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-23T00:11:54.329 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:54 smithi103 bash[100600]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0 2024-05-23T00:11:54.829 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:54 smithi103 podman[100615]: 2024-05-23 00:11:54.397856421 +0000 UTC m=+0.762402786 container remove 577f5ffd4f07cafbd772a40ab6ffab09937d37e78189fd4acce5c033f4501315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:11:55.579 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:55 smithi103 podman[100704]: 2024-05-23 00:11:55.184203315 +0000 UTC m=+0.035589314 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:11:55.579 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:55 smithi103 podman[100704]: 2024-05-23 00:11:55.307261538 +0000 UTC m=+0.158647567 container create 388d8530aa1da073137af2789c03c17169dc6faa8c9aaa0bb3a6c21be3fd0b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-23T00:11:56.082 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:55 smithi103 podman[100704]: 2024-05-23 00:11:55.753238444 +0000 UTC m=+0.604624480 container init 388d8530aa1da073137af2789c03c17169dc6faa8c9aaa0bb3a6c21be3fd0b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-23T00:11:56.082 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:55 smithi103 podman[100704]: 2024-05-23 00:11:55.765186398 +0000 UTC m=+0.616572433 container start 388d8530aa1da073137af2789c03c17169dc6faa8c9aaa0bb3a6c21be3fd0b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:11:56.082 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:55 smithi103 podman[100704]: 2024-05-23 00:11:55.868722725 +0000 UTC m=+0.720108777 container attach 388d8530aa1da073137af2789c03c17169dc6faa8c9aaa0bb3a6c21be3fd0b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-23T00:11:56.452 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:56 smithi103 podman[100731]: 2024-05-23 00:11:56.265198771 +0000 UTC m=+0.033001276 container died 388d8530aa1da073137af2789c03c17169dc6faa8c9aaa0bb3a6c21be3fd0b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-23T00:11:57.043 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.0.service' 2024-05-23T00:11:57.078 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:56 smithi103 podman[100731]: 2024-05-23 00:11:56.998976644 +0000 UTC m=+0.766779223 container remove 388d8530aa1da073137af2789c03c17169dc6faa8c9aaa0bb3a6c21be3fd0b1c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True) 2024-05-23T00:11:57.079 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:57 smithi103 systemd[1]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.0.service: Deactivated successfully. 2024-05-23T00:11:57.079 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:57 smithi103 systemd[1]: Stopped Ceph osd.0 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-23T00:11:57.079 INFO:journalctl@ceph.osd.0.smithi103.stdout:May 23 00:11:57 smithi103 systemd[1]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.0.service: Consumed 38.261s CPU time. 2024-05-23T00:11:57.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:11:57.140 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-23T00:11:57.141 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-23T00:11:57.141 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.1 2024-05-23T00:11:57.463 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:11:57 smithi105 systemd[1]: Stopping Ceph osd.1 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:11:57.807 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:11:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1[40991]: 2024-05-23T00:11:57.459+0000 7f754a022700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:11:57.807 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:11:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1[40991]: 2024-05-23T00:11:57.459+0000 7f754a022700 -1 osd.1 110 *** Got signal Terminated *** 2024-05-23T00:11:57.807 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:11:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1[40991]: 2024-05-23T00:11:57.459+0000 7f754a022700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:12:03.264 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:02 smithi105 podman[72838]: 2024-05-23 00:12:02.970448155 +0000 UTC m=+5.636143195 container died 4eafd9b0928fc85ac51b702121b055d36e5c4b0d52b2d233c95a1f1892e8350a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD) 2024-05-23T00:12:03.557 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:03 smithi105 podman[72838]: 2024-05-23 00:12:03.360954741 +0000 UTC m=+6.026649772 container cleanup 4eafd9b0928fc85ac51b702121b055d36e5c4b0d52b2d233c95a1f1892e8350a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-2e1f762) 2024-05-23T00:12:03.557 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:03 smithi105 bash[72838]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1 2024-05-23T00:12:03.987 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:03 smithi105 podman[72854]: 2024-05-23 00:12:03.645930726 +0000 UTC m=+0.668423350 container remove 4eafd9b0928fc85ac51b702121b055d36e5c4b0d52b2d233c95a1f1892e8350a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-23T00:12:04.260 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:03 smithi105 podman[72941]: 2024-05-23 00:12:03.888257557 +0000 UTC m=+0.017115377 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:12:04.260 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:04 smithi105 podman[72941]: 2024-05-23 00:12:04.013604557 +0000 UTC m=+0.142462379 container create cd067c17a3c92220ffa9ea1aa0db4fea2879a4705bbca1a434a179c342672a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-23T00:12:04.557 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:04 smithi105 podman[72941]: 2024-05-23 00:12:04.258333851 +0000 UTC m=+0.387191657 container init cd067c17a3c92220ffa9ea1aa0db4fea2879a4705bbca1a434a179c342672a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762) 2024-05-23T00:12:04.558 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:04 smithi105 podman[72941]: 2024-05-23 00:12:04.261744197 +0000 UTC m=+0.390601999 container start cd067c17a3c92220ffa9ea1aa0db4fea2879a4705bbca1a434a179c342672a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-23T00:12:04.558 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:04 smithi105 podman[72941]: 2024-05-23 00:12:04.341442257 +0000 UTC m=+0.470300068 container attach cd067c17a3c92220ffa9ea1aa0db4fea2879a4705bbca1a434a179c342672a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-23T00:12:04.937 INFO:journalctl@ceph.osd.1.smithi105.stdout:May 23 00:12:04 smithi105 podman[72941]: 2024-05-23 00:12:04.559034508 +0000 UTC m=+0.687892308 container died cd067c17a3c92220ffa9ea1aa0db4fea2879a4705bbca1a434a179c342672a9d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-23T00:12:05.278 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.1.service' 2024-05-23T00:12:05.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:12:05.317 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-23T00:12:05.317 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-23T00:12:05.317 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.2 2024-05-23T00:12:05.616 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:05 smithi174 systemd[1]: Stopping Ceph osd.2 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:12:06.116 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2[41286]: 2024-05-23T00:12:05.638+0000 7f1dc00d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:12:06.117 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2[41286]: 2024-05-23T00:12:05.638+0000 7f1dc00d3700 -1 osd.2 110 *** Got signal Terminated *** 2024-05-23T00:12:06.117 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2[41286]: 2024-05-23T00:12:05.638+0000 7f1dc00d3700 -1 osd.2 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:12:11.492 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:11 smithi174 podman[73300]: 2024-05-23 00:12:11.183702232 +0000 UTC m=+5.652465521 container died 38aa374a3e9f3891429a03b82f1311e582ec047323603949ef0689c479b714b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:12:11.834 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:11 smithi174 podman[73300]: 2024-05-23 00:12:11.614668767 +0000 UTC m=+6.083432094 container cleanup 38aa374a3e9f3891429a03b82f1311e582ec047323603949ef0689c479b714b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-23T00:12:11.834 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:11 smithi174 bash[73300]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2 2024-05-23T00:12:12.116 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:11 smithi174 podman[73315]: 2024-05-23 00:12:11.832708643 +0000 UTC m=+0.647698516 container remove 38aa374a3e9f3891429a03b82f1311e582ec047323603949ef0689c479b714b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521) 2024-05-23T00:12:12.571 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:12 smithi174 podman[73402]: 2024-05-23 00:12:12.141413811 +0000 UTC m=+0.017024423 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:12:12.571 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:12 smithi174 podman[73402]: 2024-05-23 00:12:12.250169147 +0000 UTC m=+0.125779746 container create ea817180457c118a69ec99f5915bc7c0ef5b4005e1eb4ac358053f6aaf3fd5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:12:12.571 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:12 smithi174 podman[73402]: 2024-05-23 00:12:12.469328511 +0000 UTC m=+0.344939110 container init ea817180457c118a69ec99f5915bc7c0ef5b4005e1eb4ac358053f6aaf3fd5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-23T00:12:12.571 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:12 smithi174 podman[73402]: 2024-05-23 00:12:12.472288169 +0000 UTC m=+0.347898768 container start ea817180457c118a69ec99f5915bc7c0ef5b4005e1eb4ac358053f6aaf3fd5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-23T00:12:12.572 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:12 smithi174 podman[73402]: 2024-05-23 00:12:12.569184985 +0000 UTC m=+0.444795590 container attach ea817180457c118a69ec99f5915bc7c0ef5b4005e1eb4ac358053f6aaf3fd5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-05-23T00:12:12.866 INFO:journalctl@ceph.osd.2.smithi174.stdout:May 23 00:12:12 smithi174 podman[73402]: 2024-05-23 00:12:12.773459752 +0000 UTC m=+0.649070352 container died ea817180457c118a69ec99f5915bc7c0ef5b4005e1eb4ac358053f6aaf3fd5f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-2-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:12:13.361 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.2.service' 2024-05-23T00:12:13.398 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:12:13.399 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-23T00:12:13.399 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-23T00:12:13.399 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.3 2024-05-23T00:12:13.828 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:13 smithi103 systemd[1]: Stopping Ceph osd.3 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:12:14.329 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:14 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-23T00:12:14.059+0000 7fd1cfa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:12:14.329 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:14 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-23T00:12:14.059+0000 7fd1cfa00700 -1 osd.3 110 *** Got signal Terminated *** 2024-05-23T00:12:14.329 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:14 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-23T00:12:14.059+0000 7fd1cfa00700 -1 osd.3 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:12:16.079 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:15 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-23T00:12:15.809+0000 7fd1c6200700 -1 osd.3 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.293389+0000 front 2024-05-23T00:11:49.293737+0000 (oldest deadline 2024-05-23T00:12:15.193577+0000) 2024-05-23T00:12:16.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:15 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:15.670+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:16.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:16 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:16.171+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:17.079 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:16 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-23T00:12:16.805+0000 7fd1c6200700 -1 osd.3 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.293389+0000 front 2024-05-23T00:11:49.293737+0000 (oldest deadline 2024-05-23T00:12:15.193577+0000) 2024-05-23T00:12:17.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:16 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:16.638+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:17.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:17 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:17.189+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:18.079 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:17 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-23T00:12:17.781+0000 7fd1c6200700 -1 osd.3 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.293389+0000 front 2024-05-23T00:11:49.293737+0000 (oldest deadline 2024-05-23T00:12:15.193577+0000) 2024-05-23T00:12:18.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:17 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:17.614+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:18.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:18.061+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:18.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:18 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:18.149+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:19.079 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:18 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3[52936]: 2024-05-23T00:12:18.759+0000 7fd1c6200700 -1 osd.3 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.293389+0000 front 2024-05-23T00:11:49.293737+0000 (oldest deadline 2024-05-23T00:12:15.193577+0000) 2024-05-23T00:12:19.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:18 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:18.648+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:19.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:18.835+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:19.118 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:19.063+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:19.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:19 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:19.176+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:19.829 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:19 smithi103 podman[100823]: 2024-05-23 00:12:19.566269875 +0000 UTC m=+5.699825858 container died 090f8a531765f811ea280607d82f097824445e140278fe598339072be6104ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-23T00:12:19.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:19 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:19.619+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:20.106 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:19.813+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:20.297 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:20 smithi103 podman[100823]: 2024-05-23 00:12:20.045055673 +0000 UTC m=+6.178611695 container cleanup 090f8a531765f811ea280607d82f097824445e140278fe598339072be6104ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-23T00:12:20.297 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:20 smithi103 bash[100823]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3 2024-05-23T00:12:20.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:20.104+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:20.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:20 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:20.139+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:20.578 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:20 smithi103 podman[100841]: 2024-05-23 00:12:20.293721136 +0000 UTC m=+0.716607290 container remove 090f8a531765f811ea280607d82f097824445e140278fe598339072be6104ef7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-23T00:12:21.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:20 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:20.578+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:21.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:20.862+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:21.117 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:21.106+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:21.412 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:21 smithi103 podman[100931]: 2024-05-23 00:12:21.005812081 +0000 UTC m=+0.032043431 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:12:21.412 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:21 smithi103 podman[100931]: 2024-05-23 00:12:21.139403372 +0000 UTC m=+0.165634758 container create 40dc3dcaafa56be65a4063ccf4fb5eeaeaa3b63565613f3d6885fba648024a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:12:21.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:21 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:21.092+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:21.828 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:21 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:21.554+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:21.829 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:21 smithi103 podman[100931]: 2024-05-23 00:12:21.409724418 +0000 UTC m=+0.435955814 container init 40dc3dcaafa56be65a4063ccf4fb5eeaeaa3b63565613f3d6885fba648024a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-23T00:12:21.829 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:21 smithi103 podman[100931]: 2024-05-23 00:12:21.420765442 +0000 UTC m=+0.446996798 container start 40dc3dcaafa56be65a4063ccf4fb5eeaeaa3b63565613f3d6885fba648024a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:12:21.829 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:21 smithi103 podman[100931]: 2024-05-23 00:12:21.525486326 +0000 UTC m=+0.551717755 container attach 40dc3dcaafa56be65a4063ccf4fb5eeaeaa3b63565613f3d6885fba648024a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:12:22.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:21.840+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:22.196 INFO:journalctl@ceph.osd.3.smithi103.stdout:May 23 00:12:21 smithi103 podman[100961]: 2024-05-23 00:12:21.908564656 +0000 UTC m=+0.031403278 container died 40dc3dcaafa56be65a4063ccf4fb5eeaeaa3b63565613f3d6885fba648024a81 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:12:22.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:22 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:22.133+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:22.616 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:22.115+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:22.648 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.3.service' 2024-05-23T00:12:22.741 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:12:22.742 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-23T00:12:22.742 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-23T00:12:22.742 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.4 2024-05-23T00:12:22.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:22 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:22.561+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:23.057 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:22 smithi105 systemd[1]: Stopping Ceph osd.4 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:12:23.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:23.114+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:23.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:22.876+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:23.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:23.151+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:23.558 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-23T00:12:23.081+0000 7f714de81700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:12:23.558 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-23T00:12:23.081+0000 7f714de81700 -1 osd.4 110 *** Got signal Terminated *** 2024-05-23T00:12:23.558 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:23 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-23T00:12:23.081+0000 7f714de81700 -1 osd.4 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:12:24.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:23 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:23.577+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:24.080 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:23 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:23.591+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:24.117 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:24.084+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:24.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:23.850+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:24.458 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:24 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:24.182+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:24.459 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:24 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:24.182+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:24.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:24.575+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:24.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:24.541+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:24.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:24 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:24.541+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:24.866 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:24.843+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:25.057 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:24 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-23T00:12:24.825+0000 7f714a475700 -1 osd.4 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.220420+0000 front 2024-05-23T00:11:59.220441+0000 (oldest deadline 2024-05-23T00:12:23.821299+0000) 2024-05-23T00:12:25.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:25.097+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:25.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:25 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:25.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:25.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:25 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:25.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:25.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:25 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:25.538+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:25.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:25 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:25.561+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:25.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:25 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:25.561+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:26.057 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:25 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-23T00:12:25.790+0000 7f714a475700 -1 osd.4 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.220420+0000 front 2024-05-23T00:11:59.220441+0000 (oldest deadline 2024-05-23T00:12:23.821299+0000) 2024-05-23T00:12:26.232 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:26 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:26.229+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:26.232 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:26 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:26.229+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:26.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:26.060+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:26.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:26.060+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:26.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:25.871+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:26.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:25.871+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:26.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:26.571+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:26.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:26.561+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:26.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:26 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:26.561+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:26.837 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:26.382+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:27.057 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:26 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-23T00:12:26.753+0000 7f714a475700 -1 osd.4 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.220420+0000 front 2024-05-23T00:11:59.220441+0000 (oldest deadline 2024-05-23T00:12:23.821299+0000) 2024-05-23T00:12:27.116 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:27.029+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:27.117 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:27.029+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:27.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:26.835+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:27.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:26.835+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:27.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:27 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:27.194+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:27.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:27 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:27.194+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:27.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:27.610+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:27.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:27.571+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:27.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:27 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:27.571+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:27.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:27.371+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:28.057 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:27 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4[47646]: 2024-05-23T00:12:27.775+0000 7f714a475700 -1 osd.4 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.220420+0000 front 2024-05-23T00:11:59.220441+0000 (oldest deadline 2024-05-23T00:12:23.821299+0000) 2024-05-23T00:12:28.057 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:27 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:27.958+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:28.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:27.875+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:28.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:27 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:27.875+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:28.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:28.003+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:28.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:28.003+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:28.466 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:28 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:28.180+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:28.467 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:28 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:28.180+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:28.784 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:28 smithi105 podman[73052]: 2024-05-23 00:12:28.464782995 +0000 UTC m=+5.525194380 container died 75bd44b718d2d4b8f2fb0e3ed275ce11f1ccf6b174b2868f62e7c0b96c560132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:12:28.828 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:28.574+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:28.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:28.547+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:28.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:28 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:28.547+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:28.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:28.418+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:29.057 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:28 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:28.946+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:29.057 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:28 smithi105 podman[73052]: 2024-05-23 00:12:28.882111074 +0000 UTC m=+5.942522456 container cleanup 75bd44b718d2d4b8f2fb0e3ed275ce11f1ccf6b174b2868f62e7c0b96c560132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-23T00:12:29.058 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:28 smithi105 bash[73052]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4 2024-05-23T00:12:29.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:28.974+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:29.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:28.974+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:29.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:28.881+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:29.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:28 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:28.881+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:29.554 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:29 smithi105 podman[73081]: 2024-05-23 00:12:29.15025914 +0000 UTC m=+0.682493027 container remove 75bd44b718d2d4b8f2fb0e3ed275ce11f1ccf6b174b2868f62e7c0b96c560132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-23T00:12:29.555 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:29 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:29.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:29.555 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:29 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:29.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:29.807 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:29 smithi105 podman[73168]: 2024-05-23 00:12:29.455600813 +0000 UTC m=+0.017461765 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:12:29.807 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:29 smithi105 podman[73168]: 2024-05-23 00:12:29.594792905 +0000 UTC m=+0.156653869 container create 3c8a0f1c253243b4081f55cfe8871ea77734ae8da3f6a794bf9674664b603e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:12:29.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:29.568+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:29.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:29.595+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:29.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:29.595+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:29.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:29 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:29.384+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:30.111 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:29 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:29.957+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:30.111 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:29 smithi105 podman[73168]: 2024-05-23 00:12:29.820510562 +0000 UTC m=+0.382371498 container init 3c8a0f1c253243b4081f55cfe8871ea77734ae8da3f6a794bf9674664b603e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:12:30.111 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:29 smithi105 podman[73168]: 2024-05-23 00:12:29.823574545 +0000 UTC m=+0.385435490 container start 3c8a0f1c253243b4081f55cfe8871ea77734ae8da3f6a794bf9674664b603e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-23T00:12:30.111 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:29 smithi105 podman[73168]: 2024-05-23 00:12:29.920405646 +0000 UTC m=+0.482266605 container attach 3c8a0f1c253243b4081f55cfe8871ea77734ae8da3f6a794bf9674664b603e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0) 2024-05-23T00:12:30.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:29 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:29.970+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:30.368 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:29 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:29.970+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:30.368 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:29 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:29.903+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:30.369 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:29 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:29.903+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:30.476 INFO:journalctl@ceph.osd.4.smithi105.stdout:May 23 00:12:30 smithi105 podman[73168]: 2024-05-23 00:12:30.113839615 +0000 UTC m=+0.675700558 container died 3c8a0f1c253243b4081f55cfe8871ea77734ae8da3f6a794bf9674664b603e92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-23T00:12:30.476 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:30 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:30.108+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:30.476 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:30 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:30.108+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:30.828 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.4.service' 2024-05-23T00:12:30.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:30 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:30.522+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:30.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:30 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:30.622+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:30.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:30 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:30.622+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:30.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:30 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:30.399+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:30.871 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:12:30.872 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-23T00:12:30.872 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-23T00:12:30.872 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.5 2024-05-23T00:12:31.129 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:30 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:30.960+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:31.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:30 smithi174 systemd[1]: Stopping Ceph osd.5 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:12:31.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:30.999+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:31.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:30.999+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:31.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:31.191+0000 7ff2c8842700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:12:31.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:31.191+0000 7ff2c8842700 -1 osd.5 110 *** Got signal Terminated *** 2024-05-23T00:12:31.194 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:31.191+0000 7ff2c8842700 -1 osd.5 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:12:31.195 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:30 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:30.888+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:31.195 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:30 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:30.888+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:31.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:31 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:31.127+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:31.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:31 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:31.127+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:31.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:31.421+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:31.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:31.492+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:31.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:31.492+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:31.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:31.644+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:31.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:31 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:31.644+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:32.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:31 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:31.969+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:32.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:32 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:32.151+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:32.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:32 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:32.151+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:32.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:31.955+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:32.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:31.955+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:32.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:31.955+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.543606+0000 front 2024-05-23T00:12:10.543640+0000 (oldest deadline 2024-05-23T00:12:31.643859+0000) 2024-05-23T00:12:32.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:31.899+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:32.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:31 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:31.899+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:32.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:32.444+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:32.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:32.444+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:32.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:32.657+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:32.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:32.657+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:32.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:32 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:32.657+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:32.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:32 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:32.402+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:33.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:32 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:32.956+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:33.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:33.117+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:33.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:33.117+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:33.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:32 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:32.929+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:33.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:32 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:32.929+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:33.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:32 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:32.929+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.543606+0000 front 2024-05-23T00:12:10.543640+0000 (oldest deadline 2024-05-23T00:12:31.643859+0000) 2024-05-23T00:12:33.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:32 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:32.932+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:33.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:32 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:32.932+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:33.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:32 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:32.932+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:33.692 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:33.438+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:33.692 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:33.438+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:33.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:33.377+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:34.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:33.689+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:34.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:33.689+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:34.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:33 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:33.689+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:34.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:33.949+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:34.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:33 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:33.949+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:34.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:34 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:34.158+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:34.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:34 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:34.158+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:34.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:34 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:34.158+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:34.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:33.947+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:34.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:33.947+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:34.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:33.947+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.543606+0000 front 2024-05-23T00:12:10.543640+0000 (oldest deadline 2024-05-23T00:12:31.643859+0000) 2024-05-23T00:12:34.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:33.906+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:34.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:33.906+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:34.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:33 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:33.906+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:34.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:34.401+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:34.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:34.401+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:34.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:34.644+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:34.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:34.644+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:34.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:34 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:34.644+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:34.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:34 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:34.389+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:34.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:34 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:34.389+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:35.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:34 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:34.916+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:35.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:34 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:34.916+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:35.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:35 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:35.138+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:35.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:35 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:35.138+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:35.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:35 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:35.138+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:35.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:34 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:34.938+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:35.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:34 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:34.938+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:35.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:34 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:34.938+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:35.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:34 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:34.968+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:35.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:34 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:34.968+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:35.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:34 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:34.968+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.543606+0000 front 2024-05-23T00:12:10.543640+0000 (oldest deadline 2024-05-23T00:12:31.643859+0000) 2024-05-23T00:12:35.633 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:35.381+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:35.634 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:35.381+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:35.634 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:35.630+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:35.634 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:35.630+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:35.634 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:35.630+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:35.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:35 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:35.422+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:35.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:35 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:35.422+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:36.057 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:35 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:35.946+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:36.058 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:35 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:35.946+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:36.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:35 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:35.955+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:36.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:35 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:35.955+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:36.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:35 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:35.955+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:36.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:35 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:35.921+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:52.341875+0000 front 2024-05-23T00:11:52.341905+0000 (oldest deadline 2024-05-23T00:12:17.641591+0000) 2024-05-23T00:12:36.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:35 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:35.921+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.342843+0000 front 2024-05-23T00:11:59.342551+0000 (oldest deadline 2024-05-23T00:12:25.242665+0000) 2024-05-23T00:12:36.367 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:35 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5[48172]: 2024-05-23T00:12:35.921+0000 7ff2c0e2e700 -1 osd.5 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.543606+0000 front 2024-05-23T00:12:10.543640+0000 (oldest deadline 2024-05-23T00:12:31.643859+0000) 2024-05-23T00:12:36.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:36 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:36.163+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:36.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:36 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:36.163+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:36.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:36 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:36.163+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:36.794 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:36 smithi174 podman[73519]: 2024-05-23 00:12:36.532821508 +0000 UTC m=+5.463760848 container died 5f375d34cb079aadfba9d0421617b2f20e85707ab06f4e1c695896ab4f94878d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:12:36.795 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:36 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:36.400+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:36.797 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:36 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:36.400+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:36.811 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:36 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:36.345+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:36.812 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:36 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:36.345+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:36.812 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:36 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:36.587+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:36.812 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:36 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:36.587+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:36.812 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:36 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:36.587+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:37.116 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:36 smithi174 podman[73519]: 2024-05-23 00:12:36.908861363 +0000 UTC m=+5.839800712 container cleanup 5f375d34cb079aadfba9d0421617b2f20e85707ab06f4e1c695896ab4f94878d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:12:37.117 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:36 smithi174 bash[73519]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5 2024-05-23T00:12:37.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:36 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:36.943+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:37.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:36 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:36.943+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:37.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:36 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:36.943+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:37.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:37 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:37.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:37.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:37 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:37.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:37.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:37 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:37.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:37.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:36 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:36.944+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:37.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:36 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:36.944+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:37.533 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:37 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:37.375+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:37.533 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:37 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:37.375+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:37.533 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:37 smithi174 podman[73535]: 2024-05-23 00:12:37.193594958 +0000 UTC m=+0.659586766 container remove 5f375d34cb079aadfba9d0421617b2f20e85707ab06f4e1c695896ab4f94878d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5, RELEASE=squid-2e1f762, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-23T00:12:37.632 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:37 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:37.329+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:37.632 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:37 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:37.329+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:37.798 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:37 smithi174 podman[73621]: 2024-05-23 00:12:37.43449147 +0000 UTC m=+0.017132734 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:12:37.799 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:37 smithi174 podman[73621]: 2024-05-23 00:12:37.552637452 +0000 UTC m=+0.135278716 container create 05e96cd9ff540d74941a342e0a49f2d08607b990fea983372ace1de9e99b70e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762) 2024-05-23T00:12:38.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:37 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:37.629+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:38.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:37 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:37.629+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:38.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:37 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:37.629+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:38.098 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:37 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:37.973+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:38.098 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:37 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:37.973+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:38.098 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:37 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:37.973+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:38.098 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:37 smithi174 podman[73621]: 2024-05-23 00:12:37.797326771 +0000 UTC m=+0.379968042 container init 05e96cd9ff540d74941a342e0a49f2d08607b990fea983372ace1de9e99b70e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS) 2024-05-23T00:12:38.099 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:37 smithi174 podman[73621]: 2024-05-23 00:12:37.800649433 +0000 UTC m=+0.383290698 container start 05e96cd9ff540d74941a342e0a49f2d08607b990fea983372ace1de9e99b70e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:12:38.099 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:37 smithi174 podman[73621]: 2024-05-23 00:12:37.888319011 +0000 UTC m=+0.470960279 container attach 05e96cd9ff540d74941a342e0a49f2d08607b990fea983372ace1de9e99b70e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-23T00:12:38.233 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:37 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:37.909+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:38.233 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:37 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:37.909+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:38.366 INFO:journalctl@ceph.osd.5.smithi174.stdout:May 23 00:12:38 smithi174 podman[73621]: 2024-05-23 00:12:38.097130963 +0000 UTC m=+0.679772230 container died 05e96cd9ff540d74941a342e0a49f2d08607b990fea983372ace1de9e99b70e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-23T00:12:38.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:38 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:38.231+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:38.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:38 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:38.231+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:38.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:38 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:38.231+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:38.578 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:38.314+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:38.579 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:38.314+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:38.651 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:38 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:38.366+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:38.651 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:38 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:38.366+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:38.670 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.5.service' 2024-05-23T00:12:38.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:12:38.709 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-23T00:12:38.709 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-23T00:12:38.709 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.6 2024-05-23T00:12:38.842 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:38.584+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:38.842 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:38.584+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:38.842 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:38 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:38.584+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:39.184 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:38 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:38.924+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:39.184 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:38 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:38.924+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:39.329 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:38 smithi103 systemd[1]: Stopping Ceph osd.6 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:12:39.354 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:38 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:38.951+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:39.354 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:38 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:38.951+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:39.354 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:38 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:38.951+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:39.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:39 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:39.182+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:39.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:39 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:39.182+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:39.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:39 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:39.182+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:39.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:39 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:39.351+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:39.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:39 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:39.351+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:39.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:39.363+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:39.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:39.363+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:39.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:39.373+0000 7fc62dc00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:12:39.830 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:39.373+0000 7fc62dc00700 -1 osd.6 110 *** Got signal Terminated *** 2024-05-23T00:12:39.830 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:39.373+0000 7fc62dc00700 -1 osd.6 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:12:39.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:39.587+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:39.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:39.587+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:39.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:39.587+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:40.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:40.141+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:40.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:40.141+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:40.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:40.141+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:40.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:39 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:39.905+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:40.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:39 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:39.905+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:40.324 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:39 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:39.911+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:40.324 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:39 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:39.911+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:40.324 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:39 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:39.911+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:40.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:40 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:40.321+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:40.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:40 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:40.321+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:40.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:40 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:40.388+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:40.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:40 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:40.388+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:40.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:40 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:40.556+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:40.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:40 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:40.556+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:40.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:40 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:40.556+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:41.190 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:40.906+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:41.191 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:40 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:40.906+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:41.359 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:40 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:40.879+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:41.359 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:40 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:40.879+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:41.359 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:40 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:40.879+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:41.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:41 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:41.189+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:41.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:41 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:41.189+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:41.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:41 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:41.189+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:41.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:41 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:41.356+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:41.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:41 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:41.356+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:41.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:41 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:41.371+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:41.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:41 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:41.371+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:41.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:41 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:41.371+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.548100+0000 front 2024-05-23T00:12:17.547806+0000 (oldest deadline 2024-05-23T00:12:40.447721+0000) 2024-05-23T00:12:41.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:41 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:41.537+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:41.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:41 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:41.537+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:41.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:41 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:41.537+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:41.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:41 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:41.537+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:42.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:42 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:42.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:42.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:42 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:42.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:42.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:42 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:42.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:42.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:41 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:41.926+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:42.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:41 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:41.926+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:42.317 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:41 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:41.865+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:42.317 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:41 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:41.865+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:42.317 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:41 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:41.865+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:42.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:42 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:42.315+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:42.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:42 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:42.315+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:42.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:42 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:42.315+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:42.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:42 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:42.386+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:42.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:42 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:42.386+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:42.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:42 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:42.386+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.548100+0000 front 2024-05-23T00:12:17.547806+0000 (oldest deadline 2024-05-23T00:12:40.447721+0000) 2024-05-23T00:12:42.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:42 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:42.523+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:42.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:42 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:42.523+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:42.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:42 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:42.523+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:42.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:42 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:42.523+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:43.171 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:42 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:42.912+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:43.171 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:42 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:42.912+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:43.286 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:42 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:42.868+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:43.287 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:42 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:42.868+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:43.287 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:42 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:42.868+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:43.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:43 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:43.169+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:43.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:43 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:43.169+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:43.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:43 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:43.169+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:43.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:43 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:43.169+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:43.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:43 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:43.284+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:43.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:43 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:43.284+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:43.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:43 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:43.284+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:43.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:43.350+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.439126+0000 front 2024-05-23T00:12:01.439054+0000 (oldest deadline 2024-05-23T00:12:23.139242+0000) 2024-05-23T00:12:43.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:43.350+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:07.841376+0000 front 2024-05-23T00:12:07.841417+0000 (oldest deadline 2024-05-23T00:12:31.341659+0000) 2024-05-23T00:12:43.830 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6[59607]: 2024-05-23T00:12:43.350+0000 7fc629400700 -1 osd.6 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.548100+0000 front 2024-05-23T00:12:17.547806+0000 (oldest deadline 2024-05-23T00:12:40.447721+0000) 2024-05-23T00:12:43.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:43.548+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:43.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:43.548+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:43.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:43.548+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:43.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:43 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:43.548+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:44.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:43 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:43.825+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:44.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:43 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:43.825+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:44.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:43 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:43.825+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:44.208 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:43 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:43.942+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:44.208 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:43 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:43.942+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:44.208 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:43 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:43.942+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:44.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:44 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:44.206+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:44.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:44 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:44.206+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:44.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:44 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:44.206+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:44.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:44 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:44.206+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:44.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:44 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:44.270+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:44.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:44 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:44.270+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:44.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:44 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:44.270+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:44.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:44 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:44.541+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:44.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:44 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:44.541+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:44.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:44 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:44.541+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:44.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:44 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:44.541+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:45.239 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:44 smithi103 podman[101047]: 2024-05-23 00:12:44.88940447 +0000 UTC m=+5.691179911 container died 1e337aa6ef12841bbd38548fb493bd9f057aa3d6378708cf4ca7debd7db873ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-23T00:12:45.255 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:44 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:44.903+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:45.255 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:44 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:44.903+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:45.256 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:44 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:44.903+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:45.297 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:44 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:44.866+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:45.297 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:44 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:44.866+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:45.297 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:44 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:44.866+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:45.505 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:45 smithi103 podman[101047]: 2024-05-23 00:12:45.35845387 +0000 UTC m=+6.160229342 container cleanup 1e337aa6ef12841bbd38548fb493bd9f057aa3d6378708cf4ca7debd7db873ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521) 2024-05-23T00:12:45.506 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:45 smithi103 bash[101047]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6 2024-05-23T00:12:45.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:45 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:45.254+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:45.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:45 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:45.254+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:45.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:45 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:45.254+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:45.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:45 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:45.254+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:45.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:45 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:45.295+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:45.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:45 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:45.295+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:45.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:45 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:45.295+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:45.829 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:45 smithi103 podman[101065]: 2024-05-23 00:12:45.629901211 +0000 UTC m=+0.724113350 container remove 1e337aa6ef12841bbd38548fb493bd9f057aa3d6378708cf4ca7debd7db873ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-23T00:12:45.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:45 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:45.503+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:45.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:45 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:45.503+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:45.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:45 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:45.503+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:45.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:45 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:45.503+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:46.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:45 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:45.817+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:46.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:45 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:45.817+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:46.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:45 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:45.817+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:46.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:45 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:45.945+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:46.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:45 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:45.945+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:46.237 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:45 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:45.945+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:46.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:46 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:46.234+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:46.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:46 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:46.234+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:46.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:46 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:46.234+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:46.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:46 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:46.234+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:46.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:46 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:46.317+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:46.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:46 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:46.317+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:46.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:46 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:46.317+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:46.784 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:46.489+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:46.784 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:46.489+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:46.784 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:46.489+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:46.784 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:46 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:46.489+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:46.785 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:46 smithi103 podman[101157]: 2024-05-23 00:12:46.36367946 +0000 UTC m=+0.033878048 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:12:46.785 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:46 smithi103 podman[101157]: 2024-05-23 00:12:46.494293252 +0000 UTC m=+0.164491876 container create 17ad418f776b6e494ffc13a1839aa3881d8d2ac20a5d442450b7d621d0855b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-2e1f762) 2024-05-23T00:12:46.785 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:46 smithi103 podman[101157]: 2024-05-23 00:12:46.782206819 +0000 UTC m=+0.452405486 container init 17ad418f776b6e494ffc13a1839aa3881d8d2ac20a5d442450b7d621d0855b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-23T00:12:47.079 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:46 smithi103 podman[101157]: 2024-05-23 00:12:46.79352775 +0000 UTC m=+0.463726345 container start 17ad418f776b6e494ffc13a1839aa3881d8d2ac20a5d442450b7d621d0855b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-23T00:12:47.079 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:46 smithi103 podman[101157]: 2024-05-23 00:12:46.863803268 +0000 UTC m=+0.534001898 container attach 17ad418f776b6e494ffc13a1839aa3881d8d2ac20a5d442450b7d621d0855b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:12:47.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:46 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:46.798+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:47.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:46 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:46.798+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:47.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:46 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:46.798+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:47.209 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:46 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:46.935+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:47.210 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:46 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:46.935+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:47.210 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:46 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:46.935+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:47.556 INFO:journalctl@ceph.osd.6.smithi103.stdout:May 23 00:12:47 smithi103 podman[101157]: 2024-05-23 00:12:47.248653161 +0000 UTC m=+0.918851791 container died 17ad418f776b6e494ffc13a1839aa3881d8d2ac20a5d442450b7d621d0855b29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:12:47.557 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:47 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:47.488+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:47.557 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:47 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:47.488+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:47.557 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:47 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:47.488+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:47.557 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:47 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:47.488+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:47.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:47 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:47.207+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:47.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:47 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:47.207+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:47.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:47 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:47.207+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:47.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:47 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:47.207+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:47.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:47.285+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:47.618 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:47.285+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:47.618 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:47.285+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:48.053 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.6.service' 2024-05-23T00:12:48.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:47.821+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:48.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:47.821+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:48.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:47 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:47.821+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:48.169 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:12:48.169 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-23T00:12:48.170 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-23T00:12:48.170 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.7 2024-05-23T00:12:48.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:47 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:47.967+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:48.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:47 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:47.967+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:48.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:47 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:47.967+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:48.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:48 smithi105 systemd[1]: Stopping Ceph osd.7 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:12:48.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:48.163+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:48.309 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:48.163+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:48.309 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:48.163+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:48.309 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:48.163+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:48.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:48 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:48.313+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:48.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:48 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:48.313+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:48.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:48 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:48.313+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:48.807 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:48.475+0000 7f2890ede700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:12:48.808 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:48.475+0000 7f2890ede700 -1 osd.7 110 *** Got signal Terminated *** 2024-05-23T00:12:48.808 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:48.475+0000 7f2890ede700 -1 osd.7 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:12:48.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:48.439+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:48.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:48.439+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:48.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:48.439+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:48.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:48 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:48.439+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:49.304 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:48 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:48.866+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:49.304 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:48 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:48.866+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:49.305 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:48 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:48.866+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:49.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:49.126+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:49.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:49.126+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:49.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:49.126+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:49.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:49.126+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:49.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:48.965+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:49.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:48.965+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:49.309 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:48 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:48.965+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:49.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:49 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:49.302+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:49.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:49 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:49.302+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:49.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:49 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:49.302+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:49.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:49.429+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:49.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:49.429+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:49.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:49.429+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:49.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:49.429+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:49.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:49 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:49.429+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:50.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:49.973+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:50.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:49.973+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:50.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:49.973+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:50.309 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:49 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:49.973+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.773999+0000 front 2024-05-23T00:12:23.774064+0000 (oldest deadline 2024-05-23T00:12:49.074078+0000) 2024-05-23T00:12:50.309 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:50.146+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:50.309 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:50.146+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:50.310 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:50.146+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:50.310 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:50.146+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:50.310 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:50.146+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:50.320 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:49 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:49.915+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:50.320 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:49 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:49.915+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:50.320 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:49 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:49.915+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:50.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:50 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:50.318+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:50.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:50 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:50.318+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:50.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:50 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:50.318+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:50.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:50 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:50.426+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:50.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:50 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:50.426+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:50.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:50 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:50.426+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:50.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:50 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:50.426+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:50.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:50 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:50.426+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:51.287 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:50 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:50.913+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:51.287 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:50 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:50.913+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:51.287 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:50 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:50.913+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:51.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:50.985+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:51.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:50.985+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:51.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:50.985+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:51.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:50.985+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.773999+0000 front 2024-05-23T00:12:23.774064+0000 (oldest deadline 2024-05-23T00:12:49.074078+0000) 2024-05-23T00:12:51.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:51 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:51.112+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:51.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:51 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:51.112+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:51.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:51 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:51.112+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:51.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:51 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:51.112+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:51.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:51 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:51.112+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:51.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:51.285+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:51.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:51.285+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:51.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:51.285+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:51.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:51.386+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:51.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:51.386+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:51.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:51.386+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:51.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:51.386+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:51.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:51 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:51.386+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:52.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:52.010+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:52.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:52.010+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:52.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:52.010+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:52.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:52.010+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.773999+0000 front 2024-05-23T00:12:23.774064+0000 (oldest deadline 2024-05-23T00:12:49.074078+0000) 2024-05-23T00:12:52.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:52.070+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:52.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:52.070+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:52.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:52.070+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:52.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:52.070+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:52.309 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:52 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:52.070+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:52.329 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:51.936+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:52.329 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:51.936+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:52.329 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:51 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:51.936+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:52.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:52.327+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:52.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:52.327+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:52.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:52.327+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:52.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:52.327+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:12:52.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:52 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:52.435+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:52.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:52 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:52.435+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:52.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:52 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:52.435+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:52.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:52 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:52.435+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:52.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:52 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:52.435+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:53.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:53.000+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:02.369726+0000 front 2024-05-23T00:12:02.369751+0000 (oldest deadline 2024-05-23T00:12:27.069956+0000) 2024-05-23T00:12:53.307 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:53.000+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.471461+0000 front 2024-05-23T00:12:10.471575+0000 (oldest deadline 2024-05-23T00:12:33.371479+0000) 2024-05-23T00:12:53.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:53.000+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.473732+0000 front 2024-05-23T00:12:18.473872+0000 (oldest deadline 2024-05-23T00:12:43.773651+0000) 2024-05-23T00:12:53.308 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7[54330]: 2024-05-23T00:12:53.000+0000 7f28894ca700 -1 osd.7 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.773999+0000 front 2024-05-23T00:12:23.774064+0000 (oldest deadline 2024-05-23T00:12:49.074078+0000) 2024-05-23T00:12:53.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:53.043+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:53.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:53.043+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:53.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:53.043+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:53.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:53.043+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:53.309 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:53 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:53.043+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:53.309 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:52.929+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:53.309 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:52.929+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:53.309 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:52 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:52.929+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:53.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:53.306+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:53.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:53.306+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:53.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:53.306+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:53.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:53.306+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:12:53.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:53.395+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:53.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:53.395+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:53.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:53.395+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:53.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:53.395+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:53.830 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:53 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:53.395+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:54.057 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:53 smithi105 podman[73274]: 2024-05-23 00:12:53.801081656 +0000 UTC m=+5.439848570 container died 37eb69b1735b88a0e521f97d71f2dfeb2b234fe2b059ab29e469f91d67ce9491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-05-23T00:12:54.349 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:53.887+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:54.349 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:53.887+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:54.349 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:53.887+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:54.349 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:53 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:53.887+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:12:54.463 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:54 smithi105 podman[73274]: 2024-05-23 00:12:54.177057534 +0000 UTC m=+5.815824441 container cleanup 37eb69b1735b88a0e521f97d71f2dfeb2b234fe2b059ab29e469f91d67ce9491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-23T00:12:54.463 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:54 smithi105 bash[73274]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7 2024-05-23T00:12:54.464 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:54.087+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:54.464 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:54.087+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:54.464 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:54.087+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:54.464 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:54.087+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:54.464 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:54 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:54.087+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:54.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:54.347+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:54.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:54.347+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:54.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:54.347+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:54.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:54.347+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:12:54.793 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:54 smithi105 podman[73290]: 2024-05-23 00:12:54.461828659 +0000 UTC m=+0.658934420 container remove 37eb69b1735b88a0e521f97d71f2dfeb2b234fe2b059ab29e469f91d67ce9491 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762) 2024-05-23T00:12:54.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:54 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:54.378+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:54.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:54 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:54.378+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:54.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:54 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:54.378+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:54.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:54 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:54.378+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:54.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:54 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:54.378+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:55.057 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:54 smithi105 podman[73380]: 2024-05-23 00:12:54.695697186 +0000 UTC m=+0.018033793 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:12:55.057 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:54 smithi105 podman[73380]: 2024-05-23 00:12:54.795923424 +0000 UTC m=+0.118260011 container create 22b1f3e8f27301181cb70c06ea923f98ed6ac76c90ac4dda6d859be26ac02965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-23T00:12:55.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:54.859+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:55.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:54.859+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:55.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:54.859+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:55.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:54 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:54.859+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:12:55.363 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:55 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:55.114+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:55.364 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:55 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:55.114+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:55.364 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:55 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:55.114+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:55.364 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:55 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:55.114+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:55.364 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:55 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:55.114+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:55.364 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:55 smithi105 podman[73380]: 2024-05-23 00:12:55.073516149 +0000 UTC m=+0.395852743 container init 22b1f3e8f27301181cb70c06ea923f98ed6ac76c90ac4dda6d859be26ac02965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-23T00:12:55.365 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:55 smithi105 podman[73380]: 2024-05-23 00:12:55.076650761 +0000 UTC m=+0.398987356 container start 22b1f3e8f27301181cb70c06ea923f98ed6ac76c90ac4dda6d859be26ac02965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-23T00:12:55.365 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:55 smithi105 podman[73380]: 2024-05-23 00:12:55.156592945 +0000 UTC m=+0.478929553 container attach 22b1f3e8f27301181cb70c06ea923f98ed6ac76c90ac4dda6d859be26ac02965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:12:55.638 INFO:journalctl@ceph.osd.7.smithi105.stdout:May 23 00:12:55 smithi105 podman[73380]: 2024-05-23 00:12:55.362560817 +0000 UTC m=+0.684897422 container died 22b1f3e8f27301181cb70c06ea923f98ed6ac76c90ac4dda6d859be26ac02965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:12:55.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:55 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:55.362+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:55.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:55 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:55.362+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:55.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:55 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:55.362+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:55.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:55 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:55.362+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:55.829 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:55 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:55.362+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:55.863 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:55.384+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:55.863 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:55.384+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:55.863 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:55.384+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:55.863 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:55.384+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:12:55.975 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.7.service' 2024-05-23T00:12:56.013 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:12:56.013 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-23T00:12:56.013 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-23T00:12:56.014 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.8 2024-05-23T00:12:56.116 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:56 smithi174 systemd[1]: Stopping Ceph osd.8 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:12:56.116 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:55.860+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:56.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:55.860+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:56.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:55.860+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:56.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:55 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:55.860+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:12:56.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:56.157+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:56.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:56.157+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:56.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:56.157+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:56.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:56.157+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:56.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:56 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:56.157+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:56.616 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:56.327+0000 7f81cced1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:12:56.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:56.327+0000 7f81cced1700 -1 osd.8 110 *** Got signal Terminated *** 2024-05-23T00:12:56.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:56.327+0000 7f81cced1700 -1 osd.8 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:12:56.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:56.413+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:56.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:56.413+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:56.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:56.413+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:56.617 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:56.413+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:12:56.731 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:56 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:56.409+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:56.731 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:56 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:56.409+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:56.732 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:56 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:56.409+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:56.732 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:56 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:56.409+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:56.732 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:56 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:56.409+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:57.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:56.905+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:57.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:56.905+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:57.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:56.905+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:57.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:56 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:56.905+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:12:57.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:57.139+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:57.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:57.139+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:57.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:57.139+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:57.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:57.139+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:57.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:57.139+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:57.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:57.139+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:12:57.771 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:57 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:57.392+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:57.771 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:57 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:57.392+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:57.771 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:57 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:57.392+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:57.771 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:57 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:57.392+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:57.771 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:57 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:57.392+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:57.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:57.456+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:57.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:57.456+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:57.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:57.456+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:57.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:57.456+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:12:57.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:57.456+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:31.971589+0000 front 2024-05-23T00:12:31.971565+0000 (oldest deadline 2024-05-23T00:12:57.271599+0000) 2024-05-23T00:12:58.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:57.903+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:58.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:57.903+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:58.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:57.903+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:58.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:57.903+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:12:58.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:57 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:57.903+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:12:58.429 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:58 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:58.352+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:58.430 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:58 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:58.352+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:58.430 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:58 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:58.352+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:58.430 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:58 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:58.352+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:58.430 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:58 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:58.352+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:58.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:58.134+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:58.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:58.134+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:58.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:58.134+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:58.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:58.134+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:58.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:58.134+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:58.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:58 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:58.134+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:12:58.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:58.473+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:58.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:58.473+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:58.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:58.473+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:58.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:58.473+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:12:58.868 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:58.473+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:31.971589+0000 front 2024-05-23T00:12:31.971565+0000 (oldest deadline 2024-05-23T00:12:57.271599+0000) 2024-05-23T00:12:59.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:58.913+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:12:59.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:58.913+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:12:59.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:58.913+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:12:59.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:58.913+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:12:59.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:58 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:58.913+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:12:59.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:59.091+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:12:59.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:59.091+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:12:59.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:59.091+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:12:59.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:59.091+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:12:59.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:59.091+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:12:59.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:12:59 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:12:59.091+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:12:59.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:59 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:59.365+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:12:59.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:59 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:59.365+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:12:59.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:59 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:59.365+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:12:59.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:59 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:59.365+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:12:59.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:12:59 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:12:59.365+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:12:59.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:59.490+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:12:59.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:59.490+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:12:59.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:59.490+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:12:59.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:59.490+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:12:59.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:12:59.490+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:31.971589+0000 front 2024-05-23T00:12:31.971565+0000 (oldest deadline 2024-05-23T00:12:57.271599+0000) 2024-05-23T00:13:00.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:00.055+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:00.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:00.055+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:00.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:00.055+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:00.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:00.055+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:00.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:00.055+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:00.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:00 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:00.055+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:00.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:59.948+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:00.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:59.948+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:00.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:59.948+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:00.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:59.948+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:00.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:12:59 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:12:59.948+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:00.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:00.317+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:00.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:00.317+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:00.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:00.317+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:00.581 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:00.317+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:00.581 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:00 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:00.317+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:00.866 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:13:00.462+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:00.565867+0000 front 2024-05-23T00:12:00.566003+0000 (oldest deadline 2024-05-23T00:12:25.865864+0000) 2024-05-23T00:13:00.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:13:00.462+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.467364+0000 front 2024-05-23T00:12:10.467403+0000 (oldest deadline 2024-05-23T00:12:33.967584+0000) 2024-05-23T00:13:00.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:13:00.462+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:18.068871+0000 front 2024-05-23T00:12:18.069017+0000 (oldest deadline 2024-05-23T00:12:41.568720+0000) 2024-05-23T00:13:00.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:13:00.462+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:26.071254+0000 front 2024-05-23T00:12:26.071178+0000 (oldest deadline 2024-05-23T00:12:51.971162+0000) 2024-05-23T00:13:00.867 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8[54675]: 2024-05-23T00:13:00.462+0000 7f81c94c5700 -1 osd.8 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:31.971589+0000 front 2024-05-23T00:12:31.971565+0000 (oldest deadline 2024-05-23T00:12:57.271599+0000) 2024-05-23T00:13:01.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:01 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:01.012+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:01.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:01 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:01.012+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:01.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:01 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:01.012+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:01.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:01 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:01.012+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:01.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:01 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:01.012+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:01.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:01 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:01.012+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:01.370 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:00.972+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:01.370 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:00.972+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:01.370 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:00.972+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:01.370 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:00.972+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:01.370 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:00 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:00.972+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:01.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:01.308+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:01.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:01.308+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:01.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:01.308+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:01.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:01.308+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:01.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:01 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:01.308+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:01.930 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:01 smithi174 podman[73738]: 2024-05-23 00:13:01.875183185 +0000 UTC m=+5.675016157 container died 42ef6b29fbd02b122bec10289f74bca2525698ab9f36faf7a8aa06c58bb29071 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-23T00:13:02.237 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:01 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:01.928+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:02.237 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:01 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:01.928+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:02.238 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:01 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:01.928+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:02.238 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:01 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:01.928+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:02.238 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:01 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:01.928+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:02.307 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:02 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:02.029+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:02.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:02 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:02.029+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:02.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:02 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:02.029+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:02.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:02 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:02.029+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:02.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:02 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:02.029+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:02.308 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:02 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:02.029+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:02.521 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:02 smithi174 podman[73738]: 2024-05-23 00:13:02.236168768 +0000 UTC m=+6.036001745 container cleanup 42ef6b29fbd02b122bec10289f74bca2525698ab9f36faf7a8aa06c58bb29071 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8, RELEASE=squid-2e1f762, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:13:02.522 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:02 smithi174 bash[73738]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8 2024-05-23T00:13:02.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:02 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:02.272+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:02.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:02 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:02.272+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:02.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:02 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:02.272+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:02.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:02 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:02.272+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:02.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:02 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:02.272+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:02.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:02 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:02.272+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.215334+0000 front 2024-05-23T00:12:35.215435+0000 (oldest deadline 2024-05-23T00:13:01.616542+0000) 2024-05-23T00:13:02.835 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:02 smithi174 podman[73758]: 2024-05-23 00:13:02.520425685 +0000 UTC m=+0.646636470 container remove 42ef6b29fbd02b122bec10289f74bca2525698ab9f36faf7a8aa06c58bb29071 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-23T00:13:03.116 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:02 smithi174 podman[73844]: 2024-05-23 00:13:02.736671689 +0000 UTC m=+0.016913767 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:13:03.117 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:02 smithi174 podman[73844]: 2024-05-23 00:13:02.871477244 +0000 UTC m=+0.151719301 container create 1c80a8e1081ad9f7cb7897c11b3186d6132eefc65b136b5c7410ef08e2dcac6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-23T00:13:03.117 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:03 smithi174 podman[73844]: 2024-05-23 00:13:03.115430616 +0000 UTC m=+0.395672679 container init 1c80a8e1081ad9f7cb7897c11b3186d6132eefc65b136b5c7410ef08e2dcac6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-23T00:13:03.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:02 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:02.921+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:03.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:02 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:02.921+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:03.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:02 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:02.921+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:03.117 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:02 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:02.921+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:03.118 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:02 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:02.921+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:03.409 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:03 smithi174 podman[73844]: 2024-05-23 00:13:03.11862915 +0000 UTC m=+0.398871214 container start 1c80a8e1081ad9f7cb7897c11b3186d6132eefc65b136b5c7410ef08e2dcac6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-23T00:13:03.409 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:03 smithi174 podman[73844]: 2024-05-23 00:13:03.207108464 +0000 UTC m=+0.487350530 container attach 1c80a8e1081ad9f7cb7897c11b3186d6132eefc65b136b5c7410ef08e2dcac6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-05-23T00:13:03.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:03 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:03.056+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:03.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:03 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:03.056+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:03.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:03 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:03.056+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:03.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:03 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:03.056+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:03.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:03 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:03.056+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:03.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:03 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:03.056+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:03.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:03 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:03.238+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:03.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:03 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:03.238+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:03.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:03 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:03.238+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:03.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:03 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:03.238+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:03.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:03 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:03.238+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:03.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:03 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:03.238+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.215334+0000 front 2024-05-23T00:12:35.215435+0000 (oldest deadline 2024-05-23T00:13:01.616542+0000) 2024-05-23T00:13:03.694 INFO:journalctl@ceph.osd.8.smithi174.stdout:May 23 00:13:03 smithi174 podman[73844]: 2024-05-23 00:13:03.408258506 +0000 UTC m=+0.688500580 container died 1c80a8e1081ad9f7cb7897c11b3186d6132eefc65b136b5c7410ef08e2dcac6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-23T00:13:04.048 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.8.service' 2024-05-23T00:13:04.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:04.091 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-23T00:13:04.092 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-23T00:13:04.092 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.9 2024-05-23T00:13:04.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:03 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:03.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:04.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:03 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:03.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:04.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:03 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:03.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:04.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:03 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:03.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:04.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:03 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:03.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:04.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:04 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:04.092+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:04.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:04 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:04.092+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:04.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:04 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:04.092+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:04.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:04 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:04.092+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:04.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:04 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:04.092+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:04.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:04 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:04.092+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:04.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 systemd[1]: Stopping Ceph osd.9 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:04.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.210+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:04.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.210+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:04.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.210+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:04.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.210+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:04.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.210+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:04.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.210+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.215334+0000 front 2024-05-23T00:12:35.215435+0000 (oldest deadline 2024-05-23T00:13:01.616542+0000) 2024-05-23T00:13:05.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.730+0000 7f8c14a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:05.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.730+0000 7f8c14a00700 -1 osd.9 110 *** Got signal Terminated *** 2024-05-23T00:13:05.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:04 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:04.730+0000 7f8c14a00700 -1 osd.9 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:13:05.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:04 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:04.954+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:05.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:04 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:04.954+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:05.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:04 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:04.954+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:05.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:04 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:04.954+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:05.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:04 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:04.954+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:05.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:05 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:05.142+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:05.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:05 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:05.142+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:05.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:05 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:05.142+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:05.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:05 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:05.142+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:05.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:05 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:05.142+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:05.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:05 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:05.142+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:05.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:05.259+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:05.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:05.259+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:05.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:05.259+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:05.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:05.259+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:05.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:05.259+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:05.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:05 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:05.259+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.215334+0000 front 2024-05-23T00:12:35.215435+0000 (oldest deadline 2024-05-23T00:13:01.616542+0000) 2024-05-23T00:13:06.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:05.961+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:06.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:05.961+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:06.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:05.961+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:06.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:05.961+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:06.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:05.961+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:06.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:05 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:05.961+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:06.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:06 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:06.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:06.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:06 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:06.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:06.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:06 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:06.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:06.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:06 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:06.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:06.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:06 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:06.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:06.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:06 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:06.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:06.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:06 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:06.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:06.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:06 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:06.268+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:06.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:06 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:06.268+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:06.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:06 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:06.268+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:06.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:06 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:06.268+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:06.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:06 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:06.268+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:06.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:06 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:06.268+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.215334+0000 front 2024-05-23T00:12:35.215435+0000 (oldest deadline 2024-05-23T00:13:01.616542+0000) 2024-05-23T00:13:07.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:06 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:06.931+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:07.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:06 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:06.931+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:07.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:06 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:06.931+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:07.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:06 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:06.931+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:07.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:06 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:06.931+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:07.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:06 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:06.931+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:07.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:07.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:07.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:07.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:07.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:07.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:07.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:07.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:07.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:07.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:07.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:07.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:07.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:07 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:07.140+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:07.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:07 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:07.287+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:07.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:07 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:07.287+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:07.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:07 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:07.287+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:07.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:07 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:07.287+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:07.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:07 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:07.287+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:07.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:07 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:07.287+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.215334+0000 front 2024-05-23T00:12:35.215435+0000 (oldest deadline 2024-05-23T00:13:01.616542+0000) 2024-05-23T00:13:08.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:07 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:07.952+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:08.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:07 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:07.952+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:08.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:07 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:07.952+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:08.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:07 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:07.952+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:08.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:07 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:07.952+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:08.368 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:07 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:07.952+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:08.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:08 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:08.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:08.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:08 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:08.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:08.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:08 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:08.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:08.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:08 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:08.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:08.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:08 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:08.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:08.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:08 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:08.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:08.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:08 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:08.155+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:08.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:08 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:08.278+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:08.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:08 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:08.278+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:08.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:08 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:08.278+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:08.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:08 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:08.278+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:08.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:08 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:08.278+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:08.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:08 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:08.278+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.215334+0000 front 2024-05-23T00:12:35.215435+0000 (oldest deadline 2024-05-23T00:13:01.616542+0000) 2024-05-23T00:13:09.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:08 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:08.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:09.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:08 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:08.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:09.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:08 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:08.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:09.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:08 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:08.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:09.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:08 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:08.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:09.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:08 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:08.968+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:09.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:09 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:09.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:09.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:09 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:09.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:09.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:09 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:09.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:09.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:09 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:09.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:09.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:09 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:09.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:09.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:09 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:09.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:09.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:09 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:09.188+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:09.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:09 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:09.294+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:49.202946+0000 front 2024-05-23T00:11:49.202467+0000 (oldest deadline 2024-05-23T00:12:15.102591+0000) 2024-05-23T00:13:09.579 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:09 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:09.294+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:59.204210+0000 front 2024-05-23T00:11:59.204393+0000 (oldest deadline 2024-05-23T00:12:23.904312+0000) 2024-05-23T00:13:09.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:09 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:09.294+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.207640+0000 front 2024-05-23T00:12:10.207562+0000 (oldest deadline 2024-05-23T00:12:31.907535+0000) 2024-05-23T00:13:09.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:09 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:09.294+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.209421+0000 front 2024-05-23T00:12:17.210002+0000 (oldest deadline 2024-05-23T00:12:40.709337+0000) 2024-05-23T00:13:09.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:09 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:09.294+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:23.011185+0000 front 2024-05-23T00:12:23.011046+0000 (oldest deadline 2024-05-23T00:12:48.911005+0000) 2024-05-23T00:13:09.580 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:09 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9[66435]: 2024-05-23T00:13:09.295+0000 7f8c0b200700 -1 osd.9 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.215334+0000 front 2024-05-23T00:12:35.215435+0000 (oldest deadline 2024-05-23T00:13:01.616542+0000) 2024-05-23T00:13:10.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:09 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:09.992+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:10.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:09 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:09.992+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:10.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:09 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:09.992+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:10.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:09 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:09.992+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:10.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:09 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:09.992+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:10.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:09 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:09.992+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:10.547 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:10 smithi103 podman[101271]: 2024-05-23 00:13:10.270421503 +0000 UTC m=+5.752381879 container died 7febc895a62d4958f72787199d24657f6e3b1878e393662b1d52fb8254df0538 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:13:10.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:10 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:10.147+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:10.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:10 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:10.147+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:10.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:10 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:10.147+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:10.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:10 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:10.147+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:10.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:10 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:10.147+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:10.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:10 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:10.147+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:10.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:10 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:10.147+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:10.825 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:10 smithi103 podman[101271]: 2024-05-23 00:13:10.662224111 +0000 UTC m=+6.144184507 container cleanup 7febc895a62d4958f72787199d24657f6e3b1878e393662b1d52fb8254df0538 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-23T00:13:10.825 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:10 smithi103 bash[101271]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9 2024-05-23T00:13:11.079 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:10 smithi103 podman[101290]: 2024-05-23 00:13:10.822000498 +0000 UTC m=+0.536171108 container remove 7febc895a62d4958f72787199d24657f6e3b1878e393662b1d52fb8254df0538 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-2e1f762, GIT_CLEAN=True, org.label-schema.build-date=20240521) 2024-05-23T00:13:11.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:10 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:10.997+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:11.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:10 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:10.997+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:11.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:10 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:10.997+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:11.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:10 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:10.997+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:11.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:10 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:10.997+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:11.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:10 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:10.997+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:11.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:11 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:11.181+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:11.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:11 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:11.181+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:11.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:11 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:11.181+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:11.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:11 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:11.181+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:11.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:11 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:11.181+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:11.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:11 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:11.181+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:11.559 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:11 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:11.181+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:11.937 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:11 smithi103 podman[101373]: 2024-05-23 00:13:11.531051548 +0000 UTC m=+0.038288935 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:13:11.937 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:11 smithi103 podman[101373]: 2024-05-23 00:13:11.631307205 +0000 UTC m=+0.138544621 container create 0a310a1e12aa77161eb49298a5153a12a39173e59d457fdb9f8cd18b7ca99aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:13:11.938 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:11 smithi103 podman[101373]: 2024-05-23 00:13:11.934916736 +0000 UTC m=+0.442154203 container init 0a310a1e12aa77161eb49298a5153a12a39173e59d457fdb9f8cd18b7ca99aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=) 2024-05-23T00:13:12.329 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:11 smithi103 podman[101373]: 2024-05-23 00:13:11.945811451 +0000 UTC m=+0.453048846 container start 0a310a1e12aa77161eb49298a5153a12a39173e59d457fdb9f8cd18b7ca99aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-23T00:13:12.330 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:12 smithi103 podman[101373]: 2024-05-23 00:13:12.025378648 +0000 UTC m=+0.532616118 container attach 0a310a1e12aa77161eb49298a5153a12a39173e59d457fdb9f8cd18b7ca99aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:13:12.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:11 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:11.959+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:12.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:11 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:11.959+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:12.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:11 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:11.959+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:12.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:11 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:11.959+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:12.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:11 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:11.959+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:12.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:11 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:11.959+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:12.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:12.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:12.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:12.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:12.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:12.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:12.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:12.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:12.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:12.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:12.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:12.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:12.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:12 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:12.215+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:12.694 INFO:journalctl@ceph.osd.9.smithi103.stdout:May 23 00:13:12 smithi103 podman[101373]: 2024-05-23 00:13:12.40487606 +0000 UTC m=+0.912113483 container died 0a310a1e12aa77161eb49298a5153a12a39173e59d457fdb9f8cd18b7ca99aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-23T00:13:13.055 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.9.service' 2024-05-23T00:13:13.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:13.146 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-23T00:13:13.146 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-23T00:13:13.147 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.10 2024-05-23T00:13:13.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:12 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:12.982+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:13.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:12 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:12.982+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:13.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:12 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:12.982+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:13.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:12 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:12.982+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:13.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:12 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:12.982+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:13.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:12 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:12.982+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:13.483 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 systemd[1]: Stopping Ceph osd.10 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:13.483 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.236+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:13.483 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.236+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:13.483 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.236+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:13.484 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.236+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:13.484 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.236+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:13.484 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.236+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:13.484 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.236+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:13.484 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.480+0000 7f337e31e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:13.485 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.480+0000 7f337e31e700 -1 osd.10 110 *** Got signal Terminated *** 2024-05-23T00:13:13.485 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:13 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:13.480+0000 7f337e31e700 -1 osd.10 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:13:14.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.028+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:14.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.028+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:14.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.028+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:14.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.028+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:14.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.028+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:14.368 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.028+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:14.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:14 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:14.285+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:14.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:14 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:14.285+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:14.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:14 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:14.285+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:14.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:14 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:14.285+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:14.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:14 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:14.285+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:14.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:14 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:14.285+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:14.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:14 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:14.285+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:15.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.996+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:15.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.996+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:15.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.996+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:15.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.996+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:15.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.996+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:15.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:14 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:14.996+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:15.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:15 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:15.276+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:15.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:15 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:15.276+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:15.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:15 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:15.276+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:15.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:15 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:15.277+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:15.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:15 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:15.277+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:15.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:15 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:15.277+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:15.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:15 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:15.277+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:16.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:15.988+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:16.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:15.988+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:16.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:15.988+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:16.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:15.988+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:16.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:15.988+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:16.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:15.988+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:16.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:15.988+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:16.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:16 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:16.232+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:16.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:16 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:16.232+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:16.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:16 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:16.232+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:16.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:16 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:16.232+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:16.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:16 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:16.232+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:16.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:16 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:16.232+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:16.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:16 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:16.232+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:17.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:16 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:16.974+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:17.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:16 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:16.974+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:17.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:16 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:16.974+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:17.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:16 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:16.974+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:17.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:16 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:16.974+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:17.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:16 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:16.974+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:17.368 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:16 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:16.974+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:17.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:17 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:17.247+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:17.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:17 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:17.247+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:17.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:17 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:17.247+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:17.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:17 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:17.247+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:17.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:17 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:17.247+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:17.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:17 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:17.247+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:17.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:17 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:17.247+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:18.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:18.003+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:18.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:18.003+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:18.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:18.003+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:18.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:18.003+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:18.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:18.003+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:18.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:18.003+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:18.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:18 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:18.003+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:18.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:18 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:18.199+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:51.300117+0000 front 2024-05-23T00:11:51.300048+0000 (oldest deadline 2024-05-23T00:12:15.999676+0000) 2024-05-23T00:13:18.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:18 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:18.199+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:12:01.900554+0000 front 2024-05-23T00:12:01.900534+0000 (oldest deadline 2024-05-23T00:12:23.600666+0000) 2024-05-23T00:13:18.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:18 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:18.199+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:10.002005+0000 front 2024-05-23T00:12:10.002138+0000 (oldest deadline 2024-05-23T00:12:34.101975+0000) 2024-05-23T00:13:18.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:18 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:18.199+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6814 osd.3 since back 2024-05-23T00:12:17.603236+0000 front 2024-05-23T00:12:17.603134+0000 (oldest deadline 2024-05-23T00:12:42.902866+0000) 2024-05-23T00:13:18.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:18 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:18.199+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.504157+0000 front 2024-05-23T00:12:27.504201+0000 (oldest deadline 2024-05-23T00:12:49.204282+0000) 2024-05-23T00:13:18.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:18 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:18.199+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:35.005958+0000 front 2024-05-23T00:12:35.005872+0000 (oldest deadline 2024-05-23T00:12:56.705765+0000) 2024-05-23T00:13:18.558 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:18 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10[61356]: 2024-05-23T00:13:18.199+0000 7f337690a700 -1 osd.10 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:41.407103+0000 front 2024-05-23T00:12:41.407018+0000 (oldest deadline 2024-05-23T00:13:06.106635+0000) 2024-05-23T00:13:19.239 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:18 smithi105 podman[73498]: 2024-05-23 00:13:18.959598588 +0000 UTC m=+5.617827597 container died ddb2ee1f63daed4d66b4effef6188d4ca0584e4b0353e1ee7c2f1e51f8f23f03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762) 2024-05-23T00:13:19.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:19.043+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:19.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:19.043+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:19.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:19.043+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:19.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:19.043+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:19.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:19.043+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:19.368 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:19.043+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:19.368 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:19.043+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:19.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:19 smithi105 podman[73498]: 2024-05-23 00:13:19.337433775 +0000 UTC m=+5.995662775 container cleanup ddb2ee1f63daed4d66b4effef6188d4ca0584e4b0353e1ee7c2f1e51f8f23f03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2) 2024-05-23T00:13:19.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:19 smithi105 bash[73498]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10 2024-05-23T00:13:19.956 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:19 smithi105 podman[73515]: 2024-05-23 00:13:19.588402962 +0000 UTC m=+0.627260660 container remove ddb2ee1f63daed4d66b4effef6188d4ca0584e4b0353e1ee7c2f1e51f8f23f03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-23T00:13:20.251 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:19 smithi105 podman[73601]: 2024-05-23 00:13:19.857301717 +0000 UTC m=+0.016921376 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:13:20.251 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:19 smithi105 podman[73601]: 2024-05-23 00:13:19.964146246 +0000 UTC m=+0.123765951 container create cd73ff3d3c12bfb3b8113c1245d5da72039a55a4081975f78006ec71340f38a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-05-23T00:13:20.251 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:20 smithi105 podman[73601]: 2024-05-23 00:13:20.166654813 +0000 UTC m=+0.326274472 container init cd73ff3d3c12bfb3b8113c1245d5da72039a55a4081975f78006ec71340f38a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-05-23T00:13:20.251 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:20 smithi105 podman[73601]: 2024-05-23 00:13:20.169742442 +0000 UTC m=+0.329362102 container start cd73ff3d3c12bfb3b8113c1245d5da72039a55a4081975f78006ec71340f38a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5) 2024-05-23T00:13:20.252 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:20 smithi105 podman[73601]: 2024-05-23 00:13:20.249850277 +0000 UTC m=+0.409469937 container attach cd73ff3d3c12bfb3b8113c1245d5da72039a55a4081975f78006ec71340f38a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS) 2024-05-23T00:13:20.366 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:20.093+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:20.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:20.093+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:20.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:20.093+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:20.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:20.093+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:20.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:20.093+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:20.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:20.093+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:20.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:20 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:20.093+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:20.557 INFO:journalctl@ceph.osd.10.smithi105.stdout:May 23 00:13:20 smithi105 podman[73601]: 2024-05-23 00:13:20.456900755 +0000 UTC m=+0.616520415 container died cd73ff3d3c12bfb3b8113c1245d5da72039a55a4081975f78006ec71340f38a6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521) 2024-05-23T00:13:21.091 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.10.service' 2024-05-23T00:13:21.130 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:21.131 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-23T00:13:21.131 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-23T00:13:21.131 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.11 2024-05-23T00:13:21.431 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.142+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:21.431 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.142+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:21.431 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.142+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:21.431 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.142+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:21.431 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.142+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:21.432 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.142+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:21.432 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.142+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:21.432 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 systemd[1]: Stopping Ceph osd.11 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:21.432 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.428+0000 7fea2df58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:21.432 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.428+0000 7fea2df58700 -1 osd.11 110 *** Got signal Terminated *** 2024-05-23T00:13:21.432 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:21 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:21.428+0000 7fea2df58700 -1 osd.11 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-23T00:13:22.616 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:22.174+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:22.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:22.174+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:22.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:22.174+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:22.618 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:22.174+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:22.618 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:22.174+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:22.619 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:22.174+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:22.619 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:22 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:22.174+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:23.616 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:23.216+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:23.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:23.216+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:23.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:23.216+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:23.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:23.216+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:23.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:23.216+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:23.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:23.216+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:23.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:23 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:23.216+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:24.518 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:24.238+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:24.518 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:24.238+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:24.518 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:24.238+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:24.518 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:24.238+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:24.519 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:24.238+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:24.519 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:24.238+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:24.519 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:24 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:24.238+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:25.533 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:25.189+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:25.533 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:25.189+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:25.534 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:25.189+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:25.534 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:25.189+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:25.534 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:25.189+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:25.534 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:25.189+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:25.534 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:25.189+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:25.534 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:25 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:25.189+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6820 osd.8 since back 2024-05-23T00:12:59.747812+0000 front 2024-05-23T00:12:59.747785+0000 (oldest deadline 2024-05-23T00:13:25.047793+0000) 2024-05-23T00:13:26.616 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:26.231+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6806 osd.0 since back 2024-05-23T00:11:47.437978+0000 front 2024-05-23T00:11:47.437462+0000 (oldest deadline 2024-05-23T00:12:18.037943+0000) 2024-05-23T00:13:26.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:26.231+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2024-05-23T00:11:58.038500+0000 front 2024-05-23T00:11:58.038401+0000 (oldest deadline 2024-05-23T00:12:25.038965+0000) 2024-05-23T00:13:26.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:26.231+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6804 osd.2 since back 2024-05-23T00:12:09.139840+0000 front 2024-05-23T00:12:09.139979+0000 (oldest deadline 2024-05-23T00:12:32.540523+0000) 2024-05-23T00:13:26.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:26.231+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-05-23T00:12:27.643728+0000 front 2024-05-23T00:12:27.643745+0000 (oldest deadline 2024-05-23T00:12:52.943531+0000) 2024-05-23T00:13:26.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:26.231+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6812 osd.5 since back 2024-05-23T00:12:33.444401+0000 front 2024-05-23T00:12:33.444447+0000 (oldest deadline 2024-05-23T00:12:56.944498+0000) 2024-05-23T00:13:26.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:26.231+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.103:6822 osd.6 since back 2024-05-23T00:12:39.846181+0000 front 2024-05-23T00:12:39.845938+0000 (oldest deadline 2024-05-23T00:13:05.745732+0000) 2024-05-23T00:13:26.617 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:26.231+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.105:6820 osd.7 since back 2024-05-23T00:12:51.046933+0000 front 2024-05-23T00:12:51.046895+0000 (oldest deadline 2024-05-23T00:13:15.146811+0000) 2024-05-23T00:13:26.618 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11[61700]: 2024-05-23T00:13:26.231+0000 7fea2a54c700 -1 osd.11 110 heartbeat_check: no reply from 172.21.15.174:6820 osd.8 since back 2024-05-23T00:12:59.747812+0000 front 2024-05-23T00:12:59.747785+0000 (oldest deadline 2024-05-23T00:13:25.047793+0000) 2024-05-23T00:13:27.057 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:26 smithi174 podman[73949]: 2024-05-23 00:13:26.777607204 +0000 UTC m=+5.456326042 container died 04e2fd0ef8cccb4c59f5b905b0a4e321626846a759b202fe843e1f4b6c4cc9c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-23T00:13:27.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:27 smithi174 podman[73949]: 2024-05-23 00:13:27.153883954 +0000 UTC m=+5.832602813 container cleanup 04e2fd0ef8cccb4c59f5b905b0a4e321626846a759b202fe843e1f4b6c4cc9c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:13:27.367 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:27 smithi174 bash[73949]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11 2024-05-23T00:13:27.705 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:27 smithi174 podman[73977]: 2024-05-23 00:13:27.388532948 +0000 UTC m=+0.610355072 container remove 04e2fd0ef8cccb4c59f5b905b0a4e321626846a759b202fe843e1f4b6c4cc9c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , ceph=True) 2024-05-23T00:13:27.969 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:27 smithi174 podman[74064]: 2024-05-23 00:13:27.60627857 +0000 UTC m=+0.017107700 image pull f0ae8441b76cd7a519a0186543d4b03eb8a47168c7797f2f5a8383edb0721f41 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036 2024-05-23T00:13:27.970 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:27 smithi174 podman[74064]: 2024-05-23 00:13:27.722441632 +0000 UTC m=+0.133270736 container create 973d15167527ab2d59f83bf16de5849b3747a6647d856fd2cdec0fe48d76cde5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-23T00:13:27.970 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:27 smithi174 podman[74064]: 2024-05-23 00:13:27.966162619 +0000 UTC m=+0.376991726 container init 973d15167527ab2d59f83bf16de5849b3747a6647d856fd2cdec0fe48d76cde5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-05-23T00:13:28.256 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:27 smithi174 podman[74064]: 2024-05-23 00:13:27.969437258 +0000 UTC m=+0.380266371 container start 973d15167527ab2d59f83bf16de5849b3747a6647d856fd2cdec0fe48d76cde5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, ceph=True) 2024-05-23T00:13:28.256 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:28 smithi174 podman[74064]: 2024-05-23 00:13:28.032717458 +0000 UTC m=+0.443546574 container attach 973d15167527ab2d59f83bf16de5849b3747a6647d856fd2cdec0fe48d76cde5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11-deactivate, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-23T00:13:28.528 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:28 smithi174 podman[74064]: 2024-05-23 00:13:28.255080061 +0000 UTC m=+0.665909182 container died 973d15167527ab2d59f83bf16de5849b3747a6647d856fd2cdec0fe48d76cde5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11-deactivate, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:13:28.771 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.11.service' 2024-05-23T00:13:28.794 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:28 smithi174 podman[74090]: 2024-05-23 00:13:28.743913841 +0000 UTC m=+0.483701335 container remove 973d15167527ab2d59f83bf16de5849b3747a6647d856fd2cdec0fe48d76cde5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-23T00:13:28.795 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:28 smithi174 systemd[1]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.11.service: Deactivated successfully. 2024-05-23T00:13:28.795 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:28 smithi174 systemd[1]: Stopped Ceph osd.11 for 9c0f3dc8-1895-11ef-bc9a-c7b262605968. 2024-05-23T00:13:28.795 INFO:journalctl@ceph.osd.11.smithi174.stdout:May 23 00:13:28 smithi174 systemd[1]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@osd.11.service: Consumed 17.048s CPU time. 2024-05-23T00:13:28.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:28.810 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-23T00:13:28.811 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-23T00:13:28.811 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.a 2024-05-23T00:13:29.328 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:13:28 smithi103 systemd[1]: Stopping Ceph mds.a for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:29.829 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:13:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a[73394]: 2024-05-23T00:13:29.429+0000 7fd63ea00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:29.829 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:13:29 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a[73394]: 2024-05-23T00:13:29.429+0000 7fd63ea00700 -1 mds.a *** got signal Terminated *** 2024-05-23T00:13:33.789 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:13:33 smithi103 podman[101490]: 2024-05-23 00:13:33.445083 +0000 UTC m=+4.170465563 container died 4cf31a575965abee3965adfbd8e5ddd0d4ca6f95e433f6a490c4d371a4c3c560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-23T00:13:34.079 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:13:33 smithi103 podman[101490]: 2024-05-23 00:13:33.906566013 +0000 UTC m=+4.631948588 container cleanup 4cf31a575965abee3965adfbd8e5ddd0d4ca6f95e433f6a490c4d371a4c3c560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:13:34.079 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:13:33 smithi103 bash[101490]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a 2024-05-23T00:13:34.516 INFO:journalctl@ceph.mds.a.smithi103.stdout:May 23 00:13:34 smithi103 podman[101502]: 2024-05-23 00:13:34.09672164 +0000 UTC m=+0.633302295 container remove 4cf31a575965abee3965adfbd8e5ddd0d4ca6f95e433f6a490c4d371a4c3c560 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-a, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:13:34.544 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.a.service' 2024-05-23T00:13:34.654 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:34.655 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-23T00:13:34.655 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-23T00:13:34.656 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.d 2024-05-23T00:13:34.828 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 23 00:13:34 smithi103 systemd[1]: Stopping Ceph mds.d for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:35.579 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 23 00:13:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d[72409]: 2024-05-23T00:13:35.208+0000 7fb0b0c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:35.579 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 23 00:13:35 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d[72409]: 2024-05-23T00:13:35.208+0000 7fb0b0c00700 -1 mds.d *** got signal Terminated *** 2024-05-23T00:13:37.579 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 23 00:13:37 smithi103 podman[101612]: 2024-05-23 00:13:37.280910679 +0000 UTC m=+2.196898155 container died ae06e094c8ba4b82c8a7cf4390af1b5af491b30f739b641214fdd47f46f36ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d, GIT_BRANCH=HEAD, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-23T00:13:37.972 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 23 00:13:37 smithi103 podman[101612]: 2024-05-23 00:13:37.743668246 +0000 UTC m=+2.659655754 container cleanup ae06e094c8ba4b82c8a7cf4390af1b5af491b30f739b641214fdd47f46f36ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-23T00:13:37.972 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 23 00:13:37 smithi103 bash[101612]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d 2024-05-23T00:13:38.329 INFO:journalctl@ceph.mds.d.smithi103.stdout:May 23 00:13:37 smithi103 podman[101626]: 2024-05-23 00:13:37.969952711 +0000 UTC m=+0.665829425 container remove ae06e094c8ba4b82c8a7cf4390af1b5af491b30f739b641214fdd47f46f36ae5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:13:38.535 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.d.service' 2024-05-23T00:13:38.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:38.636 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-23T00:13:38.637 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-23T00:13:38.637 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.g 2024-05-23T00:13:39.078 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 23 00:13:38 smithi103 systemd[1]: Stopping Ceph mds.g for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:39.579 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 23 00:13:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g[73008]: 2024-05-23T00:13:39.204+0000 7fc232a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:39.579 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 23 00:13:39 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g[73008]: 2024-05-23T00:13:39.204+0000 7fc232a00700 -1 mds.g *** got signal Terminated *** 2024-05-23T00:13:43.546 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 23 00:13:43 smithi103 podman[101734]: 2024-05-23 00:13:43.226147462 +0000 UTC m=+4.191276867 container died df066456361cf869bc6bdeecff7742dfc2b7b8d65ceedfec7471c9fd32c4ec0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-23T00:13:43.829 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 23 00:13:43 smithi103 podman[101734]: 2024-05-23 00:13:43.656200639 +0000 UTC m=+4.621330044 container cleanup df066456361cf869bc6bdeecff7742dfc2b7b8d65ceedfec7471c9fd32c4ec0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-23T00:13:43.829 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 23 00:13:43 smithi103 bash[101734]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g 2024-05-23T00:13:44.329 INFO:journalctl@ceph.mds.g.smithi103.stdout:May 23 00:13:43 smithi103 podman[101749]: 2024-05-23 00:13:43.907640637 +0000 UTC m=+0.669020445 container remove df066456361cf869bc6bdeecff7742dfc2b7b8d65ceedfec7471c9fd32c4ec0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-05-23T00:13:44.493 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.g.service' 2024-05-23T00:13:44.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:44.599 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-23T00:13:44.599 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-23T00:13:44.599 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.j 2024-05-23T00:13:45.078 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 23 00:13:44 smithi103 systemd[1]: Stopping Ceph mds.j for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:45.579 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 23 00:13:45 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j[72026]: 2024-05-23T00:13:45.175+0000 7fe19ac00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:45.580 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 23 00:13:45 smithi103 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j[72026]: 2024-05-23T00:13:45.175+0000 7fe19ac00700 -1 mds.j *** got signal Terminated *** 2024-05-23T00:13:49.292 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 23 00:13:49 smithi103 podman[101854]: 2024-05-23 00:13:49.018363657 +0000 UTC m=+3.955838155 container died 7bdcb80f9abdb26e8fa50f0f678bd1897f56e12109dc7eaeb30d27382b957a97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762) 2024-05-23T00:13:49.579 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 23 00:13:49 smithi103 podman[101854]: 2024-05-23 00:13:49.402646154 +0000 UTC m=+4.340120651 container cleanup 7bdcb80f9abdb26e8fa50f0f678bd1897f56e12109dc7eaeb30d27382b957a97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-2e1f762, CEPH_POINT_RELEASE=) 2024-05-23T00:13:49.579 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 23 00:13:49 smithi103 bash[101854]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j 2024-05-23T00:13:50.078 INFO:journalctl@ceph.mds.j.smithi103.stdout:May 23 00:13:49 smithi103 podman[101866]: 2024-05-23 00:13:49.604580606 +0000 UTC m=+0.569721695 container remove 7bdcb80f9abdb26e8fa50f0f678bd1897f56e12109dc7eaeb30d27382b957a97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-2e1f762, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:13:50.122 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.j.service' 2024-05-23T00:13:50.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:50.229 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-23T00:13:50.229 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-23T00:13:50.229 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.b 2024-05-23T00:13:50.556 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:13:50 smithi105 systemd[1]: Stopping Ceph mds.b for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:50.807 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:13:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b[66976]: 2024-05-23T00:13:50.553+0000 7f5601997700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:50.807 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:13:50 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b[66976]: 2024-05-23T00:13:50.553+0000 7f5601997700 -1 mds.b *** got signal Terminated *** 2024-05-23T00:13:54.306 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:13:54 smithi105 podman[73706]: 2024-05-23 00:13:53.999909656 +0000 UTC m=+3.577956972 container died 76c9c4314656676e5f3e7908ae605ca11b6cd6542e1f0aa9255a02b5298e3c17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b, RELEASE=squid-2e1f762, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-23T00:13:54.558 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:13:54 smithi105 podman[73706]: 2024-05-23 00:13:54.403822631 +0000 UTC m=+3.981869939 container cleanup 76c9c4314656676e5f3e7908ae605ca11b6cd6542e1f0aa9255a02b5298e3c17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-23T00:13:54.559 INFO:journalctl@ceph.mds.b.smithi105.stdout:May 23 00:13:54 smithi105 bash[73706]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-b 2024-05-23T00:13:54.783 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.b.service' 2024-05-23T00:13:54.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:54.818 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-23T00:13:54.818 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-23T00:13:54.818 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.e 2024-05-23T00:13:55.057 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 23 00:13:54 smithi105 systemd[1]: Stopping Ceph mds.e for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:55.557 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 23 00:13:55 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e[66360]: 2024-05-23T00:13:55.157+0000 7f20cc0f6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:55.557 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 23 00:13:55 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e[66360]: 2024-05-23T00:13:55.157+0000 7f20cc0f6700 -1 mds.e *** got signal Terminated *** 2024-05-23T00:13:56.544 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 23 00:13:56 smithi105 podman[73838]: 2024-05-23 00:13:56.271659479 +0000 UTC m=+1.239128496 container died d880e172c126df554eb131ef51eb15a1214a1465a17d6cfaa42f7c819075fdf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-05-23T00:13:56.807 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 23 00:13:56 smithi105 podman[73838]: 2024-05-23 00:13:56.632849846 +0000 UTC m=+1.600318854 container cleanup d880e172c126df554eb131ef51eb15a1214a1465a17d6cfaa42f7c819075fdf2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-23T00:13:56.808 INFO:journalctl@ceph.mds.e.smithi105.stdout:May 23 00:13:56 smithi105 bash[73838]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-e 2024-05-23T00:13:57.079 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.e.service' 2024-05-23T00:13:57.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:13:57.111 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-23T00:13:57.112 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-23T00:13:57.112 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.h 2024-05-23T00:13:57.421 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:13:57 smithi105 systemd[1]: Stopping Ceph mds.h for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:13:57.807 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:13:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h[66663]: 2024-05-23T00:13:57.419+0000 7fb7b8077700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:13:57.807 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:13:57 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h[66663]: 2024-05-23T00:13:57.419+0000 7fb7b8077700 -1 mds.h *** got signal Terminated *** 2024-05-23T00:14:01.731 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:14:01 smithi105 podman[73959]: 2024-05-23 00:14:01.432567423 +0000 UTC m=+4.112148583 container died bdac9b6bbd8f597bf97b66ce6959dcddc1d3f6fdc7e81dafd0e479c49428bf11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS) 2024-05-23T00:14:02.057 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:14:01 smithi105 podman[73959]: 2024-05-23 00:14:01.87872006 +0000 UTC m=+4.558301214 container cleanup bdac9b6bbd8f597bf97b66ce6959dcddc1d3f6fdc7e81dafd0e479c49428bf11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-23T00:14:02.057 INFO:journalctl@ceph.mds.h.smithi105.stdout:May 23 00:14:01 smithi105 bash[73959]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-h 2024-05-23T00:14:02.302 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.h.service' 2024-05-23T00:14:02.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:14:02.334 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-23T00:14:02.334 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-23T00:14:02.335 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.k 2024-05-23T00:14:02.807 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 23 00:14:02 smithi105 systemd[1]: Stopping Ceph mds.k for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:14:02.808 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 23 00:14:02 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-k[66045]: 2024-05-23T00:14:02.656+0000 7fe744701700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:14:02.808 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 23 00:14:02 smithi105 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-k[66045]: 2024-05-23T00:14:02.656+0000 7fe744701700 -1 mds.k *** got signal Terminated *** 2024-05-23T00:14:05.307 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 23 00:14:05 smithi105 podman[74080]: 2024-05-23 00:14:05.043121939 +0000 UTC m=+2.492309181 container died 6b1c153f7aecce36acb5a189fdde5dae5c264129d77004d587cf103f09c435dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-k, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2) 2024-05-23T00:14:05.780 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 23 00:14:05 smithi105 podman[74080]: 2024-05-23 00:14:05.365569709 +0000 UTC m=+2.814756951 container cleanup 6b1c153f7aecce36acb5a189fdde5dae5c264129d77004d587cf103f09c435dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-23T00:14:05.780 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 23 00:14:05 smithi105 bash[74080]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-k 2024-05-23T00:14:05.781 INFO:journalctl@ceph.mds.k.smithi105.stdout:May 23 00:14:05 smithi105 podman[74093]: 2024-05-23 00:14:05.574642396 +0000 UTC m=+0.524752729 container remove 6b1c153f7aecce36acb5a189fdde5dae5c264129d77004d587cf103f09c435dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-k, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-23T00:14:05.796 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.k.service' 2024-05-23T00:14:05.827 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:14:05.828 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-23T00:14:05.828 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-23T00:14:05.828 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.c 2024-05-23T00:14:06.116 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:14:05 smithi174 systemd[1]: Stopping Ceph mds.c for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:14:06.616 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:14:06 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c[67104]: 2024-05-23T00:14:06.159+0000 7f2310052700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:14:06.617 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:14:06 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c[67104]: 2024-05-23T00:14:06.159+0000 7f2310052700 -1 mds.c *** got signal Terminated *** 2024-05-23T00:14:10.366 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:14:10 smithi174 podman[74170]: 2024-05-23 00:14:10.010207414 +0000 UTC m=+3.985032845 container died 376efd3c5777993812fd21a83e819cc253d78012f69b757c2fa4ffe099f9be95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-2e1f762, org.label-schema.license=GPLv2, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-23T00:14:10.761 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:14:10 smithi174 podman[74170]: 2024-05-23 00:14:10.508753081 +0000 UTC m=+4.483578511 container cleanup 376efd3c5777993812fd21a83e819cc253d78012f69b757c2fa4ffe099f9be95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-05-23T00:14:10.761 INFO:journalctl@ceph.mds.c.smithi174.stdout:May 23 00:14:10 smithi174 bash[74170]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-c 2024-05-23T00:14:10.895 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.c.service' 2024-05-23T00:14:10.931 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:14:10.932 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-23T00:14:10.932 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-23T00:14:10.932 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.f 2024-05-23T00:14:11.116 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:14:11 smithi174 systemd[1]: Stopping Ceph mds.f for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:14:11.616 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:14:11 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f[66798]: 2024-05-23T00:14:11.245+0000 7fb1e3782700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:14:11.617 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:14:11 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f[66798]: 2024-05-23T00:14:11.245+0000 7fb1e3782700 -1 mds.f *** got signal Terminated *** 2024-05-23T00:14:14.313 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:14:14 smithi174 podman[74304]: 2024-05-23 00:14:14.004336327 +0000 UTC m=+2.855440722 container died 89db9aeff43f88c11ca6de9d8ad0bfd2ea1615eb54a1db25fabcf14ddf73e14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521) 2024-05-23T00:14:14.313 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:14:14 smithi174 podman[74304]: 2024-05-23 00:14:14.312156675 +0000 UTC m=+3.163261061 container cleanup 89db9aeff43f88c11ca6de9d8ad0bfd2ea1615eb54a1db25fabcf14ddf73e14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=squid-2e1f762, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-05-23T00:14:14.616 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:14:14 smithi174 bash[74304]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f 2024-05-23T00:14:14.616 INFO:journalctl@ceph.mds.f.smithi174.stdout:May 23 00:14:14 smithi174 podman[74316]: 2024-05-23 00:14:14.505004732 +0000 UTC m=+0.498950063 container remove 89db9aeff43f88c11ca6de9d8ad0bfd2ea1615eb54a1db25fabcf14ddf73e14c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=squid-2e1f762, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-23T00:14:14.645 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.f.service' 2024-05-23T00:14:14.677 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:14:14.677 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-23T00:14:14.677 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-23T00:14:14.678 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.i 2024-05-23T00:14:15.017 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:14:14 smithi174 systemd[1]: Stopping Ceph mds.i for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:14:15.366 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:14:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: 2024-05-23T00:14:15.015+0000 7fc38a14c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:14:15.366 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:14:15 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i[67409]: 2024-05-23T00:14:15.015+0000 7fc38a14c700 -1 mds.i *** got signal Terminated *** 2024-05-23T00:14:18.301 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:14:18 smithi174 podman[74424]: 2024-05-23 00:14:18.006297932 +0000 UTC m=+3.103022944 container died da829a48158a3f0e3011bd119371adc1eaa434d069560525260f8b6a5d7c3243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-2e1f762, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-23T00:14:18.616 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:14:18 smithi174 podman[74424]: 2024-05-23 00:14:18.3989146 +0000 UTC m=+3.495639604 container cleanup da829a48158a3f0e3011bd119371adc1eaa434d069560525260f8b6a5d7c3243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i, RELEASE=squid-2e1f762, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-05-23T00:14:18.617 INFO:journalctl@ceph.mds.i.smithi174.stdout:May 23 00:14:18 smithi174 bash[74424]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-i 2024-05-23T00:14:18.793 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.i.service' 2024-05-23T00:14:18.823 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:14:18.824 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-23T00:14:18.824 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-23T00:14:18.824 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.l 2024-05-23T00:14:19.137 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 23 00:14:18 smithi174 systemd[1]: Stopping Ceph mds.l for 9c0f3dc8-1895-11ef-bc9a-c7b262605968... 2024-05-23T00:14:19.616 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 23 00:14:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l[66497]: 2024-05-23T00:14:19.135+0000 7f2052366700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-23T00:14:19.617 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 23 00:14:19 smithi174 ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l[66497]: 2024-05-23T00:14:19.135+0000 7f2052366700 -1 mds.l *** got signal Terminated *** 2024-05-23T00:14:22.312 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 23 00:14:22 smithi174 podman[74545]: 2024-05-23 00:14:22.040505095 +0000 UTC m=+2.990878839 container died 3c1f3b8310c09aeae4813e8efa351b294a4f4954fd6a835a9d2695ad67319396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-23T00:14:22.616 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 23 00:14:22 smithi174 podman[74545]: 2024-05-23 00:14:22.393809694 +0000 UTC m=+3.344183437 container cleanup 3c1f3b8310c09aeae4813e8efa351b294a4f4954fd6a835a9d2695ad67319396 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:d75b7fc6f1a567cd446c157fc5a821daa32aed37a514c93b3d2651e77acb3036, name=ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-2e1f762, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=d4a0f93b06679a5c9148c9e9b367478b2a0d04e0) 2024-05-23T00:14:22.617 INFO:journalctl@ceph.mds.l.smithi174.stdout:May 23 00:14:22 smithi174 bash[74545]: ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968-mds-l 2024-05-23T00:14:22.784 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9c0f3dc8-1895-11ef-bc9a-c7b262605968@mds.l.service' 2024-05-23T00:14:22.814 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-23T00:14:22.815 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-23T00:14:22.815 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 --force --keep-logs 2024-05-23T00:14:23.132 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-23T00:14:28.083 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 --force --keep-logs 2024-05-23T00:14:28.272 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-23T00:14:30.711 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 --force --keep-logs 2024-05-23T00:14:30.902 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-23T00:14:33.209 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-23T00:14:33.300 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-23T00:14:33.332 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-23T00:14:33.357 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-23T00:14:33.360 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi103/crash 2024-05-23T00:14:33.361 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash -- . 2024-05-23T00:14:33.462 INFO:teuthology.orchestra.run.smithi103.stderr:tar: /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash: Cannot open: No such file or directory 2024-05-23T00:14:33.462 INFO:teuthology.orchestra.run.smithi103.stderr:tar: Error is not recoverable: exiting now 2024-05-23T00:14:33.465 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi105/crash 2024-05-23T00:14:33.466 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash -- . 2024-05-23T00:14:33.510 INFO:teuthology.orchestra.run.smithi105.stderr:tar: /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash: Cannot open: No such file or directory 2024-05-23T00:14:33.510 INFO:teuthology.orchestra.run.smithi105.stderr:tar: Error is not recoverable: exiting now 2024-05-23T00:14:33.512 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi174/crash 2024-05-23T00:14:33.513 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash -- . 2024-05-23T00:14:33.547 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/crash: Cannot open: No such file or directory 2024-05-23T00:14:33.547 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-05-23T00:14:33.548 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-23T00:14:33.548 DEBUG:teuthology.orchestra.run.smithi103:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-23T00:14:33.649 INFO:teuthology.orchestra.run.smithi103.stdout:2024-05-23T00:01:47.045829+0000 mds.i (mds.2) 2 : cluster [WRN] Scrub error on inode 0x100000001fb (/volumes/qa/sv_1/eed760b1-8739-432b-b036-087fb73667d4) see mds.i log and `damage ls` output for details 2024-05-23T00:14:33.650 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-05-23T00:14:33.650 INFO:tasks.cephadm:Compressing logs... 2024-05-23T00:14:33.651 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-23T00:14:33.655 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-23T00:14:33.657 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-23T00:14:33.682 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-23T00:14:33.682 INFO:teuthology.orchestra.run.smithi174.stderr:: No such file or directory 2024-05-23T00:14:33.682 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log 2024-05-23T00:14:33.682 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.2.log 2024-05-23T00:14:33.682 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.f.log 2024-05-23T00:14:33.683 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log 2024-05-23T00:14:33.683 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.5.log 2024-05-23T00:14:33.687 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.c.log 2024-05-23T00:14:33.687 INFO:teuthology.orchestra.run.smithi174.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-23T00:14:33.687 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.c.log 2024-05-23T00:14:33.688 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.z.log 2024-05-23T00:14:33.688 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.i.log 2024-05-23T00:14:33.688 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.z.log: 87.1% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log.gz 2024-05-23T00:14:33.688 INFO:teuthology.orchestra.run.smithi174.stderr: 90.4% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log.gz 2024-05-23T00:14:33.689 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-23T00:14:33.689 INFO:teuthology.orchestra.run.smithi105.stderr:: No such file or directory 2024-05-23T00:14:33.690 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log 2024-05-23T00:14:33.690 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log 2024-05-23T00:14:33.690 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.7.log 2024-05-23T00:14:33.690 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.b.log 2024-05-23T00:14:33.690 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.4.log 2024-05-23T00:14:33.690 INFO:teuthology.orchestra.run.smithi105.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-23T00:14:33.690 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.1.log 2024-05-23T00:14:33.691 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.b.log: /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.y.log 2024-05-23T00:14:33.691 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.h.log 2024-05-23T00:14:33.691 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.y.log: 87.1%gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.10.log 2024-05-23T00:14:33.691 INFO:teuthology.orchestra.run.smithi105.stderr: -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log.gz 2024-05-23T00:14:33.691 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.h.log: 90.4% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log.gz 2024-05-23T00:14:33.692 INFO:teuthology.orchestra.run.smithi105.stderr: 92.0% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.y.log.gz 2024-05-23T00:14:33.693 INFO:teuthology.orchestra.run.smithi174.stderr: 92.0% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.z.log.gz 2024-05-23T00:14:33.695 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log 2024-05-23T00:14:33.695 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.11.log 2024-05-23T00:14:33.696 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.8.log 2024-05-23T00:14:33.696 INFO:teuthology.orchestra.run.smithi174.stderr: 83.5% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log.gz 2024-05-23T00:14:33.696 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log 2024-05-23T00:14:33.698 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log 2024-05-23T00:14:33.698 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.e.log 2024-05-23T00:14:33.698 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log 2024-05-23T00:14:33.698 INFO:teuthology.orchestra.run.smithi105.stderr: 83.5% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log.gz 2024-05-23T00:14:33.704 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.e.log: 94.4% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.e.log.gz 2024-05-23T00:14:33.704 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.b.log 2024-05-23T00:14:33.704 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.k.log 2024-05-23T00:14:33.705 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.l.log 2024-05-23T00:14:33.711 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’gzip -5: No such file or directory --verbose 2024-05-23T00:14:33.711 INFO:teuthology.orchestra.run.smithi103.stderr: -- /var/log/ceph/cephadm.log 2024-05-23T00:14:33.712 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log 2024-05-23T00:14:33.713 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log 2024-05-23T00:14:33.714 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.g.log 2024-05-23T00:14:33.715 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.j.log 2024-05-23T00:14:33.716 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.g.log: 87.1% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.log.gz 2024-05-23T00:14:33.716 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.3.log 2024-05-23T00:14:33.717 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log.gz 2024-05-23T00:14:33.718 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.0.log 2024-05-23T00:14:33.719 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.3.log: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-23T00:14:33.719 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.x.log 2024-05-23T00:14:33.720 INFO:teuthology.orchestra.run.smithi103.stderr: 90.3% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.audit.log.gz 2024-05-23T00:14:33.720 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.a.log 2024-05-23T00:14:33.721 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.0.log: 94.5% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.g.log.gz 2024-05-23T00:14:33.722 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.6.log 2024-05-23T00:14:33.723 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log 2024-05-23T00:14:33.724 INFO:teuthology.orchestra.run.smithi103.stderr: 94.4% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.j.log.gz 2024-05-23T00:14:33.724 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.k.log.gz 2024-05-23T00:14:33.725 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.a.log 2024-05-23T00:14:33.726 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log 2024-05-23T00:14:33.726 INFO:teuthology.orchestra.run.smithi103.stderr: 83.9% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph.cephadm.log.gz 2024-05-23T00:14:33.728 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.d.log 2024-05-23T00:14:33.730 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log: /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.9.log 2024-05-23T00:14:33.731 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log: /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log.gz 2024-05-23T00:14:33.732 INFO:teuthology.orchestra.run.smithi174.stderr: 94.5% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.l.log.gz 2024-05-23T00:14:33.734 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-23T00:14:33.745 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/ceph-client.admin.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-23T00:14:33.745 INFO:teuthology.orchestra.run.smithi103.stderr: 90.9% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-volume.log.gz 2024-05-23T00:14:33.745 INFO:teuthology.orchestra.run.smithi103.stderr: 94.5% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.d.log.gz 2024-05-23T00:14:33.904 INFO:teuthology.orchestra.run.smithi174.stderr: 94.5% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.f.log.gz 2024-05-23T00:14:34.078 INFO:teuthology.orchestra.run.smithi174.stderr: 93.7% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.c.log.gz 2024-05-23T00:14:34.125 INFO:teuthology.orchestra.run.smithi103.stderr: 90.1% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mgr.x.log.gz 2024-05-23T00:14:34.198 INFO:teuthology.orchestra.run.smithi174.stderr: 91.8% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.c.log.gz 2024-05-23T00:14:34.206 INFO:teuthology.orchestra.run.smithi105.stderr: 92.0% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.b.log.gz 2024-05-23T00:14:34.566 INFO:teuthology.orchestra.run.smithi103.stderr: 94.3% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.a.log.gz 2024-05-23T00:14:35.065 INFO:teuthology.orchestra.run.smithi103.stderr: 90.4% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mon.a.log.gz 2024-05-23T00:14:35.344 INFO:teuthology.orchestra.run.smithi105.stderr: 94.2% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.b.log.gz 2024-05-23T00:14:35.393 INFO:teuthology.orchestra.run.smithi105.stderr: 95.3% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.h.log.gz 2024-05-23T00:14:36.098 INFO:teuthology.orchestra.run.smithi174.stderr: 92.0% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.8.log.gz 2024-05-23T00:14:36.172 INFO:teuthology.orchestra.run.smithi105.stderr: 92.5% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.10.log.gz 2024-05-23T00:14:36.467 INFO:teuthology.orchestra.run.smithi174.stderr: 91.9% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.2.log.gz 2024-05-23T00:14:36.589 INFO:teuthology.orchestra.run.smithi103.stderr: 91.9% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.0.log.gz 2024-05-23T00:14:36.613 INFO:teuthology.orchestra.run.smithi105.stderr: 92.1% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.1.log.gz 2024-05-23T00:14:36.843 INFO:teuthology.orchestra.run.smithi174.stderr: 92.0% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.11.log.gz 2024-05-23T00:14:36.880 INFO:teuthology.orchestra.run.smithi103.stderr: 91.7% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.6.log.gz 2024-05-23T00:14:36.963 INFO:teuthology.orchestra.run.smithi103.stderr: 92.2% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.3.log.gz 2024-05-23T00:14:37.188 INFO:teuthology.orchestra.run.smithi105.stderr: 92.4% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.7.log.gz 2024-05-23T00:14:37.612 INFO:teuthology.orchestra.run.smithi105.stderr: 91.8% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.4.log.gz 2024-05-23T00:14:37.614 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-05-23T00:14:37.614 INFO:teuthology.orchestra.run.smithi105.stderr:real 0m3.942s 2024-05-23T00:14:37.614 INFO:teuthology.orchestra.run.smithi105.stderr:user 0m16.233s 2024-05-23T00:14:37.614 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m0.509s 2024-05-23T00:14:37.873 INFO:teuthology.orchestra.run.smithi103.stderr: 92.1% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.9.log.gz 2024-05-23T00:14:37.877 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-05-23T00:14:37.878 INFO:teuthology.orchestra.run.smithi103.stderr:real 0m4.191s 2024-05-23T00:14:37.878 INFO:teuthology.orchestra.run.smithi103.stderr:user 0m14.897s 2024-05-23T00:14:37.878 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m1.227s 2024-05-23T00:14:38.535 INFO:teuthology.orchestra.run.smithi174.stderr: 92.0% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-osd.5.log.gz 2024-05-23T00:14:47.677 INFO:teuthology.orchestra.run.smithi174.stderr: 93.3% -- replaced with /var/log/ceph/9c0f3dc8-1895-11ef-bc9a-c7b262605968/ceph-mds.i.log.gz 2024-05-23T00:14:47.678 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-05-23T00:14:47.678 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m14.007s 2024-05-23T00:14:47.679 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m27.442s 2024-05-23T00:14:47.679 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.852s 2024-05-23T00:14:47.679 INFO:tasks.cephadm:Archiving logs... 2024-05-23T00:14:47.680 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi103/log 2024-05-23T00:14:47.681 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-23T00:14:49.971 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi105/log 2024-05-23T00:14:49.972 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-23T00:14:52.169 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi174/log 2024-05-23T00:14:52.170 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-23T00:14:56.083 INFO:tasks.cephadm:Removing cluster... 2024-05-23T00:14:56.084 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 --force 2024-05-23T00:14:56.402 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-23T00:14:56.973 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 --force 2024-05-23T00:14:57.163 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-23T00:14:57.437 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9c0f3dc8-1895-11ef-bc9a-c7b262605968 --force 2024-05-23T00:14:57.626 INFO:teuthology.orchestra.run.smithi174.stdout:Deleting cluster with fsid: 9c0f3dc8-1895-11ef-bc9a-c7b262605968 2024-05-23T00:14:57.912 INFO:tasks.cephadm:Removing cephadm ... 2024-05-23T00:14:57.912 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-23T00:14:57.964 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-23T00:14:57.981 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-23T00:14:57.996 INFO:tasks.cephadm:Teardown complete 2024-05-23T00:14:57.997 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-23T00:14:58.008 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-23T00:14:58.009 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-23T00:14:58.016 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-23T00:14:58.024 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-23T00:14:58.163 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-23T00:14:58.174 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-23T00:14:58.174 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-23T00:14:58.181 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-23T00:14:58.184 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-23T00:14:58.197 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-05-23T00:14:58.197 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-05-23T00:14:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-23T00:14:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-05-23T00:14:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 35 -200us[ -200us] +/- 63ms 2024-05-23T00:14:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -456us[ -457us] +/- 54ms 2024-05-23T00:14:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +836us[ +836us] +/- 68ms 2024-05-23T00:14:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-23T00:14:58.203 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-23T00:14:58.203 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-05-23T00:14:58.203 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv01.front.sepia.ceph.com 3 7 377 106 -185us[ -188us] +/- 62ms 2024-05-23T00:14:58.203 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 38 -421us[ -421us] +/- 54ms 2024-05-23T00:14:58.204 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 +864us[ +864us] +/- 68ms 2024-05-23T00:14:58.204 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-23T00:14:58.222 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-05-23T00:14:58.230 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-23T00:14:58.230 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-05-23T00:14:58.231 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 34 -228us[ -223us] +/- 63ms 2024-05-23T00:14:58.231 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv02.front.sepia.ceph.com 3 6 377 37 -472us[ -467us] +/- 54ms 2024-05-23T00:14:58.231 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +822us[ +827us] +/- 68ms 2024-05-23T00:14:58.231 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-23T00:14:58.232 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-23T00:14:58.243 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-23T00:14:58.243 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-23T00:14:58.253 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-23T00:14:58.474 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-23T00:14:58.546 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-23T00:14:58.613 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-23T00:14:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-23T00:14:58.703 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-23T00:14:58.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-23T00:14:58.732 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-23T00:14:58.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-23T00:14:58.759 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-23T00:14:58.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-23T00:14:58.791 INFO:teuthology.task.internal:Duration was 2166.409284 seconds 2024-05-23T00:14:58.791 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-23T00:14:58.800 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-23T00:14:58.800 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-23T00:14:58.808 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-23T00:14:58.811 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-23T00:14:58.849 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-23T00:14:58.850 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-23T00:14:58.936 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-23T00:14:59.160 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-23T00:14:59.160 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-05-23T00:14:59.160 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-23T00:14:59.233 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-05-23T00:14:59.233 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-23T00:14:59.253 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-05-23T00:14:59.253 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-23T00:14:59.272 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-23T00:14:59.272 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-23T00:14:59.276 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-23T00:14:59.296 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-23T00:14:59.351 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-23T00:14:59.352 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-23T00:14:59.357 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-23T00:14:59.362 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-23T00:14:59.862 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-23T00:14:59.982 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-23T00:14:59.982 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-23T00:15:00.077 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-23T00:15:00.101 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-23T00:15:00.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-23T00:15:00.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-23T00:15:00.147 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-23T00:15:00.149 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-23T00:15:00.168 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-05-23T00:15:00.189 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-05-23T00:15:00.234 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-05-23T00:15:00.271 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-23T00:15:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-23T00:15:00.316 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-23T00:15:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-23T00:15:00.330 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-23T00:15:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-23T00:15:00.344 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-23T00:15:00.357 INFO:teuthology.task.internal:Transferring archived files... 2024-05-23T00:15:00.358 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi103 2024-05-23T00:15:00.358 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-23T00:15:00.464 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi105 2024-05-23T00:15:00.464 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-23T00:15:00.499 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-22_21:24:03-fs-squid-distro-default-smithi/7721265/remote/smithi174 2024-05-23T00:15:00.499 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-23T00:15:00.536 INFO:teuthology.task.internal:Removing archive directory... 2024-05-23T00:15:00.536 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-23T00:15:00.544 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-23T00:15:00.546 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-23T00:15:00.594 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-23T00:15:00.607 INFO:teuthology.task.internal:Not uploading archives. 2024-05-23T00:15:00.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-23T00:15:00.624 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-23T00:15:00.624 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-23T00:15:00.632 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-23T00:15:00.634 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-23T00:15:00.646 INFO:teuthology.orchestra.run.smithi105.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 23 00:15 /home/ubuntu/cephtest 2024-05-23T00:15:00.648 INFO:teuthology.orchestra.run.smithi174.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 23 00:15 /home/ubuntu/cephtest 2024-05-23T00:15:00.683 INFO:teuthology.orchestra.run.smithi103.stdout: 393505 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 23 00:15 /home/ubuntu/cephtest 2024-05-23T00:15:00.686 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-23T00:15:00.698 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-23T00:15:00.725 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 4fed890e8ce3966dbe380e9eb4b0a796eddb841c description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} duration: 2166.4092841148376 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi103 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=935217f1cb4dc722d5303f5ad364c883e9133376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=944ffd0891f944b483c44adb67388137 status: fail success: false 2024-05-23T00:15:00.726 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-23T00:15:00.813 INFO:teuthology.run:FAIL